use IsisDB module instead of OpenIsis -- this will fix various problems in

[webpac] / openisis / doc / Struct.txt

structuring ISIS records using subfields or subrecords


*       structures

The means by which an Isis record can be structured into "data elements"
("A defined unit of information", Z39.2 a.k.a. ISO2709)
fall in one of two broad categories (citing Z39.2):
-       subfields
        "A data element considered as a component of a field."
        In *ML (SGML,HTML,XML...), subfields correspond to a node's attributes.
        In MIME, subfields correspond to attributes of a MIME header value.
-       subrecords
        "A group of fields within a record that may be treated as a logical entity.
        (When a record describes more than one entity,
        the descriptions of individual entities may be treated as subrecords.)"
        In *ML, subrecords correspond to a node's childs.
        In MIME, subrecords correspond to multipart body parts.


*       subfields

Since a field value can actually be anything,
including XML text or a serialized (textual or binary) Isis record,
it can be arbitrarily structured according to a regular expression
or some other grammar (machine parseable or not).


The term subfield, however, is used for a range of characters in the value
which is identified by rather simple means:
-       fixed
        if all (or all but the last) subfields have a fixed length
        and are neither optional nor repeatable,
        then each subfield can be found at a fixed position.
-       delimited with optional identifier
        this is the proper Z39.2 notion of a subfield.

If a special delimiter character is found in the field,
it breaks the field into subfields.
Z39.2, and thus MARC, use the character 31 as delimiter
(hex 1F, CTRL-_, ASCII "unit separator" US).
Traditional Isis uses the caret '^'.

OpenIsis permits any character, including the horizontal TAB and semicolon.
More precisely, OpenIsis reverts Z39.2's notion that
"every subfield is INTRODUCED by a delimiter, unless it isn't"
to the principle that for every data element, it is specified
how it's end is detected, including by fixed length or varying delimiters.


The initial n characters of a subfield are used to identify the subfield.
Z39.2 permits any (small) fixed value for n, including 0, i.e. not identified.
The MARC family of standards uses n=1.
OpenIsis allows for any value, including variable length identifiers,
which are themselves delimited by some character like a '=' (see below).


Z39.2 states that if identifiers are used, each must be preceeded
by a delimiter, and every data element, including the first,
must be identified that way. However, an initial range of m characters
(i.e. preceeding the first delimiter) in every field may serve as "indicator",
which is not regarded a "data element". Again, m is a small fixed number;
MARC uses m=2. Traditional Isis has no special support for indicators.
OpenIsis allows to access whatever is before the first delimiter.


Different subfielding methods can be mixed or nested.
Typical cases are:
-       mixed fixed/delimited
        After some initial fixed subfields, following subffields are delimited.
        This can be used to describe MARC's fixed indicators.
-       nested delimited/fixed
        A delimited subfield has itself a fixed substructure.
        Actually the leading identifier in a subfield can be regarded
        as fixed part in a mixed substructure.
-       nested unidentified delimited
        A delimited subfield has itself a delimited structure.
        This can be used to model variable length identifiers.

In other words, identifiers are themselves nothing but subfields
used as keys on some level of nesting.
On the other hand, any subfield could serve as a key for it's parent.
This is used e.g. to select a field by a subfield indicating a language
(see below for keyed subrecords).

If you look at the
>       Serialized      plaintext representation of an Isis record,
actually the whole record is a newline delimited value,
the whole database is a blankline (double newline) delimited value
and each field has it's tag as initial tab-delimited subfield.


In the future, OpenIsis will add support for a wide variety of
subfielding techniques such as defined by regular expressions,
MIME headers or produced in typical "character/comma separated values" files
(opionally using quotes).

Since splitting subfields is mostly and can always be done on the
application level (i.e. a database server rarely needs to care),
"support" essentially boils down to the definition of appropriate meta data.


*       subrecords

A subrecord consists of a typically continuous range of fields within a record,
started by some field to introduce the subrecord.
Some variants, however, like keyed subfields,
can be freely scattered and don't need a "header" field.


There are basically four ways to denote the boundaries of structures:
-       embraced
        where a special field is used to denote the structures end.
        This resembles SGML-style notations,
        where each opening tag is matched by a closing tag.
        This is relatively easy and recommended for every day use.
-       marked
        where the fields of the child structure are marked as such.
        This is sort of the opposite approach of embracing.
        Marking comes in several powerful flavours,
        see below for a more detailled discussion.
-       counted
        where the number of fields (not childs) belonging to the
        structure is given in (any leading digits of) the initial field.
        This allows for safe embedding regardless of the
        structure's contents and is thus used in contexts where
        full generality is needed like when embedding result records
        within a server's response.
-       implicit
        where the number of childs is fixed.
        An example of this is the parse tree of a query,
        where the structure "AND" has exactly two childs
        (which in turn might be structures).
        This is used mostly for internal structures like parsed
        queries or formats, which are not meant to be exchanged.

The field introducing a subrecord might have any subfields
just like other fields, similar to the attributes that might
be assigned to a tag in SGML applications like HTML.

However, the first subfield (unidentified initial characters)
of a field opening an embraced or counted subrecord is reserved as indicator:
-       a plus sign '+' as first character
        indicates explicity opening a subrecord
-       a minus sign '-' as first character
        indicates an empty subrecord (containing no childs)
-       an empty value
        indicates explicity closing a subrecord
        (similar to the closing blank line used in several protocols)
-       an initial numeric value
        (of decimal digits) gives the number of fields to follow.
-       an initial character @A-Z
        gives the number of childs to follow (@=0,A=1,B=2...) (rarely used)

Auxiliary information about the child,
like an embedded records row number and type,
are stored in subfields of the parent.


*       conventions

While the intented usage of subrecords might be specified in
more detail in the
>       Meta    table metadata
, the schema can also be used standalone (without referring to metadata),
if some conventions on tag ranges are followed.

The extend of subrecords by length or braces can be safely
determined if you just know that you want the given field
to be regarded as subrecord.

For subrecords of fixed number of childs (meant for internal use),
it is necessary to recognize whether a following field is itself a structure.
If they are used at all, the tag range -1..-99 should be reserved for this
purpose.

In this context, typically one of two modes is used:
-       the MIME processing mode for processing list-style content,
        assumes that negative tags denote structures,
        while positive contain plain data.
-       in XML processing mode, everything but the 0 tag (text node) is a structure.

If a parent has a subfield ^0,
that should contain the childs identity as dbname or mfn or dbname.mfn.
If the parents indicator is delimited by a tab instead of a ^,
the next tab-delimited subfield is interpreted that way (where applicable).


*       marked structures

There is a wide variety of techniques for marking fields as "childs"
of other fields. Marking techniques work especially well for a single
level of substructuring; for nested structures, some restrictions apply.

We give some commonly used examples:

-       quoting
        is done by prefixing every child field value with a special string,
        which is not used as prefix outside the child fields.
        However, at least for a single level of quoting, it does not impose
        a problem if the child fields themselves started with the same prefix:
        Still, the original value is retrieved by stripping the (first) prefix.
        This even works for multiple levels, as long as the record was properly
        constructed, i.e. the quoting prefix is not used outside childs.
        Examples are the output of the diff command (which is driving the
        RCS/CVS revision control system very reliably) and the '>' quoting
        used in e-mail replies.
-       tagging
        Instead of the field value, of course also the field tag can be used
        as child mark. In some situations it might be possible to choose
        appropriate reserved tags for the childs.
        In other situations, where some given child tag must be kept,
        it can be stored as prefix in the field value according to the canonical
>       Serialized
        plain text format.
-       keying
        If the mark used is dependent on an attribute of the parent field,
        the childs can be determined even if non-continuous.
        With some more cooperation of the childs, the mark might be an
        attribute (subfield) instead of a prefix (indicator).
        That way, childs and parents are linked together rather logically
        than "physically" by a common key just like in relational databases.
        This easily extends to multiple levels using segmented keys
        (consisting of several attributes/subfields).
        While this scheme only works with well behaved childs and may waste
        some space by replicating keys, it is simple and robust and gives
        convenient access to the childs without inspecting the structure.


*design childs vs. attributes

Every information that can be represented using an attribute,
can also be represented using a child.
From that point of view, attributes are a redundant "language" construct
and one might deem a model using only childs as the simpler one.
We call such an attributeless model "canonical verbose" representation.
It's a little bit similar to the "everything is an object"
approach of pure OO languages like Smalltalk.


But then, having a richer language isn't always such a bad thing,
if you know how to use it appropriately.
(This "if" is the core of almost any serious criticism of rich languages,
but for now, let's assume we know what we're doing).
Appropriate use basically boils down to choosing the language construct
that was just made for your situation, i.e. not the most general one,
but quite to the opposite the most specific (restricted) one.
That way you will not only have the most efficient representation,
but also express additional information about what's going on. 


In short, a "canonical compact" modelling can be based upon the principle
"Use attributes wherever possible".

Some logical property of a logical structure can be represented by
means of attributes, if
-       it is simple,
        i.e. one single string value.
-       or at least flat,
        i.e. itself a structure that can be represented based on attributes
        that do not interfere with the parents attributes.
        In the latter case, the property will show up as several
        logically interrelated attributes of the parent.
        However, such a flat group of attributes might be a candidate
        for a child under some circumstances.
-       it is not repeatable.
        Although OpenIsis supports repeated subfields as used by some MARCs,
        XML/SGML attributes can not be repeated.
        (Technically, they can, but there neither is defined semantics for
        repeated attributes nor is access supported by parsers or the DOM).
        Moreover, traditional CDS/ISIS implementations do not support
        repeated subfields, so it's probably a good idea to not use them
        without a pretty good reason.

Basically, when you think C, one field's attributes take everything
that goes into a simple struct, without using arrays or pointers.


The detailled modelling should also take into account the intended usage.

For example, one might devise some attribute candidates to childs, if
-       they are likely to be accessed or modified together
        but independent of other properties
-       they are candidates to be inherited or overridden as a group in a
>       PatchWork
-       the parent would otherwise become very large


*variants       variant structures

The C language construct of a "union" is frequently used in bibliographic
databases. The typical form resembles the PASCAL "variant record",
using an initial field as indicator for the usage of the given field.
Sometimes, however, the more liberal C practice is used,
where the intented interpretation is specified somewhere in the record, somehow.

A similar construct is used in ALGOL-derived OO languages like C++ or Java,
where the indicator (of what object is this ?) is out-of-band data
(i.e. cannot be modified or inspected like any other data).


In Isis records, fields always have a tag
(and subfields commonly have an identifier) indicating the kind of data.
Therefore, there is little need to introduce another level of switches.
A canonically decomposed model
-       would not reuse fields or subfields with different structure
-       would not contain rules like
        "if subfield a has value b then subfield c must be present"

However, on the other hand, full decomposition might be tedious and
even hide relationships. Moreover, from a given point of view,
tags and identifiers are just ordinary subfields on some level.


In general, if the same tag is used for variants of a field,
the risk of misinterpretation of data should be minimized by
not reusing the same subfields with different structure.
After all, defining another indicator and ignoring an unexpected subfield
or moaning on the lack of an expected one is cheaper and more robust and clear
than verifying an expected structure based on other subfield values.


*       examples

A typical HTML table definition starting with
$
<table width="100%" cellpadding="0" cellspacing="0"
  marginwidth="0" marginheight="0" topmargin="0" leftmargin="0" border="0">
<tr>
<td valign="top" width="160">
this is the textbody <br/> of the td node
</td>
</tr>
...
$
will be compacted to, say,
$
100     +^w100%^p0^s0^m0^h0^t0^l0^b0
101     +
102     +^vtop^w160
0       this is the textbody
103     -
0       of the td node
102
101
...
$
For a detailed description of the transformation, see
>       xmlisis the XML-ISIS doku

A six field result record might be embedded within a response like
$
908     6       cds.47
24      Hydrological achievements and social problems
...
$

Assuming we gave tag -20 to "OR" (and 0 to a literal),
the query "plant OR water" might be parsed to
$
-20     B
0       plant
0       water
$

"frog AND (plant OR water)" might look like, if -21 is assigned to "AND"
$
-21     B
0       frog
-20     B
0       plant
0       water
$

For implicit tags, the number of childs is redundant
(fixed per tag in a given use) and will typically be omitted.

---
        $Id: Struct.txt,v 1.8 2003/06/23 14:44:29 kripke Exp $