With the emerging use of unicode in ISIS databases,
the 30 byte maximum on keys might become a somewhat
narrow limit for languages using 2 or 3 UTF-8 bytes per
character while still having lots of characters per word.
Where UTF-16/UCS-2 is used, all languages would be
equally affected by an effective maximum key of
15 characters.

Therefore, there is a clear need for a new index
structure supporting longer keys for some environments.
This would in itself not raise any compatibility issues,
since an index is always redundant and can be reconstructed
by any plattform in a suitable format for this plattform
(of course according to the plattform's builtin limits).

While it would be straightforward to just create some n03/l03
pair of files or change 01/02 parameters, and some tools
might even care for the control-file contents and work without
modification using such indexes, some other tools wouldn't.


In this situation, one may want to spend some time thinking
about how an index should be organised. For programmers having
coped with ISAM and databases in theory and practice,
the reasons for some design decisions of traditional
ISIS index structures are less than obvious.


*	blank padding
a typical index-block lookup routine does not benefit in speed
from blank-padding. while the fixed-array organisation allows
for very easy Pascal implementation (where you can't cast
some part of a block to a struct), C code doesn't care.
instead, the additional space accounts for
additional IO and therefore slows down index access.
considering that many DBs even allow for compressed index files,
where every key is stored as a number of leading bytes it
shares with it's predecessor followed by the trailing difference,
one should expect some performance improvement from at least
eliminating all those blanks.

*	block alignment
the node and leaves files are made up of blocks not aligned
at any page boundaries. Every B*tree in textbooks as well as
in real-world databases uses page-aligned blocks for best IO
performance. potential reader/writer concurrency problems
(readers reading partially updated blocks) are also reduced
for most systems when IO occurs at filesystem page boundaries.

*	multiple files
The separation of one index structures nodes and leaves is
due to the fact that they do have different block sizes.
The obvious advantage of having two index file structures is to
reduce the space wasted by blank padding.
Using page-aligned blocks of fix size with unpadded keys
would allow the complete index to go into one single file.
Moreover, the postings could be stored together with the
leaves, further reducing IO.

---
	$Id: longidx.txt,v 1.2 2002/11/13 18:52:55 kripke Exp $