1 <!doctype linuxdoc system>
4 $Id: zebra.sgml,v 1.42 1998-09-22 10:03:39 adam Exp $
8 <title>Zebra Server - Administrators's Guide and Reference
9 <author><htmlurl url="http://www.indexdata.dk/" name="Index Data">,
10 <tt><htmlurl url="mailto:info@indexdata.dk" name="info@indexdata.dk"></>
11 <date>$Revision: 1.42 $
13 The Zebra information server combines a versatile fielded/free-text
14 search engine with a Z39.50-1995 frontend to provide a powerful and flexible
15 information management system. This document explains the procedure for
16 installing and configuring the system, and outlines the possibilities
17 for managing data and providing Z39.50
18 services with the software.
28 The Zebra system is a fielded free-text indexing and retrieval engine with a
29 Z39.50 frontend. You can use any commercial or freeware Z39.50 client
30 to access data stored in Zebra.
32 The Zebra server is our first step towards the development of a fully
33 configurable, open information system. Eventually, it will be paired
34 off with a powerful Z39.50 client to support complex information
35 management tasks within almost any application domain. We're making
36 the server available now because it's no fun to be in the open
37 information retrieval business all by yourself. We want to allow
38 people with interesting data to make their things
39 available in interesting ways, without having to start out
40 by implementing yet another protocol stack from scratch.
42 This document is an introduction to the Zebra system. It will tell you
43 how to compile the software, and how to prepare your first database.
44 It also explains how the server can be configured to give you the
45 functionality that you need.
47 If you find the software interesting, you should join the support
48 mailing-list by sending email to <tt/zebra-request@indexdata.dk/.
53 This is a list of some of the most important features of the
59 Supports updating - records can be added and deleted without
60 rebuilding the index from scratch.
61 The update procedure is tolerant to crashes or hard interrupts
62 during register updating - registers can be reconstructed following a crash.
63 Registers can be safely updated even while users are accessing the server.
66 Supports large databases - files for indices, etc. can be
67 automatically partitioned over multiple disks.
70 Supports arbitrarily complex records - base input format is an
71 SGML-like syntax which allows nested (structured) data elements, as
72 well as variant forms of data.
75 Supports random storage formats. A system of input filters driven by
76 regular expressions allows you to easily process most ASCII-based
77 data formats. SGML, ISO2709 (MARC), and raw text are also supported.
80 Supports boolean queries as well as relevance-ranking (free-text)
81 searching. Right truncation and masking in terms are supported, as
82 well as full regular expressions.
85 Supports multiple concrete syntaxes
86 for record exchange (depending on the configuration): GRS-1, SUTRS,
87 ISO2709 (*MARC). Records can be mapped between record syntaxes and
91 Supports approximate matching in registers (ie. spelling mistakes,
102 Protocol facilities: Init, Search, Retrieve, Browse and Sort.
105 Piggy-backed presents are honored in the search-request.
108 Named result sets are supported.
111 Easily configured to support different application profiles, with
112 tables for attribute sets, tag sets, and abstract syntaxes.
113 Additional tables control facilities such as element mappings to
114 different schema (eg., GILS-to-USMARC).
117 Complex composition specifications using Espec-1 are partially
118 supported (simple element requests only).
121 Element Set Names are defined using the Espec-1 capability of the
122 system, and are given in configuration files as simple element
123 requests (and possibly variant requests).
126 Some variant support (not fully implemented yet).
129 Using the YAZ toolkit for the protocol implementation, the
130 server can utilise a plug-in XTI/mOSI implementation (not included) to
131 provide SR services over an OSI stack, as well as Z39.50 over TCP/IP.
134 Zebra runs on most Unix-like systems as well as Windows NT - a binary
135 distribution for Windows NT is forthcoming - so far, the installation
136 requires MSVC++ to compile the system (we use version 5.0).
143 This is a beta-release of the software, to allow you to look at
144 it - try it out, and assess whether it can be of use to you.
146 These are some of the plans that we have for the software in the near
147 and far future, approximately ordered after their relative importance.
149 asterisk will be implemented before the
155 *Complete the support for variants.
158 *Finalize the data element <it/include/ facility to support multimedia
159 data elements in records.
162 Add more sophisticated relevance ranking mechanisms. Add support for soundex
163 and stemming. Add relevance <it/feedback/ support.
166 Complete EXPLAIN support.
169 Add support for very large records by implementing segmentation and/or
173 Support the Item Update extended service of the protocol.
176 We want to add a management system that allows you to
177 control your databases and configuration tables from a graphical
178 interface. We'll probably use Tcl/Tk to stay platform-independent.
182 Programmers thrive on user feedback. If you are interested in a facility that
183 you don't see mentioned here, or if there's something you think we
184 could do better, please drop us a mail. If you think it's all really
185 neat, you're welcome to drop us a line saying that, too. You'll find
186 contact info at the end of this file.
188 <sect>Compiling the software
191 An ANSI C compiler is required to compile the Zebra
192 server system — <tt/gcc/ works fine if your own system doesn't
193 provide an adequate compiler.
195 Unpack the distribution archive. The <tt>configure</tt> shell script
196 attempts to guess correct values for various system-dependent variables
197 used during compilation. It uses those values to create a 'Makefile' in
198 each directory of Zebra.
200 To run the configure script type:
205 The configure script attempts to use C compiler specified by
206 the <tt>CC</tt> environment variable. If not set, <tt>cc</tt>
207 will be used. The <tt>CFLAGS</tt> environment variable holds
208 options to be passed to the C compiler. If you're using a Bourne-shell
209 compatible shell you may pass something like this:
211 CC=/opt/ccs/bin/cc CFLAGS=-O ./configure
214 When configured build the software by typing:
219 As an option you may type <tt>make depend</tt> to create
220 source file dependencies for the package. This is only needed,
221 however, if you alter the source.
223 If successful, two executables have been created in the sub-directory
226 <tag><tt>zebrasrv</tt></tag> The Z39.50 server and search engine.
227 <tag><tt>zebraidx</tt></tag> The administrative tool for the search index.
232 In this section, we will test the system by indexing a small set of sample
233 GILS records that are included with the software distribution. Go to the
234 <tt>test/gils</tt> subdirectory of the distribution archive. There you will
236 file named <tt>zebra.cfg</tt> with the following contents:
238 # Where are the YAZ tables located.
239 profilePath: ../../../yaz/tab ../../tab
241 # Files that describe the attribute sets supported.
246 Now, edit the file and set <tt>profilePath</tt> to the path of the
247 YAZ profile tables (sub directory <tt>tab</tt> of the YAZ distribution
250 The 48 test records are located in the sub directory <tt>records</tt>.
251 To index these, type:
253 $ ../../index/zebraidx -t grs.sgml update records
256 In the command above the option <tt>-t</tt> specified the record
257 type — in this case <tt>grs.sgml</tt>. The word <tt>update</tt> followed
258 by a directory root updates all files below that directory node.
260 If your indexing command was successful, you are now ready to
261 fire up a server. To start a server on port 2100, type:
263 $ ../../index/zebrasrv tcp:@:2100
266 The Zebra index that you have just created has a single database
267 named <tt/Default/. The database contains records structured according to
268 the GILS profile, and the server will
269 return records in either either USMARC, GRS-1, or SUTRS depending
270 on what your client asks
273 To test the server, you can use any Z39.50 client (1992 or later). For
274 instance, you can use the demo client that comes with YAZ: Just cd to
275 the <tt/client/ subdirectory of the YAZ distribution and type:
278 $ client tcp:localhost:2100
281 When the client has connected, you can type:
288 The default retrieval syntax for the client is USMARC. To try other
289 formats for the same record, try:
300 <it>NOTE: You may notice that more fields are returned when your
301 client requests SUTRS or GRS-1 records. When retrieving GILS records,
302 this is normal - not all of the GILS data elements have mappings in
303 the USMARC record format.</it>
305 If you've made it this far, there's a good chance that
306 you've got through the compilation OK.
308 <sect>Administrating Zebra<label id="administrating">
311 Unlike many simpler retrieval systems, Zebra supports safe, incremental
312 updates to an existing index.
314 Normally, when Zebra modifies the index it reads a number of records
316 Depending on your specifications and on the contents of each record
317 one the following events take place for each record:
319 <tag>Insert</tag> The record is indexed as if it never occurred
320 before. Either the Zebra system doesn't know how to identify the record or
321 Zebra can identify the record but didn't find it to be already indexed.
322 <tag>Modify</tag> The record has already been indexed. In this case
323 either the contents of the record or the location (file) of the record
324 indicates that it has been indexed before.
325 <tag>Delete</tag> The record is deleted from the index. As in the
326 update-case it must be able to identify the record.
329 Please note that in both the modify- and delete- case the Zebra
330 indexer must be able to generate a unique key that identifies the record in
331 question (more on this below).
333 To administrate the Zebra retrieval system, you run the
334 <tt>zebraidx</tt> program. This program supports a number of options
335 which are preceded by a minus, and a few commands (not preceded by
338 Both the Zebra administrative tool and the Z39.50 server share a
339 set of index files and a global configuration file. The
340 name of the configuration file defaults to <tt>zebra.cfg</tt>.
341 The configuration file includes specifications on how to index
342 various kinds of records and where the other configuration files
343 are located. <tt>zebrasrv</tt> and <tt>zebraidx</tt> <em>must</em>
344 be run in the directory where the configuration file lives unless you
345 indicate the location of the configuration file by option
348 <sect1>Record Types<label id="record-types">
350 Indexing is a per-record process, in which either insert/modify/delete
351 will occur. Before a record is indexed search keys are extracted from
352 whatever might be the layout the original record (sgml,html,text, etc..).
353 The Zebra system currently supports two fundamantal types of records:
354 structured and simple text.
355 To specify a particular extraction process, use either the
356 command line option <tt>-t</tt> or specify a
357 <tt>recordType</tt> setting in the configuration file.
359 <sect1>The Zebra Configuration File<label id="configuration-file">
361 The Zebra configuration file, read by <tt>zebraidx</tt> and
362 <tt>zebrasrv</tt> defaults to <tt>zebra.cfg</tt> unless specified
363 by <tt>-c</tt> option.
365 You can edit the configuration file with a normal text editor.
366 Parameter names and values are seperated by colons in the file. Lines
367 starting with a hash sign (<tt/#/) are treated as comments.
369 If you manage different sets of records that share common
370 characteristics, you can organize the configuration settings for each
371 type into &dquot;groups&dquot;.
372 When <tt>zebraidx</tt> is run and you wish to address a given group
373 you specify the group name with the <tt>-g</tt> option. In this case
374 settings that have the group name as their prefix will be used
375 by <tt>zebraidx</tt>. If no <tt/-g/ option is specified, the settings
376 with no prefix are used.
378 In the configuration file, the group name is placed before the option
379 name itself, separated by a dot (.). For instance, to set the record type
380 for group <tt/public/ to <tt/grs.sgml/ (the SGML-like format for structured
381 records) you would write:
384 public.recordType: grs.sgml
387 To set the default value of the record type to <tt/text/ write:
393 The available configuration settings are summarized below. They will be
394 explained further in the following sections.
397 <tag><it>group</it>.recordType[<it>.name</it>]</tag>
398 Specifies how records with the file extension <it>name</it> should
399 be handled by the indexer. This option may also be specified
400 as a command line option (<tt>-t</tt>). Note that if you do not
401 specify a <it/name/, the setting applies to all files. In general,
402 the record type specifier consists of the elements (each
403 element separated by dot), <it>fundamental-type</it>,
404 <it>file-read-type</it> and arguments. Currently, two
405 fundamental types exist, <tt>text</tt> and <tt>grs</tt>.
406 <tag><it>group</it>.recordId</tag>
407 Specifies how the records are to be identified when updated. See
408 section <ref id="locating-records" name="Locating Records">.
409 <tag><it>group</it>.database</tag>
410 Specifies the Z39.50 database name.
411 <tag><it>group</it>.storeKeys</tag>
412 Specifies whether key information should be saved for a given
413 group of records. If you plan to update/delete this type of
414 records later this should be specified as 1; otherwise it
415 should be 0 (default), to save register space. See section
416 <ref id="file-ids" name="Indexing With File Record IDs">.
417 <tag><it>group</it>.storeData</tag>
418 Specifies whether the records should be stored internally
419 in the Zebra system files. If you want to maintain the raw records yourself,
420 this option should be false (0). If you want Zebra to take care of the records
421 for you, it should be true(1).
423 Specifies the location of the various register files that Zebra uses
424 to represent your databases. See section
425 <ref id="register-location" name="Register Location">.
427 Enables the <it/safe update/ facility of Zebra, and tells the system
428 where to place the required, temporary files. See section
429 <ref id="shadow-registers" name="Safe Updating - Using Shadow Registers">.
431 Directory in which various lock files are stored.
433 Directory in which temporary files used during zebraidx' update
436 Specifies the directory that the server uses for temporary result sets.
437 If not specified <tt>/tmp</tt> will be used.
438 <tag>profilePath</tag>
439 Specifies the location of profile specification files.
441 Specifies the filename(s) of attribute set files for use in
442 searching. At least the Bib-1 set should be loaded (<tt/bib1.att/).
443 The <tt/profilePath/ setting is used to look for the specified files.
444 See section <ref id="attset-files" name="The Attribute Set Files">
446 Specifies size of internal memory to use for the zebraidx program. The
447 amount is given in megabytes - default is 4 (4 MB).
449 <sect1>Locating Records<label id="locating-records">
451 The default behaviour of the Zebra system is to reference the
452 records from their original location, i.e. where they were found when you
453 ran <tt/zebraidx/. That is, when a client wishes to retrieve a record
454 following a search operation, the files are accessed from the place
455 where you originally put them - if you remove the files (without
456 running <tt/zebraidx/ again, the client will receive a diagnostic
459 If your input files are not permanent - for example if you retrieve
460 your records from an outside source, or if they were temporarily
461 mounted on a CD-ROM drive,
462 you may want Zebra to make an internal copy of them. To do this,
463 you specify 1 (true) in the <tt>storeData</tt> setting. When
464 the Z39.50 server retrieves the records they will be read from the
465 internal file structures of the system.
467 <sect1>Indexing with no Record IDs (Simple Indexing)
470 If you have a set of records that are not expected to change over time
471 you may can build your database without record IDs.
472 This indexing method uses less space than the other methods and
475 To use this method, you simply omit the <tt>recordId</tt> entry
476 for the group of files that you index. To add a set of records you use
477 <tt>zebraidx</tt> with the <tt>update</tt> command. The
478 <tt>update</tt> command will always add all of the records that it
479 encounters to the index - whether they have already been indexed or
480 not. If the set of indexed files change, you should delete all of the
481 index files, and build a new index from scratch.
483 Consider a system in which you have a group of text files called
484 <tt>simple</tt>. That group of records should belong to a Z39.50 database
485 called <tt>textbase</tt>. The following <tt/zebra.cfg/ file will suffice:
488 profilePath: /usr/local/yaz
490 simple.recordType: text
491 simple.database: textbase
494 Since the existing records in an index can not be addressed by their
495 IDs, it is impossible to delete or modify records when using this method.
497 <sect1>Indexing with File Record IDs<label id="file-ids">
500 If you have a set of files that regularly change over time: Old files
501 are deleted, new ones are added, or existing files are modified, you
502 can benefit from using the <it/file ID/ indexing methodology. Examples
503 of this type of database might include an index of WWW resources, or a
504 USENET news spool area. Briefly speaking, the file key methodology
505 uses the directory paths of the individual records as a unique
506 identifier for each record. To perform indexing of a directory with
507 file keys, again, you specify the top-level directory after the
508 <tt>update</tt> command. The command will recursively traverse the
509 directories and compare each one with whatever have been indexed before in
510 that same directory. If a file is new (not in the previous version of
511 the directory) it is inserted into the registers; if a file was
512 already indexed and it has been modified since the last update,
513 the index is also modified; if a file has been removed since the last
514 visit, it is deleted from the index.
516 The resulting system is easy to administrate. To delete a record you
517 simply have to delete the corresponding file (say, with the <tt/rm/
518 command). And to add records you create new files (or directories with
519 files). For your changes to take effect in the register you must run
520 <tt>zebraidx update</tt> with the same directory root again. This mode
521 of operation requires more disk space than simpler indexing methods,
522 but it makes it easier for you to keep the index in sync with a
523 frequently changing set of data. If you combine this system with the
524 <it/safe update/ facility (see below), you never have to take your
525 server offline for maintenance or register updating purposes.
527 To enable indexing with pathname IDs, you must specify <tt>file</tt> as
528 the value of <tt>recordId</tt> in the configuration file. In addition,
529 you should set <tt>storeKeys</tt> to <tt>1</tt>, since the Zebra
530 indexer must save additional information about the contents of each record
531 in order to modify the indices correctly at a later time.
533 For example, to update records of group <tt>esdd</tt> located below
534 <tt>/data1/records/</tt> you should type:
536 $ zebraidx -g esdd update /data1/records
539 The corresponding configuration file includes:
542 esdd.recordType: grs.sgml
546 <em>Important note: You cannot start out with a group of records with simple
547 indexing (no record IDs as in the previous section) and then later
548 enable file record Ids. Zebra must know from the first time that you
550 the files should be indexed with file record IDs.
553 You cannot explicitly delete records when using this method (using the
554 <bf/delete/ command to <tt/zebraidx/. Instead
555 you have to delete the files from the file system (or move them to a
557 and then run <tt>zebraidx</tt> with the <bf/update/ command.
559 <sect1>Indexing with General Record IDs
561 When using this method you construct an (almost) arbritrary, internal
562 record key based on the contents of the record itself and other system
563 information. If you have a group of records that explicitly associates
564 an ID with each record, this method is convenient. For example, the
565 record format may contain a title or a ID-number - unique within the group.
566 In either case you specify the Z39.50 attribute set and use-attribute
567 location in which this information is stored, and the system looks at
568 that field to determine the identity of the record.
570 As before, the record ID is defined by the <tt>recordId</tt> setting
571 in the configuration file. The value of the record ID specification
572 consists of one or more tokens separated by whitespace. The resulting
574 represented in the index by concatenating the tokens and separating them by
577 There are three kinds of tokens:
579 <tag>Internal record info</tag> The token refers to a key that is
580 extracted from the record. The syntax of this token is
581 <tt/(/ <em/set/ <tt/,/ <em/use/ <tt/)/, where <em/set/ is the
582 attribute set name <em/use/ is the name or value of the attribute.
583 <tag>System variable</tag> The system variables are preceded by
584 <verb>$</verb> and immediately followed by the system variable name, which
587 <tag>group</tag> Group name.
588 <tag>database</tag> Current database specified.
589 <tag>type</tag> Record type.
591 <tag>Constant string</tag> A string used as part of the ID — surrounded
592 by single- or double quotes.
595 For instance, the sample GILS records that come with the Zebra
596 distribution contain a unique ID in the data tagged Control-Identifier.
597 The data is mapped to the Bib-1 use attribute Identifier-standard
598 (code 1007). To use this field as a record id, specify
599 <tt>(bib1,Identifier-standard)</tt> as the value of the
600 <tt>recordId</tt> in the configuration file.
601 If you have other record types that uses the same field for a
602 different purpose, you might add the record type
603 (or group or database name) to the record id of the gils
604 records as well, to prevent matches with other types of records.
605 In this case the recordId might be set like this:
607 gils.recordId: $type (bib1,Identifier-standard)
610 (see section <ref id="data-model" name="Configuring Your Data Model">
611 for details of how the mapping between elements of your records and
612 searchable attributes is established).
614 As for the file record ID case described in the previous section,
615 updating your system is simply a matter of running <tt>zebraidx</tt>
616 with the <tt>update</tt> command. However, the update with general
617 keys is considerably slower than with file record IDs, since all files
618 visited must be (re)read to discover their IDs.
620 As you might expect, when using the general record IDs
621 method, you can only add or modify existing records with the <tt>update</tt>
622 command. If you wish to delete records, you must use the,
623 <tt>delete</tt> command, with a directory as a parameter.
624 This will remove all records that match the files below that root
627 <sect1>Register Location<label id="register-location">
630 Normally, the index files that form dictionaries, inverted
631 files, record info, etc., are stored in the directory where you run
632 <tt>zebraidx</tt>. If you wish to store these, possibly large, files
633 somewhere else, you must add the <tt>register</tt> entry to the
634 <tt/zebra.cfg/ file. Furthermore, the Zebra system allows its file
636 span multiple file systems, which is useful for managing very large
639 The value of the <tt>register</tt> setting is a sequence of tokens.
640 Each token takes the form:
642 <em>dir</em><tt>:</tt><em>size</em>.
644 The <em>dir</em> specifies a directory in which index files will be
645 stored and the <em>size</em> specifies the maximum size of all
646 files in that directory. The Zebra indexer system fills each directory
647 in the order specified and use the next specified directories as needed.
648 The <em>size</em> is an integer followed by a qualifier
649 code, <tt>M</tt> for megabytes, <tt>k</tt> for kilobytes.
651 For instance, if you have allocated two disks for your register, and
652 the first disk is mounted
653 on <tt>/d1</tt> and has 200 Mb of free space and the
654 second, mounted on <tt>/d2</tt> has 300 Mb, you could
655 put this entry in your configuration file:
657 register: /d1:200M /d2:300M
660 Note that Zebra does not verify that the amount of space specified is
661 actually available on the directory (file system) specified - it is
662 your responsibility to ensure that enough space is available, and that
663 other applications do not attempt to use the free space. In a large production system,
664 it is recommended that you allocate one or more filesystem exclusively
665 to the Zebra register files.
667 <sect1>Safe Updating - Using Shadow Registers<label id="shadow-registers">
672 The Zebra server supports <it/updating/ of the index structures. That is,
673 you can add, modify, or remove records from databases managed by Zebra
674 without rebuilding the entire index. Since this process involves
675 modifying structured files with various references between blocks of
676 data in the files, the update process is inherently sensitive to
677 system crashes, or to process interruptions: Anything but a
678 successfully completed update process will leave the register files in
679 an unknown state, and you will essentially have no recourse but to
680 re-index everything, or to restore the register files from a backup
681 medium. Further, while the update process is active, users cannot be
682 allowed to access the system, as the contents of the register files
683 may change unpredictably.
685 You can solve these problems by enabling the shadow register system in
686 Zebra. During the updating procedure, <tt/zebraidx/ will temporarily
687 write changes to the involved files in a set of &dquot;shadow
688 files&dquot;, without modifying the files that are accessed by the
689 active server processes. If the update procedure is interrupted by a
690 system crash or a signal, you simply repeat the procedure - the
691 register files have not been changed or damaged, and the partially
692 written shadow files are automatically deleted before the new updating
695 At the end of the updating procedure (or in a separate operation, if
696 you so desire), the system enters a &dquot;commit mode&dquot;. First,
697 any active server processes are forced to access those blocks that
698 have been changed from the shadow files rather than from the main
699 register files; the unmodified blocks are still accessed at their
700 normal location (the shadow files are not a complete copy of the
701 register files - they only contain those parts that have actually been
702 modified). If the commit process is interrupted at any point during the
703 commit process, the server processes will continue to access the
704 shadow files until you can repeat the commit procedure and complete
705 the writing of data to the main register files. You can perform
706 multiple update operations to the registers before you commit the
707 changes to the system files, or you can execute the commit operation
708 at the end of each update operation. When the commit phase has
709 completed successfully, any running server processes are instructed to
710 switch their operations to the new, operational register, and the
711 temporary shadow files are deleted.
713 <sect2>How to Use Shadow Register Files
716 The first step is to allocate space on your system for the shadow
717 files. You do this by adding a <tt/shadow/ entry to the <tt/zebra.cfg/
718 file. The syntax of the <tt/shadow/ entry is exactly the same as for
719 the <tt/register/ entry (see section <ref name="Register Location"
720 id="register-location">). The location of the shadow area should be
721 <it/different/ from the location of the main register area (if you
722 have specified one - remember that if you provide no <tt/register/
723 setting, the default register area is the
724 working directory of the server and indexing processes).
726 The following excerpt from a <tt/zebra.cfg/ file shows one example of
727 a setup that configures both the main register location and the shadow
728 file area. Note that two directories or partitions have been set aside
729 for the shadow file area. You can specify any number of directories
730 for each of the file areas, but remember that there should be no
731 overlaps between the directories used for the main registers and the
732 shadow files, respectively.
737 shadow: /scratch1:100M /scratch2:200M
740 When shadow files are enabled, an extra command is available at the
741 <tt/zebraidx/ command line. In order to make changes to the system
742 take effect for the users, you'll have to submit a
743 &dquot;commit&dquot; command after a (sequence of) update
744 operation(s). You can ask the indexer to commit the changes
745 immediately after the update operation:
748 $ zebraidx update /d1/records update /d2/more-records commit
751 Or you can execute multiple updates before committing the changes:
754 $ zebraidx -g books update /d1/records update /d2/more-records
755 $ zebraidx -g fun update /d3/fun-records
759 If one of the update operations above had been interrupted, the commit
760 operation on the last line would fail: <tt/zebraidx/ will not let you
761 commit changes that would destroy the running register. You'll have to
762 rerun all of the update operations since your last commit operation,
763 before you can commit the new changes.
765 Similarly, if the commit operation fails, <tt/zebraidx/ will not let
766 you start a new update operation before you have successfully repeated
767 the commit operation. The server processes will keep accessing the
768 shadow files rather than the (possibly damaged) blocks of the main
769 register files until the commit operation has successfully completed.
771 You should be aware that update operations may take slightly longer
772 when the shadow register system is enabled, since more file access
773 operations are involved. Further, while the disk space required for
774 the shadow register data is modest for a small update operation, you
775 may prefer to disable the system if you are adding a very large number
776 of records to an already very large database (we use the terms
777 <it/large/ and <it/modest/ very loosely here, since every
778 application will have a different perception of size). To update the system
779 without the use of the the shadow files, simply run <tt/zebraidx/ with
780 the <tt/-n/ option (note that you do not have to execute the
781 <bf/commit/ command of <tt/zebraidx/ when you temporarily disable the
782 use of the shadow registers in this fashion. Note also that, just as
783 when the shadow registers are not enabled, server processes will be
784 barred from accessing the main register while the update procedure
787 <sect>Running the Maintenance Interface (zebraidx)
790 The following is a complete reference to the command line interface to
791 the <tt/zebraidx/ application.
795 $ zebraidx [options] command [directory] ...
799 <tag>-t <it/type/</tag>Update all files as <it/type/. Currently, the
800 types supported are <tt/text/ and <tt/grs/<it/.subtype/. If no
801 <it/subtype/ is provided for the GRS (General Record Structure) type,
802 the canonical input format is assumed (see section <ref
803 id="local-representation" name="Local Representation">). Generally, it
804 is probably advisable to specify the record types in the
805 <tt/zebra.cfg/ file (see section <ref id="record-types" name="Record
806 Types">), to avoid confusion at subsequent updates.
808 <tag>-c <it/config-file/</tag>Read the configuration file
809 <it/config-file/ instead of <tt/zebra.cfg/.
811 <tag>-g <it/group/</tag>Update the files according to the group
812 settings for <it/group/ (see section <ref id="configuration-file"
813 name="The Zebra Configuration File">).
815 <tag>-d <it/database/</tag>The records located should be associated
816 with the database name <it/database/ for access through the Z39.50
819 <tag>-m <it/mbytes/</tag>Use <it/mbytes/ of megabytes before flushing
820 keys to background storage. This setting affects performance when
821 updating large databases.
823 <tag>-n</tag>Disable the use of shadow registers for this operation
824 (see section <ref id="shadow-registers" name="Robust Updating - Using
827 <tag>-s</tag>Show analysis of the indexing process. The maintenance
828 program works in a read-only mode and doesn't change the state
829 of the index. This options is very useful when you wish to test a
832 <tag>-V</tag>Show Zebra version.
834 <tag>-v <it/level/</tag>Set the log level to <it/level/. <it/level/
835 should be one of <tt/none/, <tt/debug/, and <tt/all/.
841 <tag>Update <it/directory/</tag>Update the register with the files
842 contained in <it/directory/. If no directory is provided, a list of
843 files is read from <tt/stdin/. See section <ref
844 id="administrating" name="Administrating Zebra">.
846 <tag>Delete <it/directory/</tag>Remove the records corresponding to
847 the files found under <it/directory/ from the register.
849 <tag/Commit/Write the changes resulting from the last <bf/update/
850 commands to the register. This command is only available if the use of
851 shadow register files is enabled (see section <ref
852 id="shadow-registers" name="Robust Updating - Using Shadow
857 <sect>The Z39.50 Server
859 <sect1>Running the Z39.50 Server (zebrasrv)
864 zebrasrv [options] [listener-address ...]
869 <tag>-a <it/APDU file/</tag> Specify a file for dumping PDUs (for diagnostic purposes).
870 The special name &dquot;-&dquot; sends output to <tt/stderr/.
872 <tag>-c <it/config-file/</tag> Read configuration information from <it/config-file/. The default configuration is <tt>./zebra.cfg</tt>.
874 <tag/-S/Don't fork on connection requests. This can be useful for
875 symbolic-level debugging. The server can only accept a single
876 connection in this mode.
878 <tag/-s/Use the SR protocol.
880 <tag/-z/Use the Z39.50 protocol (default). These two options complement
881 eachother. You can use both multiple times on the same command
882 line, between listener-specifications (see below). This way, you
883 can set up the server to listen for connections in both protocols
884 concurrently, on different local ports.
886 <tag>-l <it/logfile/</tag>Specify an output file for the diagnostic
887 messages. The default is to write this information to <tt/stderr/.
889 <tag>-v <it/log-level/</tag>The log level. Use a comma-separated list of members of the set
890 {fatal,debug,warn,log,all,none}.
892 <tag>-u <it/username/</tag>Set user ID. Sets the real UID of the server process to that of the
893 given <it/username/. It's useful if you aren't comfortable with having the
894 server run as root, but you need to start it as such to bind a
897 <tag>-w <it/working-directory/</tag>Change working directory.
899 <tag>-i</tag>Run under the Internet superserver, <tt/inetd/. Make
900 sure you use the logfile option <tt/-l/ in conjunction with this
901 mode and specify the <tt/-l/ option before any other options.
903 <tag>-t <it/timeout/</tag>Set the idle session timeout (default 60 minutes).
905 <tag>-k <it/kilobytes/</tag>Set the (approximate) maximum size of
906 present response messages. Default is 1024 Kb (1 Mb).
909 A <it/listener-address/ consists of a transport mode followed by a
910 colon (:) followed by a listener address. The transport mode is
911 either <tt/osi/ or <tt/tcp/.
913 For TCP, an address has the form
916 hostname | IP-number [: portnumber]
919 The port number defaults to 210 (standard Z39.50 port).
921 For OSI (only available if the server is compiled with XTI/mOSI
922 support enabled), the address form is
925 [t-selector /] hostname | IP-number [: portnumber]
928 The transport selector is given as a string of hex digits (with an even
929 number of digits). The default port number is 102 (RFC1006 port).
937 osi:0402/dbserver.osiworld.com:3000
941 In both cases, the special hostname &dquot;@&dquot; is mapped to
942 the address INADDR_ANY, which causes the server to listen on any local
943 interface. To start the server listening on the registered ports for
944 Z39.50 and SR over OSI/RFC1006, and to drop root privileges once the
945 ports are bound, execute the server like this (from a root shell):
948 zebrasrv -u daemon tcp:@ -s osi:@
951 You can replace <tt/daemon/ with another user, eg. your own account, or
952 a dedicated IR server account.
954 The default behavior for <tt/zebrasrv/ is to establish a single TCP/IP
955 listener, for the Z39.50 protocol, on port 9999.
957 <sect1>Z39.50 Protocol Support and Behavior
959 <sect2>Initialization
962 During initialization, the server will negotiate to version 3 of the
963 Z39.50 protocol, and the option bits for Search, Present, Scan,
964 NamedResultSets, and concurrentOperations will be set, if requested by
965 the client. The maximum PDU size is negotiated down to a maximum of
968 <sect2>Search<label id="search">
971 The supported query type are 1 and 101. All operators are currently
972 supported with the restriction that only proximity units of type "word" are
973 supported for the proximity operator.
974 Queries can be arbitrarily complex.
975 Named result sets are supported, and result sets can be used as operands
977 Searches may span multiple databases.
979 The server has full support for piggy-backed present requests (see
980 also the following section).
982 <bf/Use/ attributes are interpreted according to the attribute sets which
983 have been loaded in the <tt/zebra.cfg/ file, and are matched against
984 specific fields as specified in the <tt/.abs/ file which describes the
985 profile of the records which have been loaded. If no <bf/Use/
986 attribute is provided, a default of Bib-1 <bf/Any/ is assumed.
988 If a <bf/Structure/ attribute of <bf/Phrase/ is used in conjunction with a
989 <bf/Completeness/ attribute of <bf/Complete (Sub)field/, the term is
990 matched against the contents of the phrase (long word) register, if one
991 exists for the given <bf/Use/ attribute.
992 A phrase register is created for those fields in the <tt/.abs/
993 file that contains a <tt/p/-specifier.
995 If <bf/Structure/=<bf/Phrase/ is used in conjunction with
996 <bf/Incomplete Field/ - the default value for <bf/Completeness/, the
997 search is directed against the normal word registers, but if the term
998 contains multiple words, the term will only match if all of the words
999 are found immediately adjacent, and in the given order.
1000 The word search is performed on those fields that are indexed as
1001 type <tt/w/ in the <tt/.abs/ file.
1003 If the <bf/Structure/ attribute is <bf/Word List/,
1004 <bf/Free-form Text/, or <bf/Document Text/, the term is treated as a
1005 natural-language, relevance-ranked query.
1006 This search type uses the word register, i.e. those fields
1007 that are indexed as type <tt/w/ in the <tt/.abs/ file.
1009 If the <bf/Structure/ attribute is <bf/Numeric String/ the
1010 term is treated as an integer. The search is performed on those
1011 fields that are indexed as type <tt/n/ in the <tt/.abs/ file.
1013 If the <bf/Structure/ attribute is <bf/URx/ the
1014 term is treated as a URX (URL) entity. The search is performed on those
1015 fields that are indexed as type <tt/u/ in the <tt/.abs/ file.
1017 If the <bf/Structure/ attribute is <bf/Local Number/ the
1018 term is treated as native Zebra Record Identifier.
1020 If the <bf/Relation/ attribute is <bf/Equals/ (default), the term is
1021 matched in a normal fashion (modulo truncation and processing of
1022 individual words, if required). If <bf/Relation/ is <bf/Less Than/,
1023 <bf/Less Than or Equal/, <bf/Greater than/, or <bf/Greater than or
1024 Equal/, the term is assumed to be numerical, and a standard regular
1025 expression is constructed to match the given expression. If
1026 <bf/Relation/ is <bf/Relevance/, the standard natural-language query
1027 processor is invoked.
1029 For the <bf/Truncation/ attribute, <bf/No Truncation/ is the default.
1030 <bf/Left Truncation/ is not supported. <bf/Process #/ is supported, as
1031 is <bf/Regxp-1/. <bf/Regxp-2/ enables the fault-tolerant (fuzzy)
1032 search. As a default, a single error (deletion, insertion,
1033 replacement) is accepted when terms are matched against the register
1036 <sect3>Regular expressions
1039 Each term in a query is interpreted as a regular expression if
1040 the truncation value is either <bf/Regxp-1/ (102) or <bf/Regxp-2/ (103).
1041 Both query types follow the same syntax with the operands:
1043 <tag/x/ Matches the character <it/x/.
1044 <tag/./ Matches any character.
1045 <tag><tt/[/..<tt/]/</tag> Matches the set of characters specified;
1046 such as <tt/[abc]/ or <tt/[a-c]/.
1050 <tag/x*/ Matches <it/x/ zero or more times. Priority: high.
1051 <tag/x+/ Matches <it/x/ one or more times. Priority: high.
1052 <tag/x?/ Matches <it/x/ once or twice. Priority: high.
1053 <tag/xy/ Matches <it/x/, then <it/y/. Priority: medium.
1054 <tag/x|y/ Matches either <it/x/ or <it/y/. Priority: low.
1056 The order of evaluation may be changed by using parentheses.
1058 If the first character of the <bf/Regxp-2/ query is a plus character
1059 (<tt/+/) it marks the beginning of a section with non-standard
1060 specifiers. The next plus character marks the end of the section.
1061 Currently Zebra only supports one specifier, the error tolerance,
1062 which consists one digit.
1064 Since the plus operator is normally a suffix operator the addition to
1065 the query syntax doesn't violate the syntax for standard regular
1068 <sect3>Query examples
1071 Phrase search for <bf/information retrieval/ in the title-register:
1073 @attr 1=4 "information retrieval"
1076 Ranked search for the same thing:
1078 @attr 1=4 @attr 2=102 "Information retrieval"
1081 Phrase search with a regular expression:
1083 @attr 1=4 @attr 5=102 "informat.* retrieval"
1086 Ranked search with a regular expression:
1088 @attr 1=4 @attr 5=102 @attr 2=102 "informat.* retrieval"
1091 In the GILS schema (<tt/gils.abs/), the west-bounding-coordinate is
1092 indexed as type <tt/n/, and is therefore searched by specifying
1093 <bf/structure/=<bf/Numeric String/.
1094 To match all those records with west-bounding-coordinate greater
1095 than -114 we use the following query:
1097 @attr 4=109 @attr 2=5 @attr gils 1=2038 -114
1102 The present facility is supported in a standard fashion. The requested
1103 record syntax is matched against the ones supported by the profile of
1104 each record retrieved. If no record syntax is given, SUTRS is the
1105 default. The requested element set name, again, is matched against any
1106 provided by the relevant record profiles.
1111 The attribute combinations provided with the TermListAndStartPoint are
1112 processed in the same way as operands in a query (see above).
1113 Currently, only the term and the globalOccurrences are returned with
1114 the TermInfo structure.
1119 Z39.50 specifies three diffent types of sort criterias.
1120 Of these Zebra supports the attribute specification type in which
1121 case the use attribute specifies the "Sort register".
1122 Sort registers are created for those fields that are of type "sort" in
1123 the default.idx file.
1124 The corresponding character mapping file in default.idx specifies the
1125 ordinal of each character used in the actual sort.
1127 Z39.50 allows the client to specify sorting on one or more input
1128 result sets and one output result set.
1129 Zebra supports sorting on one result set only which may or may not
1130 be the same as the output result set.
1135 If a Close PDU is received, the server will respond with a Close PDU
1136 with reason=FINISHED, no matter which protocol version was negotiated
1137 during initialization. If the protocol version is 3 or more, the
1138 server will generate a Close PDU under certain circumstances,
1139 including a session timeout (60 minutes by default), and certain kinds of
1140 protocol errors. Once a Close PDU has been sent, the protocol
1141 association is considered broken, and the transport connection will be
1142 closed immediately upon receipt of further data, or following a short
1145 <sect>The Record Model
1148 The Zebra system is designed to support a wide range of data management
1149 applications. The system can be configured to handle virtually any
1150 kind of structured data. Each record in the system is associated with
1151 a <it/record schema/ which lends context to the data elements of the
1152 record. Any number of record schema can coexist in the system.
1153 Although it may be wise to use only a single schema within
1154 one database, the system poses no such restrictions.
1156 The record model described in this chapter applies to the fundamental,
1158 record type <tt>grs</tt> as introduced in
1159 section <ref id="record-types" name="Record Types">.
1161 Records pass through three different states during processing in the
1165 <item>When records are accessed by the system, they are represented
1166 in their local, or native format. This might be SGML or HTML files,
1167 News or Mail archives, MARC records. If the system doesn't already
1168 know how to read the type of data you need to store, you can set up an
1169 input filter by preparing conversion rules based on regular
1170 expressions and possibly augmented by a flexible scripting language (Tcl). The input filter
1171 produces as output an internal representation:
1173 <item>When records are processed by the system, they are represented
1174 in a tree-structure, constructed by tagged data elements hanging off a
1175 root node. The tagged elements may contain data or yet more tagged
1176 elements in a recursive structure. The system performs various
1177 actions on this tree structure (indexing, element selection, schema
1180 <item>Before transmitting records to the client, they are first
1181 converted from the internal structure to a form suitable for exchange
1182 over the network - according to the Z39.50 standard.
1185 <sect1>Local Representation<label id="local-representation">
1188 As mentioned earlier, Zebra places few restrictions on the type of
1189 data that you can index and manage. Generally, whatever the form of
1190 the data, it is parsed by an input filter specific to that format, and
1191 turned into an internal structure that Zebra knows how to handle. This
1192 process takes place whenever the record is accessed - for indexing and
1196 The RecordType parameter in the <tt/zebra.cfg/ file, or the <tt/-t/
1197 option to the indexer tells Zebra how to process input records. Two
1198 basic types of processing are available - raw text and structured
1199 data. Raw text is just that, and it is selected by providing the
1200 argument <bf/text/ to Zebra. Structured records are all handled
1201 internally using the basic mechanisms described in the subsequent
1202 sections. Zebra can read structured records in many different formats.
1203 How this is done is governed by additional parameters after the
1204 &dquot;grs&dquot; keyboard, separated by &dquot;.&dquot; characters.
1206 Three basic subtypes to the <bf/grs/ type are currently available:
1209 <tag>grs.sgml</tag>This is the canonical input format —
1210 described below. It is a simple SGML-like syntax.
1212 <tag>grs.regx.<it/filter/</tag>This enables a user-supplied input
1213 filter. The mechanisms of these filters are described below.
1215 <tag>grs.marc.<it/abstract syntax/</tag>This allows Zebra to read
1216 records in the ISO2709 (MARC) encoding standard. In this case, the
1217 last paramemeter <it/abstract syntax/ names the .abs file (see below)
1218 which describes the specific MARC structure of the input record as
1219 well as the indexing rules.
1222 <sect2>Canonical Input Format
1225 Although input data can take any form, it is sometimes useful to
1226 describe the record processing capabilities of the system in terms of
1227 a single, canonical input format that gives access to the full
1228 spectrum of structure and flexibility in the system. In Zebra, this
1229 canonical format is an &dquot;SGML-like&dquot; syntax.
1231 To use the canonical format specify <tt>grs.sgml</tt> as the record
1234 Consider a record describing an information resource (such a record is
1235 sometimes known as a <it/locator record/). It might contain a field
1236 describing the distributor of the information resource, which might in
1237 turn be partitioned into various fields providing details about the
1238 distributor, like this:
1242 <Name> USGS/WRD &etago;Name>
1243 <Organization> USGS/WRD &etago;Organization>
1245 U.S. GEOLOGICAL SURVEY, 505 MARQUETTE, NW
1246 &etago;Street-Address>
1247 <City> ALBUQUERQUE &etago;City>
1248 <State> NM &etago;State>
1249 <Zip-Code> 87102 &etago;Zip-Code>
1250 <Country> USA &etago;Country>
1251 <Telephone> (505) 766-5560 &etago;Telephone>
1255 <it>NOTE: The indentation used above is used to illustrate how Zebra
1256 interprets the markup. The indentation, in itself, has no
1257 significance to the parser for the canonical input format, which
1258 discards superfluous whitespace.</it>
1260 The keywords surrounded by <...> are <it/tags/, while the
1261 sections of text in between are the <it/data elements/. A data element
1262 is characterized by its location in the tree that is made up by the
1263 nested elements. Each element is terminated by a closing tag -
1264 beginning with <tt/&etago;/, and containing the same symbolic tag-name as
1265 the corresponding opening tag. The general closing tag - <tt/&etago;>/ -
1266 terminates the element started by the last opening tag. The
1267 structuring of elements is significant. The element <bf/Telephone/,
1268 for instance, may be indexed and presented to the client differently,
1269 depending on whether it appears inside the <bf/Distributor/ element,
1270 or some other, structured data element such a <bf/Supplier/ element.
1275 The first tag in a record describes the root node of the tree that
1276 makes up the total record. In the canonical input format, the root tag
1277 should contain the name of the schema that lends context to the
1278 elements of the record (see section <ref id="internal-representation"
1279 name="Internal Representation">). The following is a GILS record that
1280 contains only a single element (strictly speaking, that makes it an
1281 illegal GILS record, since the GILS profile includes several mandatory
1282 elements - Zebra does not validate the contents of a record against
1283 the Z39.50 profile, however - it merely attempts to match up elements
1284 of a local representation with the given schema):
1288 <title>Zen and the Art of Motorcycle Maintenance&etago;title>
1295 Zebra allows you to provide individual data elements in a number of
1296 <it/variant forms/. Examples of variant forms are textual data
1297 elements which might appear in different languages, and images which
1298 may appear in different formats or layouts. The variant system in
1300 essentially a representation of the variant mechanism of
1303 The following is an example of a title element which occurs in two
1304 different languages.
1308 <var lang lang "eng">
1309 Zen and the Art of Motorcycle Maintenance&etago;>
1310 <var lang lang "dan">
1311 Zen og Kunsten at Vedligeholde en Motorcykel&etago;>
1315 The syntax of the <it/variant element/ is <tt><<bf/var/ <it/class
1316 type value/></tt>. The available values for the <it/class/ and
1317 <it/type/ fields are given by the variant set that is associated with the
1318 current schema (see section <ref id="variant-set" name="Variant Set
1321 Variant elements are terminated by the general end-tag &etago;>, by
1322 the variant end-tag &etago;var>, by the appearance of another variant
1323 tag with the same <it/class/ and <it/value/ settings, or by the
1324 appearance of another, normal tag. In other words, the end-tags for
1325 the variants used in the example above could have been saved.
1327 Variant elements can be nested. The element
1331 <var lang lang "eng"><var body iana "text/plain">
1332 Zen and the Art of Motorcycle Maintenance
1336 Associates two variant components to the variant list for the title
1339 Given the nesting rules described above, we could write
1343 <var body iana "text/plain>
1344 <var lang lang "eng">
1345 Zen and the Art of Motorcycle Maintenance
1346 <var lang lang "dan">
1347 Zen og Kunsten at Vedligeholde en Motorcykel
1351 The title element above comes in two variants. Both have the IANA body
1352 type &dquot;text/plain&dquot;, but one is in English, and the other in
1353 Danish. The client, using the element selection mechanism of Z39.50,
1354 can retrieve information about the available variant forms of data
1355 elements, or it can select specific variants based on the requirements
1358 <sect2>Input Filters
1361 In order to handle general input formats, Zebra allows the
1362 operator to define filters which read individual records in their native format
1363 and produce an internal representation that the system can
1366 Input filters are ASCII files, generally with the suffix <tt/.flt/.
1367 The system looks for the files in the directories given in the
1368 <bf/profilePath/ setting in the <tt/zebra.cfg/ files. The record type
1369 for the filter is <tt>grs.regx.</tt><it>filter-filename</it>
1370 (fundamental type <tt>grs</tt>, file read type <tt>regx</tt>, argument
1371 <it>filter-filename</it>).
1373 Generally, an input filter consists of a sequence of rules, where each
1374 rule consists of a sequence of expressions, followed by an action. The
1375 expressions are evaluated against the contents of the input record,
1376 and the actions normally contribute to the generation of an internal
1377 representation of the record.
1379 An expression can be either of the following:
1382 <tag/INIT/The action associated with this expression is evaluated
1383 exactly once in the lifetime of the application, before any records
1384 are read. It can be used in conjunction with an action that
1385 initializes tables or other resources that are used in the processing
1388 <tag/BEGIN/Matches the beginning of the record. It can be used to
1389 initialize variables, etc. Typically, the <bf/BEGIN/ rule is also used
1390 to establish the root node of the record.
1392 <tag/END/Matches the end of the record - when all of the contents
1393 of the record has been processed.
1395 <tag>/pattern/</tag>Matches a string of characters from the input
1398 <tag/BODY/This keyword may only be used between two patterns. It
1399 matches everything between (not including) those patterns.
1401 <tag/FINISH/THe expression asssociated with this pattern is evaluated
1402 once, before the application terminates. It can be used to release
1403 system resources - typically ones allocated in the <bf/INIT/ step.
1407 An action is surrounded by curly braces ({...}), and consists of a
1408 sequence of statements. Statements may be separated by newlines or
1409 semicolons (;). Within actions, the strings that matched the
1410 expressions immediately preceding the action can be referred to as
1411 $0, $1, $2, etc.
1413 The available statements are:
1417 <tag>begin <it/type [parameter ... ]/</tag>Begin a new
1418 data element. The type is one of the following:
1420 <tag/record/Begin a new record. The followingparameter should be the
1421 name of the schema that describes the structure of the record, eg.
1422 <tt/gils/ or <tt/wais/ (see below). The <tt/begin record/ call should
1424 any other use of the <bf/begin/ statement.
1426 <tag/element/Begin a new tagged element. The parameter is the
1427 name of the tag. If the tag is not matched anywhere in the tagsets
1428 referenced by the current schema, it is treated as a local string
1431 <tag/variant/Begin a new node in a variant tree. The parameters are
1432 <it/class type value/.
1436 <tag/data/Create a data element. The concatenated arguments make
1437 up the value of the data element. The option <tt/-text/ signals that
1438 the layout (whitespace) of the data should be retained for
1439 transmission. The option <tt/-element/ <it/tag/ wraps the data up in
1440 the <it/tag/. The use of the <tt/-element/ option is equivalent to
1441 preceding the command with a <bf/begin element/ command, and following
1442 it with the <bf/end/ command.
1444 <tag>end <it/[type]/</tag>Close a tagged element. If no parameter is given,
1445 the last element on the stack is terminated. The first parameter, if
1446 any, is a type name, similar to the <bf/begin/ statement. For the
1447 <bf/element/ type, a tag name can be provided to terminate a specific tag.
1451 The following input filter reads a Usenet news file, producing a
1452 record in the WAIS schema. Note that the body of a news posting is
1453 separated from the list of headers by a blank line (or rather a
1454 sequence of two newline characters.
1457 BEGIN { begin record wais }
1459 /^From:/ BODY /$/ { data -element name $1 }
1460 /^Subject:/ BODY /$/ { data -element title $1 }
1461 /^Date:/ BODY /$/ { data -element lastModified $1 }
1463 begin element bodyOfDisplay
1464 begin variant body iana "text/plain"
1470 If Zebra is compiled with support for Tcl (Tool Command Language)
1471 enabled, the statements described above are supplemented with a complete
1472 scripting environment, including control structures (conditional
1473 expressions and loop constructs), and powerful string manipulation
1474 mechanisms for modifying the elements of a record. Tcl is a popular
1475 scripting environment, with several tutorials available both online
1478 <it>NOTE: Tcl support is not currently available, but will be
1479 included with one of the next alpha or beta releases.</it>
1481 <it>NOTE: Variant support is not currently available in the input
1482 filter, but will be included with one of the next alpha or beta
1485 <sect1>Internal Representation<label id="internal-representation">
1488 When records are manipulated by the system, they're represented in a
1489 tree-structure, with data elements at the leaf nodes, and tags or
1490 variant components at the non-leaf nodes. The root-node identifies the
1491 schema that lends context to the tagging and structuring of the
1492 record. Imagine a simple record, consisting of a 'title' element and
1493 an 'author' element:
1496 TITLE "Zen and the Art of Motorcycle Maintenance"
1498 AUTHOR "Robert Pirsig"
1501 A slightly more complex record would have the author element consist
1502 of two elements, a surname and a first name:
1505 TITLE "Zen and the Art of Motorcycle Maintenance"
1512 The root of the record will refer to the record schema that describes
1513 the structuring of this particular record. The schema defines the
1514 element tags (TITLE, FIRST-NAME, etc.) that may occur in the record, as
1515 well as the structuring (SURNAME should appear below AUTHOR, etc.). In
1516 addition, the schema establishes element set names that are used by
1517 the client to request a subset of the elements of a given record. The
1518 schema may also establish rules for converting the record to a
1519 different schema, by stating, for each element, a mapping to a
1522 <sect2>Tagged Elements
1525 A data element is characterized by its tag, and its position in the
1526 structure of the record. For instance, while the tag &dquot;telephone
1527 number&dquot; may be used different places in a record, we may need to
1528 distinguish between these occurrences, both for searching and
1529 presentation purposes. For instance, while the phone numbers for the
1530 &dquot;customer&dquot; and the &dquot;service provider&dquot; are both
1531 representatives for the same type of resource (a telephone number), it
1532 is essential that they be kept separate. The record schema provides
1533 the structure of the record, and names each data element (defined by
1534 the sequence of tags - the tag path - by which the element can be
1535 reached from the root of the record).
1540 The children of a tag node may be either more tag nodes, a data node
1541 (possibly accompanied by tag nodes),
1542 or a tree of variant nodes. The children of variant nodes are either
1543 more variant nodes or a data node (possibly accompanied by more
1544 variant nodes). Each leaf node, which is normally a
1545 data node, corresponds to a <it/variant form/ of the tagged element
1546 identified by the tag which parents the variant tree. The following
1547 title element occurs in two different languages:
1550 VARIANT LANG=ENG "War and Peace"
1552 VARIANT LANG=DAN "Krig og Fred"
1555 Which of the two elements are transmitted to the client by the server
1556 depends on the specifications provided by the client, if any.
1558 In practice, each variant node is associated with a triple of class,
1559 type, value, corresponding to the variant mechanism of Z39.50.
1561 <sect2>Data Elements
1564 Data nodes have no children (they are always leaf nodes in the record
1567 <it>NOTE: Documentation needs extension here about types of nodes - numerical,
1568 textual, etc., plus the various types of inclusion notes.</it>
1570 <sect1>Configuring Your Data Model<label id="data-model">
1573 The following sections describe the configuration files that govern
1574 the internal management of data records. The system searches for the files
1575 in the directories specified by the <bf/profilePath/ setting in the
1576 <tt/zebra.cfg/ file.
1578 <sect2>The Abstract Syntax
1581 The abstract syntax definition (also known as an Abstract Record
1582 Structure, or ARS) is the focal point of the
1583 record schema description. For a given schema, the ABS file may state any
1584 or all of the following:
1587 <item>The object identifier of the Z39.50 schema associated
1588 with the ARS, so that it can be referred to by the client.
1590 <item>The attribute set (which can possibly be a compound of multiple
1591 sets) which applies in the profile. This is used when indexing and
1592 searching the records belonging to the given profile.
1594 <item>The Tag set (again, this can consist of several different sets).
1595 This is used when reading the records from a file, to recognize the
1596 different tags, and when transmitting the record to the client -
1597 mapping the tags to their numerical representation, if they are
1600 <item>The variant set which is used in the profile. This provides a
1601 vocabulary for specifying the <it/forms/ of data that appear inside
1604 <item>Element set names, which are a shorthand way for the client to
1605 ask for a subset of the data elements contained in a record. Element
1606 set names, in the retrieval module, are mapped to <it/element
1607 specifications/, which contain information equivalent to the
1608 <it/Espec-1/ syntax of Z39.50.
1610 <item>Map tables, which may specify mappings to <it/other/ database
1611 profiles, if desired.
1613 <item>Possibly, a set of rules describing the mapping of elements to a
1614 MARC representation.
1616 <item>A list of element descriptions (this is the actual ARS of the
1617 schema, in Z39.50 terms), which lists the ways in which the various
1618 tags can be used and organized hierarchically.
1621 Several of the entries above simply refer to other files, which
1622 describe the given objects.
1624 <sect2>The Configuration Files
1627 This section describes the syntax and use of the various tables which
1628 are used by the retrieval module.
1630 The number of different file types may appear daunting at first, but
1631 each type corresponds fairly clearly to a single aspect of the Z39.50
1632 retrieval facilities. Further, the average database administrator,
1633 who is simply reusing an existing profile for which tables already
1634 exist, shouldn't have to worry too much about the contents of these tables.
1636 Generally, the files are simple ASCII files, which can be maintained
1637 using any text editor. Blank lines, and lines beginning with a (#) are
1638 ignored. Any characters on a line followed by a (#) are also ignored.
1640 lines contain <it/directives/, which provide some setting or value
1641 to the system. Generally, settings are characterized by a single
1642 keyword, identifying the setting, followed by a number of parameters.
1643 Some settings are repeatable (r), while others may occur only once in a
1644 file. Some settings are optional (o), whicle others again are
1647 <sect2>The Abstract Syntax (.abs) Files
1650 The name of this file type is slightly misleading in Z39.50 terms,
1651 since, apart from the actual abstract syntax of the profile, it also
1652 includes most of the other definitions that go into a database
1655 When a record in the canonical, SGML-like format is read from a file
1656 or from the database, the first tag of the file should reference the
1657 profile that governs the layout of the record. If the first tag of the
1658 record is, say, <tt><gils></tt>, the system will look for the profile
1659 definition in the file <tt/gils.abs/. Profile definitions are cached,
1660 so they only have to be read once during the lifespan of the current
1663 When writing your own input filters, the <bf/record-begin/ command
1664 introduces the profile, and should always be called first thing when
1665 introducing a new record.
1667 The file may contain the following directives:
1670 <tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or
1671 description for the profile. Mostly useful for diagnostic purposes.
1673 <tag>reference <it/OID-name/</tag> (m) The reference name of the OID for
1674 the profile. The reference names can be found in the <bf/util/
1677 <tag>attset <it/filename/</tag> (m) The attribute set that is used for
1678 indexing and searching records belonging to this profile.
1680 <tag>tagset <it/filename/</tag> (o) The tag set (if any) that describe
1681 that fields of the records.
1683 <tag>varset <it/filename/</tag> (o) The variant set used in the profile.
1685 <tag>maptab <it/filename/</tag> (o,r) This points to a
1686 conversion table that might be used if the client asks for the record
1687 in a different schema from the native one.
1689 <tag>marc <it/filename/</tag> (o) Points to a file containing parameters
1690 for representing the record contents in the ISO2709 syntax. Read the
1691 description of the MARC representation facility below.
1693 <tag>esetname <it/name filename/</tag> (o,r) Associates the
1694 given element set name with an element selection file. If an (@) is
1695 given in place of the filename, this corresponds to a null mapping for
1696 the given element set name.
1698 <tag>any <it/tags/</tag> (o) This directive specifies a list of
1699 attributes which should be appended to the attribute list given for each
1700 element. The effect is to make every single element in the abstract
1701 syntax searchable by way of the given attributes. This directive
1702 provides an efficient way of supporting free-text searching across all
1703 elements. However, it does increase the size of the index
1704 significantly. The attributes can be qualified with a structure, as in
1705 the <bf/elm/ directive below.
1707 <tag>elm <it/path name attributes/</tag> (o,r) Adds an element
1708 to the abstract record syntax of the schema. The <it/path/ follows the
1709 syntax which is suggested by the Z39.50 document - that is, a sequence
1710 of tags separated by slashes (/). Each tag is given as a
1711 comma-separated pair of tag type and -value surrounded by parenthesis.
1712 The <it/name/ is the name of the element, and the <it/attributes/
1713 specifies which attributes to use when indexing the element in a
1714 comma-separated list. A ! in
1715 place of the attribute name is equivalent to specifying an attribute
1716 name identical to the element name. A - in place of the attribute name
1717 specifies that no indexing is to take place for the given element. The
1718 attributes can be qualified with <it/field types/ to specify which
1719 character set should govern the indexing procedure for that field. The
1720 same data element may be indexed into several different fields, using
1721 different character set definitions. See the section
1722 <ref id="field structure and character sets"
1723 name="Field Structure and Character Sets">.
1724 The default field type is &dquot;w&dquot; for
1729 NOTE: The mechanism for controlling indexing is not adequate for
1730 complex databases, and will probably be moved into a separate
1731 configuration table eventually.
1734 The following is an excerpt from the abstract syntax file for the GILS
1739 reference GILS-schema
1744 maptab gils-usmarc.map
1748 esetname VARIANT gils-variant.est # for WAIS-compliance
1749 esetname B gils-b.est
1750 esetname G gils-g.est
1755 elm (1,14) localControlNumber Local-number
1756 elm (1,16) dateOfLastModification Date/time-last-modified
1757 elm (2,1) Title w:!,p:!
1758 elm (4,1) controlIdentifier Identifier-standard
1759 elm (2,6) abstract Abstract
1760 elm (4,51) purpose !
1761 elm (4,52) originator -
1762 elm (4,53) accessConstraints !
1763 elm (4,54) useConstraints !
1764 elm (4,70) availability -
1765 elm (4,70)/(4,90) distributor -
1766 elm (4,70)/(4,90)/(2,7) distributorName !
1767 elm (4,70)/(4,90)/(2,10 distributorOrganization !
1768 elm (4,70)/(4,90)/(4,2) distributorStreetAddress !
1769 elm (4,70)/(4,90)/(4,3) distributorCity !
1772 <sect2>The Attribute Set (.att) Files<label id="attset-files">
1775 This file type describes the <bf/Use/ elements of an attribute set.
1776 It contains the following directives.
1780 <tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or
1781 description for the attribute set. Mostly useful for diagnostic purposes.
1783 <tag>reference <it/OID-name/</tag> (m) The reference name of the OID for
1784 the attribute set. The reference names can be found in the <bf/util/
1787 <tag>ordinal <it/integer/</tag> (m) This value will be used to represent the
1788 attribute set in the index. Care should be taken that each attribute
1789 set has a unique ordinal value.
1791 <tag>include <it/filename/</tag> (o,r) This directive is used to
1792 include another attribute set as a part of the current one. This is
1793 used when a new attribute set is defined as an extension to another
1794 set. For instance, many new attribute sets are defined as extensions
1795 to the <bf/bib-1/ set. This is an important feature of the retrieval
1796 system of Z39.50, as it ensures the highest possible level of
1797 interoperability, as those access points of your database which are
1798 derived from the external set (say, bib-1) can be used even by clients
1799 who are unaware of the new set.
1801 <tag>att <it/att-value att-name [local-value]/</tag> (o,r) This
1802 repeatable directive introduces a new attribute to the set. The
1803 attribute value is stored in the index (unless a <it/local-value/ is
1804 given, in which case this is stored). The name is used to refer to the
1805 attribute from the <it/abstract syntax/. </descrip>
1807 This is an excerpt from the GILS attribute set definition. Notice how
1808 the file describing the <it/bib-1/ attribute set is referenced.
1812 reference GILS-attset
1816 att 2001 distributorName
1817 att 2002 indexTermsControlled
1819 att 2004 accessConstraints
1820 att 2005 useConstraints
1823 <sect2>The Tag Set (.tag) Files
1826 This file type defines the tagset of the profile, possibly by
1827 referencing other tag sets (most tag sets, for instance, will include
1828 tagsetG and tagsetM from the Z39.50 specification. The file may
1829 contain the following directives.
1832 <tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or
1833 description for the tag set. Mostly useful for diagnostic purposes.
1835 <tag>reference <it/OID-name/</tag> (o) The reference name of the OID for
1836 the tag set. The reference names can be found in the <bf/util/
1837 module of <bf/YAZ/. The directive is optional, since not all tag sets
1838 are registered outside of their schema.
1840 <tag>type <it/integer/</tag> (m) The type number of the tagset within the schema
1841 profile (note: this specification really should belong to the .abs
1842 file. This will be fixed in a future release).
1844 <tag>include <it/filename/</tag> (o,r) This directive is used
1845 to include the definitions of other tag sets into the current one.
1847 <tag>tag <it/number names type/</tag> (o,r) Introduces a new
1848 tag to the set. The <it/number/ is the tag number as used in the protocol
1849 (there is currently no mechanism for specifying string tags at this
1850 point, but this would be quick work to add). The <it/names/ parameter
1851 is a list of names by which the tag should be recognized in the input
1852 file format. The names should be separated by slashes (/). The
1853 <it/type/ is th recommended datatype of the tag. It should be one of
1861 <item>generalizedtime
1869 The following is an excerpt from the TagsetG definition file.
1878 tag 3 publicationPlace string
1879 tag 4 publicationDate string
1880 tag 5 documentId string
1881 tag 6 abstract string
1883 tag 8 date generalizedtime
1884 tag 9 bodyOfDisplay string
1885 tag 10 organization string
1888 <sect2>The Variant Set (.var) Files<label id="variant-set">
1891 The variant set file is a straightforward representation of the
1892 variant set definitions associated with the protocol. At present, only
1893 the <it/Variant-1/ set is known.
1895 These are the directives allowed in the file.
1898 <tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or
1899 description for the variant set. Mostly useful for diagnostic purposes.
1901 <tag>reference <it/OID-name/</tag> (o) The reference name of the OID for
1902 the variant set, if one is required. The reference names can be found
1903 in the <bf/util/ module of <bf/YAZ/.
1905 <tag>class <it/integer class-name/</tag> (m,r) Introduces a new
1906 class to the variant set.
1908 <tag>type <it/integer type-name datatype/</tag> (m,r) Addes a
1909 new type to the current class (the one introduced by the most recent
1910 <bf/class/ directive). The type names belong to the same name space as
1911 the one used in the tag set definition file.
1914 The following is an excerpt from the file describing the variant set
1923 type 1 variantId octetstring
1928 type 2 z39.50 string
1932 <sect2>The Element Set (.est) Files
1935 The element set specification files describe a selection of a subset
1936 of the elements of a database record. The element selection mechanism
1937 is equivalent to the one supplied by the <it/Espec-1/ syntax of the
1938 Z39.50 specification. In fact, the internal representation of an
1939 element set specification is identical to the <it/Espec-1/ structure,
1940 and we'll refer you to the description of that structure for most of
1941 the detailed semantics of the directives below.
1944 NOTE: Not all of the Espec-1 functionality has been implemented yet.
1945 The fields that are mentioned below all work as expected, unless
1949 The directives available in the element set file are as follows:
1952 <tag>defaultVariantSetId <it/OID-name/</tag> (o) If variants are used in
1953 the following, this should provide the name of the variantset used
1954 (it's not currently possible to specify a different set in the
1955 individual variant request). In almost all cases (certainly all
1956 profiles known to us), the name <tt/Variant-1/ should be given here.
1958 <tag>defaultVariantRequest <it/variant-request/</tag> (o) This directive
1959 provides a default variant request for
1960 use when the individual element requests (see below) do not contain a
1961 variant request. Variant requests consist of a blank-separated list of
1962 variant components. A variant compont is a comma-separated,
1963 parenthesized triple of variant class, type, and value (the two former
1964 values being represented as integers). The value can currently only be
1965 entered as a string (this will change to depend on the definition of
1966 the variant in question). The special value (@) is interpreted as a
1967 null value, however.
1969 <tag>simpleElement <it/path ['variant' variant-request]/</tag>
1970 (o,r) This corresponds to a simple element request in <it/Espec-1/. The
1971 path consists of a sequence of tag-selectors, where each of these can
1975 <item>A simple tag, consisting of a comma-separated type-value pair in
1976 parenthesis, possibly followed by a colon (:) followed by an
1977 occurrences-specification (see below). The tag-value can be a number
1978 or a string. If the first character is an apostrophe ('), this forces
1979 the value to be interpreted as a string, even if it appears to be numerical.
1981 <item>A WildThing, represented as a question mark (?), possibly
1982 followed by a colon (:) followed by an occurrences specification (see
1985 <item>A WildPath, represented as an asterisk (*). Note that the last
1986 element of the path should not be a wildPath (wildpaths don't work in
1990 The occurrences-specification can be either the string <tt/all/, the
1991 string <tt/last/, or an explicit value-range. The value-range is
1992 represented as an integer (the starting point), possibly followed by a
1993 plus (+) and a second integer (the number of elements, default being
1996 The variant-request has the same syntax as the defaultVariantRequest
1997 above. Note that it may sometimes be useful to give an empty variant
1998 request, simply to disable the default for a specific set of fields
1999 (we aren't certain if this is proper <it/Espec-1/, but it works in
2000 this implementation).
2003 The following is an example of an element specification belonging to
2007 simpleelement (1,10)
2008 simpleelement (1,12)
2010 simpleelement (1,14)
2012 simpleelement (4,52)
2015 <sect2>The Schema Mapping (.map) Files<label id="schema-mapping">
2018 Sometimes, the client might want to receive a database record in
2019 a schema that differs from the native schema of the record. For
2020 instance, a client might only know how to process WAIS records, while
2021 the database record is represented in a more specific schema, such as
2022 GILS. In this module, a mapping of data to one of the MARC formats is
2023 also thought of as a schema mapping (mapping the elements of the
2024 record into fields consistent with the given MARC specification, prior
2025 to actually converting the data to the ISO2709). This use of the
2026 object identifier for USMARC as a schema identifier represents an
2027 overloading of the OID which might not be entirely proper. However,
2028 it represents the dual role of schema and record syntax which
2029 is assumed by the MARC family in Z39.50.
2032 NOTE: The schema-mapping functions are so far limited to a
2033 straightforward mapping of elements. This should be extended with
2034 mechanisms for conversions of the element contents, and conditional
2035 mappings of elements based on the record contents.
2038 These are the directives of the schema mapping file format:
2041 <tag>targetName <it/name/</tag> (m) A symbolic name for the target schema
2042 of the table. Useful mostly for diagnostic purposes.
2044 <tag>targetRef <it/OID-name/</tag> (m) An OID name for the target schema.
2045 This is used, for instance, by a server receiving a request to present
2046 a record in a different schema from the native one. The name, again,
2047 is found in the <bf/oid/ module of <bf/YAZ/.
2049 <tag>map <it/element-name target-path/</tag> (o,r) Adds
2050 an element mapping rule to the table.
2053 <sect2>The MARC (ISO2709) Representation (.mar) Files
2056 This file provides rules for representing a record in the ISO2709
2057 format. The rules pertain mostly to the values of the constant-length
2058 header of the record.
2060 <it>NOTE: This will be described better. We're in the process of
2061 re-evaluating and most likely changing the way that MARC records are
2062 handled by the system.</it>
2064 <sect2>Field Structure and Character Sets
2065 <label id="field structure and character sets">
2068 In order to provide a flexible approach to national character set
2069 handling, Zebra allows the administrator to configure the set up the
2070 system to handle any 8-bit character set — including sets that
2071 require multi-octet diacritics or other multi-octet characters. The
2072 definition of a character set includes a specification of the
2073 permissible values, their sort order (this affects the display in the
2074 SCAN function), and relationships between upper- and lowercase
2075 characters. Finally, the definition includes the specification of
2076 space characters for the set.
2078 The operator can define different character sets for different fields,
2079 typical examples being standard text fields, numerical fields, and
2080 special-purpose fields such as WWW-style linkages (URx).
2082 The field types, and hence character sets, are associated with data
2083 elements by the .abs files (see above). The file <tt/default.idx/
2084 provides the association between field type codes (as used in the .abs
2085 files) and the character map files (with the .chr suffix). The format
2086 of the .idx file is as follows
2089 <tag>index <it/field type code/</tag>This directive introduces a new
2090 search index code. The argument is a one-character code to be used in the
2091 .abs files to select this particular index type. An index, roughly,
2092 corresponds to a particular structure attribute during search. Refer
2093 to section <ref id="search" name="Search">.
2095 <tag>sort <it/field code type/</tag>This directive introduces a
2096 sort index. The argument is a one-character code to be used in the
2097 .abs fie to select this particular index type. The corresponding
2098 use attribute must be used in the sort request to refer to this
2099 particular sort index. The corresponding character map (see below)
2100 is used in the sort process.
2102 <tag>completeness <it/boolean/</tag>This directive enables or disables
2103 complete field indexing. The value of the <it/boolean/ should be 0
2104 (disable) or 1. If completeness is enabled, the index entry will
2105 contain the complete contents of the field (up to a limit), with words
2106 (non-space characters) separated by single space characters
2107 (normalized to &dquot; &dquot; on display). When completeness is
2108 disabled, each word is indexed as a separate entry. Complete subfield
2109 indexing is most useful for fields which are typically browsed (eg.
2110 titles, authors, or subjects), or instances where a match on a
2111 complete subfield is essential (eg. exact title searching). For fields
2112 where completeness is disabled, the search engine will interpret a
2113 search containing space characters as a word proximity search.
2115 <tag>charmap <it/filename/</tag> This is the filename of the character
2116 map to be used for this index for field type.
2119 The contents of the character map files are structured as follows:
2122 <tag>lowercase <it/value-set/</tag>This directive introduces the basic
2123 value set of the field type. The format is an ordered list (without
2124 spaces) of the characters which may occur in &dquot;words&dquot; of
2125 the given type. The order of the entries in the list determines the
2126 sort order of the index. In addition to single characters, the
2127 following combinations are legal:
2130 <item>Backslashes may be used to introduce three-digit octal, or
2131 two-digit hex representations of single characters (preceded by <tt/x/).
2132 In addition, the combinations
2133 \\, \\r, \\n, \\t, \\s (space — remember that real space-characters
2134 may ot occur in the value definition), and \\ are recognised,
2135 with their usual interpretation.
2137 <item>Curly braces {} may be used to enclose ranges of single
2138 characters (possibly using the escape convention described in the
2139 preceding point), eg. {a-z} to entroduce the standard range of ASCII
2140 characters. Note that the interpretation of such a range depends on
2141 the concrete representation in your local, physical character set.
2143 <item>Paranthesises () may be used to enclose multi-byte characters -
2144 eg. diacritics or special national combinations (eg. Spanish
2145 &dquot;ll&dquot;). When found in the input stream (or a search term),
2146 these characters are viewed and sorted as a single character, with a
2147 sorting value depending on the position of the group in the value
2151 <tag>uppercase <it/value-set/</tag>This directive introduces the
2152 upper-case equivalencis to the value set (if any). The number and
2153 order of the entries in the list should be the same as in the
2154 <tt/lowercase/ directive.
2156 <tag>space <it/value-set/</tag>This directive introduces the character
2157 which separate words in the input stream. Depending on the
2158 completeness mode of the field in question, these characters either
2159 terminate an index entry, or delimit individual &dquot;words&dquot; in
2160 the input stream. The order of the elements is not significant —
2161 otherwise the representation is the same as for the <tt/upercase/ and
2162 <tt/lowercase/ directives.
2164 <tag>map <it/value-set/ <it/target/</tag>This directive introduces a
2165 mapping between each of the members of the value-set on the left to
2166 the character on the right. The character on the right must occur in
2167 the value set (the <tt/lowercase/ directive) of the character set, but
2168 it may be a paranthesis-enclosed multi-octet character. This directive
2169 may be used to map diacritics to their base characters, or to map
2170 HTML-style character-representations to their natural form, etc.
2173 <sect1>Exchange Formats
2176 Converting records from the internal structure to en exchange format
2177 is largely an automatic process. Currently, the following exchange
2178 formats are supported:
2181 <item>GRS-1. The internal representation is based on GRS-1, so the
2182 conversion here is straightforward. The system will create
2183 applied variant and supported variant lists as required, if a record
2184 contains variant information.
2186 <item>SUTRS. Again, the mapping is fairly straighforward. Indentation
2187 is used to show the hierarchical structure of the record. All
2188 &dquot;GRS&dquot; type records support both the GRS-1 and SUTRS
2191 <item>ISO2709-based formats (USMARC, etc.). Only records with a
2192 two-level structure (corresponding to fields and subfields) can be
2193 directly mapped to ISO2709. For records with a different structuring
2194 (eg., GILS), the representation in a structure like USMARC involves a
2195 schema-mapping (see section <ref id="schema-mapping" name="Schema
2196 Mapping">), to an &dquot;implied&dquot; USMARC schema (implied,
2197 because there is no formal schema which specifies the use of the
2198 USMARC fields outside of ISO2709). The resultant, two-level record is
2199 then mapped directly from the internal representation to ISO2709. See
2200 the GILS schema definition files for a detailed example of this
2203 <item>Explain. This representation is only available for records
2204 belonging to the Explain schema.
2206 <item>Summary. This ASN-1 based structure is only available for records
2207 belonging to the Summary schema - or schema which provide a mapping
2208 to this schema (see the description of the schema mapping facility
2211 <item>SOIF. Support for this syntax is experimental, and is currently
2212 keyed to a private Index Data OID (1.2.840.10003.5.1000.81.2). All
2213 abstract syntaxes can be mapped to the SOIF format, although nested
2214 elements are represented by concatenation of the tag names at each
2222 Copyright © 1995-1998 Index Data.
2224 All rights reserved.
2226 Use and redistribution in source or binary form, with or without
2227 modification, of any or all of this software and documentation is
2228 permitted, provided that the following conditions are met:
2230 1. This copyright and permission notice appear with all copies of the
2231 software and its documentation. Notices of copyright or attribution
2232 which appear at the beginning of any file must remain unchanged.
2234 2. The names of Index Data or the individual authors may not be used to
2235 endorse or promote products derived from this software without specific
2236 prior written permission.
2238 3. Source code or binary versions of this software and its
2239 documentation may be used freely in not-for-profit applications. For
2240 profit applications - such as providing for-pay database services,
2241 marketing a product based in whole or in part on this software or its
2242 documentation, or generally distributing this software or its
2243 documentation under a different license - requires a commercial
2244 license from Index Data. The software may be installed and used for
2245 evaluation purposes in conjunction with a commercial application for a
2246 trial period of no more than 60 days.
2248 THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT WARRANTY OF ANY KIND,
2249 EXPRESS, IMPLIED, OR OTHERWISE, INCLUDING WITHOUT LIMITATION, ANY
2250 WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
2251 IN NO EVENT SHALL INDEX DATA BE LIABLE FOR ANY SPECIAL, INCIDENTAL,
2252 INDIRECT OR CONSEQUENTIAL DAMAGES OF ANY KIND, OR ANY DAMAGES
2253 WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER OR
2254 NOT ADVISED OF THE POSSIBILITY OF DAMAGE, AND ON ANY THEORY OF
2255 LIABILITY, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE
2258 <sect>About Index Data and the Zebra Server
2261 Index Data is a consulting and software-development enterprise that
2262 specialises in library and information management systems. Our
2263 interests and expertise span a broad range of related fields, and one
2264 of our primary, long-term objectives is the development of a powerful
2265 information management
2266 system with open network interfaces and hypermedia capabilities.
2268 We make this software available free of charge for not-for-profit
2269 purposes, as a service to the networking community, and to further
2270 the development and use of quality software for open network
2273 If you like this software, and would like to use all or part of it in
2274 a commercial product, or to provide a commercial database service,
2275 please contact us to discuss the details. We'll be happy to answer
2276 questions about the software, and about our services in general. If
2277 you have specific requirements to the software, we'll be glad to offer
2278 our advice - and if you need to adapt the software to a special
2279 purpose, our consulting services and expert knowledge of the software
2280 is available to you at favorable rates.
2285 DK-2200 Copenhagen N
2290 Phone: +45 3536 3672
2292 Email: info@indexdata.dk
2295 The <it>Random House College Dictionary</it>, 1975 edition
2296 offers this definition of the
2297 word &dquot;Zebra&dquot;:
2300 Zebra, n., any of several horselike, African mammals of the genus Equus,
2301 having a characteristic pattern of black or dark-brown stripes on
2302 a whitish background.