1 <!doctype linuxdoc system>
4 $Id: zebra.sgml,v 1.35 1997-02-19 16:22:26 adam Exp $
8 <title>Zebra Server - Administrators's Guide and Reference
9 <author><htmlurl url="http://www.indexdata.dk/" name="Index Data">, <tt><htmlurl url="mailto:info@index.ping.dk" name="info@index.ping.dk"></>
10 <date>$Revision: 1.35 $
12 The Zebra information server combines a versatile fielded/free-text
13 search engine with a Z39.50-1995 frontend to provide a powerful and flexible
14 information management system. This document explains the procedure for
15 installing and configuring the system, and outlines the possibilities
16 for managing data and providing Z39.50
17 services with the software.
27 The Zebra system is a fielded free-text indexing and retrieval engine with a
28 Z39.50 frontend. You can use any commercial or freeware Z39.50 client
29 to access data stored in Zebra.
31 The Zebra server is our first step towards the development of a fully
32 configurable, open information system. Eventually, it will be paired
33 off with a powerful Z39.50 client to support complex information
34 management tasks within almost any application domain. We're making
35 the server available now because it's no fun to be in the open
36 information retrieval business all by yourself. We want to allow
37 people with interesting data to make their things
38 available in interesting ways, without having to start out
39 by implementing yet another protocol stack from scratch.
41 This document is an introduction to the Zebra system. It will tell you
42 how to compile the software, and how to prepare your first database.
43 It also explains how the server can be configured to give you the
44 functionality that you need.
46 If you find the software interesting, you should join the support
47 mailing-list by sending Email to <tt/zebra-request@index.ping.dk/.
52 This is a list of some of the most important features of the
58 Supports updating - records can be added and deleted without
59 rebuilding the index from scratch.
60 The update procedure is tolerant to crashes or hard interrupts
61 during register updating - registers can be reconstructed following a crash.
62 Registers can be safely updated even while users are accessing the server.
65 Supports large databases - files for indices, etc. can be
66 automatically partitioned over multiple disks.
69 Supports arbitrarily complex records - base input format is an
70 SGML-like syntax which allows nested (structured) data elements, as
71 well as variant forms of data.
74 Supports random storage formats. A system of input filters driven by
75 regular expressions allows you to easily process most ASCII-based
79 Supports boolean queries as well as relevance-ranking (free-text)
80 searching. Right truncation and masking in terms are supported, as
81 well as full regular expressions.
84 Supports multiple concrete syntaxes
85 for record exchange (depending on the configuration): GRS-1, SUTRS,
86 ISO2709 (*MARC). Records can be mapped between record syntaxes and
90 Supports approximate matching in registers (ie. spelling mistakes,
99 Protocol facilities: Init, Search, Retrieve, Browse.
102 Piggy-backed presents are honored in the search-request.
105 Named result sets are supported.
108 Easily configured to support different application profiles, with
109 tables for attribute sets, tag sets, and abstract syntaxes.
110 Additional tables control facilities such as element mappings to
111 different schema (eg., GILS-to-USMARC).
114 Complex composition specifications using Espec-1 are partially
115 supported (simple element requests only).
118 Element Set Names are defined using the Espec-1 capability of the
119 system, and are given in configuration files as simple element
120 requests (and possibly variant requests).
123 Some variant support (not fully implemented yet).
126 Using the YAZ toolkit for the protocol implementation, the
127 server can utilise a plug-in XTI/mOSI implementation (not included) to
128 provide SR services over an OSI stack, as well as Z39.50 over TCP/IP.
137 This is an alfa-release of the software, to allow you to look at
138 it - try it out, and assess whether it can be of use to you. We expect
139 this version to be followed by a succession of beta-releases until we
140 arrive at a stable first version.
142 These are some of the plans that we have for the software in the near
143 and far future, approximately ordered after their relative importance.
145 asterisk will be implemented before the
151 *Complete the support for variants. Finalize support for the WAIS
152 retrieval methodology.
155 *Finalize the data element <it/include/ facility to support multimedia
156 data elements in records.
159 *Port the system to Windows NT.
162 Add more sophisticated relevance ranking mechanisms. Add support for soundex
163 and stemming. Add relevance <it/feedback/ support.
169 Add support for very large records by implementing segmentation and/or
173 Support the Item Update extended service of the protocol.
176 We want to add a management system that allows you to
177 control your databases and configuration tables from a graphical
178 interface. We'll probably use Tcl/Tk to stay platform-independent.
182 Programmers thrive on user feedback. If you are interested in a facility that
183 you don't see mentioned here, or if there's something you think we
184 could do better, please drop us a mail. If you think it's all really
185 neat, you're welcome to drop us a line saying that, too. You'll find
186 contact info at the end of this file.
188 <sect>Compiling the software
191 An ANSI C compiler is required to compile the Zebra
192 server system — <tt/gcc/ works fine if your own system doesn't
193 provide an adequate compiler.
195 Unpack the distribution archive. In some cases, you may want to edit
196 the top-level <tt/Makefile/, eg. to select a different C compiler, or
197 to specify machine-specific libraries in the <bf/ELIBS/ variable.
199 When you are done editing the <tt>Makefile</tt> type:
204 If successful, two executables have been created in the sub-directory
207 <tag><tt>zebrasrv</tt></tag> The Z39.50 server and search engine.
208 <tag><tt>zebraidx</tt></tag> The administrative tool for the search index.
213 In this section, we will test the system by indexing a small set of sample
214 GILS records that are included with the software distribution. Go to the
215 <tt>test</tt> subdirectory of the distribution archive. There you will
217 file named <tt>zebra.cfg</tt> with the following contents:
219 # Where are the YAZ tables located.
220 profilePath: ../../yaz/tab ../tab
222 # Files that describe the attribute sets supported.
226 # Name of character map file.
230 Now, edit the file and set <tt>profilePath</tt> to the path of the
231 YAZ profile tables (sub directory <tt>tab</tt> of the YAZ distribution
234 The 48 test records are located in the sub directory <tt>records</tt>.
235 To index these, type:
237 $ ../index/zebraidx -t grs.sgml update records
240 In the command above the option <tt>-t</tt> specified the record
241 type — in this case <tt>grs.sgml</tt>. The word <tt>update</tt> followed
242 by a directory root updates all files below that directory node.
244 If your indexing command was successful, you are now ready to
245 fire up a server. To start a server on port 2100, type:
247 $ ../index/zebrasrv tcp:@:2100
250 The Zebra index that you have just created has a single database
251 named <tt/Default/. The database contains records structured according to
252 the GILS profile, and the server will
253 return records in either either USMARC, GRS-1, or SUTRS depending
254 on what your client asks
257 To test the server, you can use any Z39.50 client (1992 or later). For
258 instance, you can use the demo client that comes with YAZ: Just cd to
259 the <tt/client/ subdirectory of the YAZ distribution and type:
262 $ client tcp:localhost:2100
265 When the client has connected, you can type:
272 The default retrieval syntax for the client is USMARC. To try other
273 formats for the same record, try:
284 <it>NOTE: You may notice that more fields are returned when your
285 client requests SUTRS or GRS-1 records. When retrieving GILS records,
286 this is normal - not all of the GILS data elements have mappings in
287 the USMARC record format.</it>
289 If you've made it this far, there's a good chance that
290 you've got through the compilation OK.
292 <sect>Administrating Zebra<label id="administrating">
295 Unlike many simpler retrieval systems, Zebra supports safe, incremental
296 updates to an existing index.
298 Normally, when Zebra modifies the index it reads a number of records
300 Depending on your specifications and on the contents of each record
301 one the following events take place for each record:
303 <tag>Insert</tag> The record is indexed as if it never occurred
304 before. Either the Zebra system doesn't know how to identify the record or
305 Zebra can identify the record but didn't find it to be already indexed.
306 <tag>Modify</tag> The record has already been indexed. In this case
307 either the contents of the record or the location (file) of the record
308 indicates that it has been indexed before.
309 <tag>Delete</tag> The record is deleted from the index. As in the
310 update-case it must be able to identify the record.
313 Please note that in both the modify- and delete- case the Zebra
314 indexer must be able to generate a unique key that identifies the record in
315 question (more on this below).
317 To administrate the Zebra retrieval system, you run the
318 <tt>zebraidx</tt> program. This program supports a number of options
319 which are preceded by a minus, and a few commands (not preceded by
322 Both the Zebra administrative tool and the Z39.50 server share a
323 set of index files and a global configuration file. The
324 name of the configuration file defaults to <tt>zebra.cfg</tt>.
325 The configuration file includes specifications on how to index
326 various kinds of records and where the other configuration files
327 are located. <tt>zebrasrv</tt> and <tt>zebraidx</tt> <em>must</em>
328 be run in the directory where the configuration file lives unless you
329 indicate the location of the configuration file by option
332 <sect1>Record Types<label id="record-types">
334 Indexing is a per-record process, in which
335 either insert/modify/delete will occur. Before a record is indexed
336 search keys are extracted from whatever might be the layout the
337 original record (sgml,html,text, etc..). The Zebra system
338 currently only supports SGML-like, structured records and unstructured text
340 To specify a particular extraction process, use either the
341 command line option <tt>-t</tt> or specify a
342 <tt>recordType</tt> setting in the configuration file.
344 <sect1>The Zebra Configuration File<label id="configuration-file">
346 The Zebra configuration file, read by <tt>zebraidx</tt> and
347 <tt>zebrasrv</tt> defaults to <tt>zebra.cfg</tt> unless specified
348 by <tt>-c</tt> option.
350 You can edit the configuration file with a normal text editor.
351 Parameter names and values are seperated by colons in the file. Lines
352 starting with a hash sign (<tt/#/) are treated as comments.
354 If you manage different sets of records that share common
355 characteristics, you can organize the configuration settings for each
356 type into &dquot;groups&dquot;.
357 When <tt>zebraidx</tt> is run and you wish to address a given group
358 you specify the group name with the <tt>-g</tt> option. In this case
359 settings that have the group name as their prefix will be used
360 by <tt>zebraidx</tt>. If no <tt/-g/ option is specified, the settings
361 with no prefix are used.
363 In the configuration file, the group name is placed before the option
364 name itself, separated by a dot (.). For instance, to set the record type
365 for group <tt/public/ to <tt/grs.sgml/ (the SGML-like format for structured
366 records) you would write:
369 public.recordType: grs.sgml
372 To set the default value of the record type to <tt/text/ write:
378 The available configuration settings are summarized below. They will be
379 explained further in the following sections.
382 <tag><it>group</it>.recordType[<it>.name</it>]</tag>
383 Specifies how records with the file extension <it>name</it> should
384 be handled by the indexer. This option may also be specified
385 as a command line option (<tt>-t</tt>). Note that if you do not
386 specify a <it/name/, the setting applies to all files. In general,
387 the record type specifier consists of the elements (each
388 element separated by dot), <it>fundamental-type</it>,
389 <it>file-read-type</it> and arguments. Currently, two
390 fundamental types exist, <tt>text</tt> and <tt>grs</tt>.
391 <tag><it>group</it>.recordId</tag>
392 Specifies how the records are to be identified when updated. See
393 section <ref id="locating-records" name="Locating Records">.
394 <tag><it>group</it>.database</tag>
395 Specifies the Z39.50 database name.
396 <tag><it>group</it>.storeKeys</tag>
397 Specifies whether key information should be saved for a given
398 group of records. If you plan to update/delete this type of
399 records later this should be specified as 1; otherwise it
400 should be 0 (default), to save register space. See section
401 <ref id="file-ids" name="Indexing With File Record IDs">.
402 <tag><it>group</it>.storeData</tag>
403 Specifies whether the records should be stored internally
404 in the Zebra system files. If you want to maintain the raw records yourself,
405 this option should be false (0). If you want Zebra to take care of the records
406 for you, it should be true(1).
408 Specifies the location of the various register files that Zebra uses
409 to represent your databases. See section
410 <ref id="register-location" name="Register Location">.
412 Enables the <it/safe update/ facility of Zebra, and tells the system
413 where to place the required, temporary files. See section
414 <ref id="shadow-registers" name="Safe Updating - Using Shadow Registers">.
416 Directory in which various lock files are stored.
418 Directory in which temporary files used during zebraidx' update
421 Specifies the directory that the server uses for temporary result sets.
422 If not specified <tt>/tmp</tt> will be used.
423 <tag>profilePath</tag>
424 Specifies the location of profile specification files.
426 Specifies the filename(s) of attribute set files for use in
427 searching. At least the Bib-1 set should be loaded (<tt/bib1.att/).
428 The <tt/profilePath/ setting is used to look for the specified files.
429 See section <ref id="attset-files" name="The Attribute Set Files">
431 Specifies the filename of a character mapping. Zebra uses the path,
432 <tt>profilePath</tt>, to locate this file.
434 Specifies size of internal memory to use for the zebraidx program. The
435 amount is given in megabytes - default is 4 (4 MB).
437 <sect1>Locating Records<label id="locating-records">
439 The default behaviour of the Zebra system is to reference the
440 records from their original location, i.e. where they were found when you
441 ran <tt/zebraidx/. That is, when a client wishes to retrieve a record
442 following a search operation, the files are accessed from the place
443 where you originally put them - if you remove the files (without
444 running <tt/zebraidx/ again, the client will receive a diagnostic
447 If your input files are not permanent - for example if you retrieve
448 your records from an outside source, or if they were temporarily
449 mounted on a CD-ROM drive,
450 you may want Zebra to make an internal copy of them. To do this,
451 you specify 1 (true) in the <tt>storeData</tt> setting. When
452 the Z39.50 server retrieves the records they will be read from the
453 internal file structures of the system.
455 <sect1>Indexing with no Record IDs (Simple Indexing)
458 If you have a set of records that is not expected to change over time
459 you may can build your database without record IDs.
460 This indexing method uses less space than the other methods and
463 To use this method, you simply don't provide the <tt>recordId</tt> entry
464 for the group of files that you index. To add a set of records you use
465 <tt>zebraidx</tt> with the <tt>update</tt> command. The
466 <tt>update</tt> command will always add all of the records that it
467 encounters to the index - whether they have already been indexed or
468 not. If the set of indexed files change, you should delete all of the
469 index files, and build a new index from scratch.
471 Consider a system in which you have a group of text files called
472 <tt>simple</tt>. That group of records should belong to a Z39.50 database
473 called <tt>textbase</tt>. The following <tt/zebra.cfg/ file will suffice:
476 profilePath: /usr/local/yaz
478 simple.recordType: text
479 simple.database: textbase
482 Since the existing records in an index can not be addressed by their
483 IDs, it is impossible to delete or modify records when using this method.
485 <sect1>Indexing with File Record IDs<label id="file-ids">
488 If you have a set of files that regularly change over time: Old files
489 are deleted, new ones are added, or existing files are modified, you
490 can benefit from using the <it/file ID/ indexing methodology. Examples
491 of this type of database might include an index of WWW resources, or a
492 USENET news spool area. Briefly speaking, the file key methodology
493 uses the directory paths of the individual records as a unique
494 identifier for each record. To perform indexing of a directory with
495 file keys, again, you specify the top-level directory after the
496 <tt>update</tt> command. The command will recursively traverse the
497 directories and compare each one with whatever have been indexed before in
498 that same directory. If a file is new (not in the previous version of
499 the directory) it is inserted into the registers; if a file was
500 already indexed and it has been modified since the last update,
501 the index is also modified; if a file has been removed since the last
502 visit, it is deleted from the index.
504 The resulting system is easy to administrate. To delete a record you
505 simply have to delete the corresponding file (say, with the <tt/rm/
506 command). And to add records you create new files (or directories with
507 files). For your changes to take effect in the register you must run
508 <tt>zebraidx update</tt> with the same directory root again. This mode
509 of operation requires more disk space than simpler indexing methods,
510 but it makes it easier for you to keep the index in sync with a
511 frequently changing set of data. If you combine this system with the
512 <it/safe update/ facility (see below), you never have to take your
513 server offline for maintenance or register updating purposes.
515 To enable indexing with pathname IDs, you must specify <tt>file</tt> as
516 the value of <tt>recordId</tt> in the configuration file. In addition,
517 you should set <tt>storeKeys</tt> to <tt>1</tt>, since the Zebra
518 indexer must save additional information about the contents of each record
519 in order to modify the indices correctly at a later time.
521 For example, to update records of group <tt>esdd</tt> located below
522 <tt>/data1/records/</tt> you should type:
524 $ zebraidx -g esdd update /data1/records
527 The corresponding configuration file includes:
534 <em>Important note: You cannot start out with a group of records with simple
535 indexing (no record IDs as in the previous section) and then later
536 enable file record Ids. Zebra must know from the first time that you
538 the files should be indexed with file record IDs.
541 You cannot explicitly delete records when using this method (using the
542 <bf/delete/ command to <tt/zebraidx/. Instead
543 you have to delete the files from the file system (or move them to a
545 and then run <tt>zebraidx</tt> with the <bf/update/ command.
547 <sect1>Indexing with General Record IDs
549 When using this method you construct an (almost) arbritrary, internal
550 record key based on the contents of the record itself and other system
551 information. If you have a group of records that explicitly associates
552 an ID with each record, this method is convenient. For example, the
553 record format may contain a title or a ID-number - unique within the group.
554 In either case you specify the Z39.50 attribute set and use-attribute
555 location in which this information is stored, and the system looks at
556 that field to determine the identity of the record.
558 As before, the record ID is defined by the <tt>recordId</tt> setting
559 in the configuration file. The value of the record ID specification
560 consists of one or more tokens separated by whitespace. The resulting
562 represented in the index by concatenating the tokens and separating them by
565 There are three kinds of tokens:
567 <tag>Internal record info</tag> The token refers to a key that is
568 extracted from the record. The syntax of this token is
569 <tt/(/ <em/set/ <tt/,/ <em/use/ <tt/)/, where <em/set/ is the
570 attribute set ordinal number and <em/use/ is the use value of the attribute.
571 <tag>System variable</tag> The system variables are preceded by
572 <verb>$</verb> and immediately followed by the system variable name, which
575 <tag>group</tag> Group name.
576 <tag>database</tag> Current database specified.
577 <tag>type</tag> Record type.
579 <tag>Constant string</tag> A string used as part of the ID — surrounded
580 by single- or double quotes.
583 For instance, the sample GILS records that come with the Zebra
584 distribution contain a
586 in the Control-Identifier field. This field is mapped to the Bib-1
587 use attribute 1007. To use this field as a record id, specify
588 <tt>(1,1007)</tt> as the value of the <tt>recordId</tt> in the
589 configuration file. If you have other record types that uses
590 the same field for a different purpose, you might add the record type (or group or database name)
591 to the record id of the gils records as well, to prevent matches
592 with other types of records. In this case the recordId might be
595 gils.recordId: $type (1,1007)
598 (see section <ref id="data-model" name="Configuring Your Data Model">
599 for details of how the mapping between elements of your records and
600 searchable attributes is established).
602 As for the file record ID case described in the previous section,
603 updating your system is simply a matter of running <tt>zebraidx</tt>
604 with the <tt>update</tt> command. However, the update with general
605 keys is considerably slower than with file record IDs, since all files
606 visited must be (re)read to discover their IDs.
608 As you might expect, when using the general record IDs
609 method, you can only add or modify existing records with the <tt>update</tt>
610 command. If you wish to delete records, you must use the,
611 <tt>delete</tt> command, with a directory as a parameter.
612 This will remove all records that match the files below that root
615 <sect1>Register Location<label id="register-location">
618 Normally, the index files that form dictionaries, inverted
619 files, record info, etc., are stored in the directory where you run
620 <tt>zebraidx</tt>. If you wish to store these, possibly large, files
621 somewhere else, you must add the <tt>register</tt> entry to the
622 <tt/zebra.cfg/ file. Furthermore, the Zebra system allows its file
624 span multiple file systems, which is useful for managing very large
627 The value of the <tt>register</tt> setting is a sequence of tokens.
628 Each token takes the form:
630 <em>dir</em><tt>:</tt><em>size</em>.
632 The <em>dir</em> specifies a directory in which index files will be
633 stored and the <em>size</em> specifies the maximum size of all
634 files in that directory. The Zebra indexer system fills each directory
635 in the order specified and use the next specified directories as needed.
636 The <em>size</em> is an integer followed by a qualifier
637 code, <tt>M</tt> for megabytes, <tt>k</tt> for kilobytes.
639 For instance, if you have allocated two disks for your register, and
640 the first disk is mounted
641 on <tt>/d1</tt> and has 200 Mb of free space and the
642 second, mounted on <tt>/d2</tt> has 300 Mb, you could
643 put this entry in your configuration file:
645 register: /d1:200M /d2:300M
648 Note that Zebra does not verify that the amount of space specified is
649 actually available on the directory (file system) specified - it is
650 your responsibility to ensure that enough space is available, and that
651 other applications do not attempt to use the free space. In a large production system,
652 it is recommended that you allocate one or more filesystem exclusively
653 to the Zebra register files.
655 <sect1>Safe Updating - Using Shadow Registers<label id="shadow-registers">
660 The Zebra server supports <it/updating/ of the index structures. That is,
661 you can add, modify, or remove records from databases managed by Zebra
662 without rebuilding the entire index. Since this process involves
663 modifying structured files with various references between blocks of
664 data in the files, the update process is inherently sensitive to
665 system crashes, or to process interruptions: Anything but a
666 successfully completed update process will leave the register files in
667 an unknown state, and you will essentially have no recourse but to
668 re-index everything, or to restore the register files from a backup
669 medium. Further, while the update process is active, users cannot be
670 allowed to access the system, as the contents of the register files
671 may change unpredictably.
673 You can solve these problems by enabling the shadow register system in
674 Zebra. During the updating procedure, <tt/zebraidx/ will temporarily
675 write changes to the involved files in a set of &dquot;shadow
676 files&dquot;, without modifying the files that are accessed by the
677 active server processes. If the update procedure is interrupted by a
678 system crash or a signal, you simply repeat the procedure - the
679 register files have not been changed or damaged, and the partially
680 written shadow files are automatically deleted before the new updating
683 At the end of the updating procedure (or in a separate operation, if
684 you so desire), the system enters a &dquot;commit mode&dquot;. First,
685 any active server processes are forced to access those blocks that
686 have been changed from the shadow files rather than from the main
687 register files; the unmodified blocks are still accessed at their
688 normal location (the shadow files are not a complete copy of the
689 register files - they only contain those parts that have actually been
690 modified). If the commit process is interrupted at any point during the
691 commit process, the server processes will continue to access the
692 shadow files until you can repeat the commit procedure and complete
693 the writing of data to the main register files. You can perform
694 multiple update operations to the registers before you commit the
695 changes to the system files, or you can execute the commit operation
696 at the end of each update operation. When the commit phase has
697 completed successfully, any running server processes are instructed to
698 switch their operations to the new, operational register, and the
699 temporary shadow files are deleted.
701 <sect2>How to Use Shadow Register Files
704 The first step is to allocate space on your system for the shadow
705 files. You do this by adding a <tt/shadow/ entry to the <tt/zebra.cfg/
706 file. The syntax of the <tt/shadow/ entry is exactly the same as for
707 the <tt/register/ entry (see section <ref name="Register Location"
708 id="register-location">). The location of the shadow area should be
709 <it/different/ from the location of the main register area (if you
710 have specified one - remember that if you provide no <tt/register/
711 setting, the default register area is the
712 working directory of the server and indexing processes).
714 The following excerpt from a <tt/zebra.cfg/ file shows one example of
715 a setup that configures both the main register location and the shadow
716 file area. Note that two directories or partitions have been set aside
717 for the shadow file area. You can specify any number of directories
718 for each of the file areas, but remember that there should be no
719 overlaps between the directories used for the main registers and the
720 shadow files, respectively.
725 shadow: /scratch1:100M /scratch2:200M
728 When shadow files are enabled, an extra command is available at the
729 <tt/zebraidx/ command line. In order to make changes to the system
730 take effect for the users, you'll have to submit a
731 &dquot;commit&dquot; command after a (sequence of) update
732 operation(s). You can ask the indexer to commit the changes
733 immediately after the update operation:
736 $ zebraidx update /d1/records update /d2/more-records commit
739 Or you can execute multiple updates before committing the changes:
742 $ zebraidx -g books update /d1/records update /d2/more-records
743 $ zebraidx -g fun update /d3/fun-records
747 If one of the update operations above had been interrupted, the commit
748 operation on the last line would fail: <tt/zebraidx/ will not let you
749 commit changes that would destroy the running register. You'll have to
750 rerun all of the update operations since your last commit operation,
751 before you can commit the new changes.
753 Similarly, if the commit operation fails, <tt/zebraidx/ will not let
754 you start a new update operation before you have successfully repeated
755 the commit operation. The server processes will keep accessing the
756 shadow files rather than the (possibly damaged) blocks of the main
757 register files until the commit operation has successfully completed.
759 You should be aware that update operations may take slightly longer
760 when the shadow register system is enabled, since more file access
761 operations are involved. Further, while the disk space required for
762 the shadow register data is modest for a small update operation, you
763 may prefer to disable the system if you are adding a very large number
764 of records to an already very large database (we use the terms
765 <it/large/ and <it/modest/ very loosely here, since every
766 application will have a different perception of size). To update the system
767 without the use of the the shadow files, simply run <tt/zebraidx/ with
768 the <tt/-n/ option (note that you do not have to execute the
769 <bf/commit/ command of <tt/zebraidx/ when you temporarily disable the
770 use of the shadow registers in this fashion. Note also that, just as
771 when the shadow registers are not enabled, server processes will be
772 barred from accessing the main register while the update procedure
775 <sect>Running the Maintenance Interface (zebraidx)
778 The following is a complete reference to the command line interface to
779 the <tt/zebraidx/ application.
783 $ zebraidx [options] command [directory] ...
787 <tag>-t <it/type/</tag>Update all files as <it/type/. Currently, the
788 types supported are <tt/text/ and <tt/grs/<it/.filter/. If no
789 <it/filter/ is provided for the GRS (General Record Structure) type,
790 the canonical input format is assumed (see section <ref
791 id="local-representation" name="Local Representation">). Generally, it
792 is probably advisable to specify the record types in the
793 <tt/zebra.cfg/ file (see section <ref id="record-types" name="Record Types">).
795 <tag>-c <it/config-file/</tag>Read the configuration file
796 <it/config-file/ instead of <tt/zebra.cfg/.
798 <tag>-g <it/group/</tag>Update the files according to the group
799 settings for <it/group/ (see section <ref id="configuration-file"
800 name="The Zebra Configuration File">).
802 <tag>-d <it/database/</tag>The records located should be associated
803 with the database name <it/database/ for access through the Z39.50
806 <tag>-d <it/mbytes/</tag>Use <it/mbytes/ of megabytes before flushing
807 keys to background storage. This setting affects performance when
808 updating large databases.
810 <tag>-n</tag>Disable the use of shadow registers for this operation
811 (see section <ref id="shadow-registers" name="Robust Updating - Using
814 <tag>-v <it/level/</tag>Set the log level to <it/level/. <it/level/
815 should be one of <tt/none/, <tt/debug/, and <tt/all/.
821 <tag>Update <it/directory/</tag>Update the register with the files
822 contained in <it/directory/. If no directory is provided, a list of
823 files is read from <tt/stdin/. See section <ref
824 id="administrating" name="Administrating Zebra">.
826 <tag>Delete <it/directory/</tag>Remove the records corresponding to
827 the files found under <it/directory/ from the register.
829 <tag/Commit/Write the changes resulting from the last <bf/update/
830 commands to the register. This command is only available if the use of
831 shadow register files is enabled (see section <ref
832 id="shadow-registers" name="Robust Updating - Using Shadow
837 <sect>The Z39.50 Server
839 <sect1>Running the Z39.50 Server (zebrasrv)
844 zebrasrv [options] [listener-address ...]
849 <tag>-a <it/APDU file/</tag> Specify a file for dumping PDUs (for diagnostic purposes).
850 The special name &dquot;-&dquot; sends output to <tt/stderr/.
852 <tag>-c <it/config-file/</tag> Read configuration information from <it/config-file/. The default configuration is <tt>./zebra.cfg</tt>.
854 <tag/-S/Don't fork on connection requests. This can be useful for
855 symbolic-level debugging. The server can only accept a single
856 connection in this mode.
858 <tag/-s/Use the SR protocol.
860 <tag/-z/Use the Z39.50 protocol (default). These two options complement
861 eachother. You can use both multiple times on the same command
862 line, between listener-specifications (see below). This way, you
863 can set up the server to listen for connections in both protocols
864 concurrently, on different local ports.
866 <tag>-l <it/logfile/</tag>Specify an output file for the diagnostic
867 messages. The default is to write this information to <tt/stderr/.
869 <tag>-v <it/log-level/</tag>The log level. Use a comma-separated list of members of the set
870 {fatal,debug,warn,log,all,none}.
872 <tag>-u <it/username/</tag>Set user ID. Sets the real UID of the server process to that of the
873 given <it/username/. It's useful if you aren't comfortable with having the
874 server run as root, but you need to start it as such to bind a
877 <tag>-w <it/working-directory/</tag>Change working directory.
879 <tag>-i <it/minutes/</tag>Run under the Internet superserver, <tt/inetd/.
881 <tag>-t <it/timeout/</tag>Set the idle session timeout (default 60 minutes).
883 <tag>-k <it/kilobytes/</tag>Set the (approximate) maximum size of
884 present response messages. Default is 1024 Kb (1 Mb).
887 A <it/listener-address/ consists of a transport mode followed by a
888 colon (:) followed by a listener address. The transport mode is
889 either <tt/osi/ or <tt/tcp/.
891 For TCP, an address has the form
894 hostname | IP-number [: portnumber]
897 The port number defaults to 210 (standard Z39.50 port).
899 For OSI (only available if the server is compiled with XTI/mOSI
900 support enabled), the address form is
903 [t-selector /] hostname | IP-number [: portnumber]
906 The transport selector is given as a string of hex digits (with an even
907 number of digits). The default port number is 102 (RFC1006 port).
915 osi:0402/dbserver.osiworld.com:3000
919 In both cases, the special hostname &dquot;@&dquot; is mapped to
920 the address INADDR_ANY, which causes the server to listen on any local
921 interface. To start the server listening on the registered ports for
922 Z39.50 and SR over OSI/RFC1006, and to drop root privileges once the
923 ports are bound, execute the server like this (from a root shell):
926 zebrasrv -u daemon tcp:@ -s osi:@
929 You can replace <tt/daemon/ with another user, eg. your own account, or
930 a dedicated IR server account.
932 The default behavior for <tt/zebrasrv/ is to establish a single TCP/IP
933 listener, for the Z39.50 protocol, on port 9999.
935 <sect1>Z39.50 Protocol Support and Behavior
937 <sect2>Initialization
940 During initialization, the server will negotiate to version 3 of the
941 Z39.50 protocol, and the option bits for Search, Present, Scan,
942 NamedResultSets, and concurrentOperations will be set, if requested by
943 the client. The maximum PDU size is negotiated down to a maximum of
949 The supported query type are 1 and 101. All operators are currently
950 supported except that only proximity units of type "word" are supported
951 for the proximity operator. Queries can be arbitrarily complex. Named
952 result sets are supported, and result sets can be used as operands with
953 no limitations. Searches may span multiple databases.
955 The server has full support for piggy-backed present requests (see
956 also the following section).
958 <bf/Use/ attributes are interpreted according to the attribute sets which
959 have been loaded in the <tt/zebra.cfg/ file, and are matched against
960 specific fields as specified in the <tt/.abs/ file which describes the
961 profile of the records which have been loaded. If no <bf/Use/
962 attribute is provided, a default of <bf/Any/ is assumed.
964 If a <bf/Structure/ attribute of <bf/Phrase/ is used in conjunction with a
965 <bf/Completeness/ attribute of <bf/Complete (Sub)field/, the term is
966 matched against the contents of a phrase (long word) register, if one
967 exists for the given <bf/Use/ attribute. If <bf/Structure/=<bf/Phrase/
968 is used in conjunction with <bf/Incomplete Field/ - the default value
969 for <bf/Completeness/, the search is directed against the normal word
970 registers, but if the term contains multiple words, the term will only
971 match if all of the words are found immediately adjacent, and in the
972 given order. If the <bf/Structure/ attribute is <bf/Word List/,
973 <bf/Free-form Text/, or <bf/Document Text/, the term is treated as a
974 natural-language, relevance-ranked query.
976 If the <bf/Relation/ attribute is <bf/Equals/ (default), the term is
977 matched in a normal fashion (modulo truncation and processing of
978 individual words, if required). If <bf/Relation/ is <bf/Less Than/,
979 <bf/Less Than or Equal/, <bf/Greater than/, or <bf/Greater than or
980 Equal/, the term is assumed to be numerical, and a standard regular
981 expression is constructed to match the given expression. If
982 <bf/Relation/ is <bf/Relevance/, the standard natural-language query
983 processor is invoked.
985 For the <bf/Truncation/ attribute, <bf/No Truncation/ is the default.
986 <bf/Left Truncation/ is not supported. <bf/Process #/ is supported, as
987 is <bf/Regxp-1/. <bf/Regxp-2/ enables the fault-tolerant (fuzzy)
988 search. As a default, a single error (deletion, insertion,
989 replacement) is accepted when terms are matched against the register
992 Zebra interprets queries in one the following ways:
994 <tag>1 Phrase search</tag>
995 Each token separated by white space is truncated according to the
996 value of truncation attribute. If the completeness attribute
997 is <bf/complete subfield/ the search is directed to the phrase
998 register. For other completeness attribute values the term is split
999 into tokens according to the white-space specification in the
1000 character map. Only records in which each token exists in the order
1001 specified are matched.
1002 <tag>2 Word search</tag>
1003 The token is truncated according to the value of truncation attribute.
1004 The completeness attribute is ignored.
1005 <tag>3 Ranked search</tag>
1006 Each token separated by white space is truncated according to the value
1007 of truncation attribute. The completenss attribute is ignored.
1008 <tag>4 Numeric relation</tag>
1009 The token should consist of decimal digits. The integer is matched
1010 against integers in the register according to the relation attribute.
1011 The truncation - and the completenss attribute is ignored.
1012 <tag>5 Document identifier</tag>
1013 The token consists of exactly one document identifier. The
1014 truncation - and the completeness attribute is ignored.
1017 For ranked searches the result sets are ranked and a score
1018 is associated with each record. All other result sets from the
1019 remaining four types are non-ranked.
1021 Combinations of the structure attribute and the relation attribute
1022 determine how the query is interpreted. The two following tables
1026 Structure Attribute (4)
1027 none phrase(1) word(2) word list(6)
1032 Relation <= (2) 4 4 4 4
1033 Attribute >= (4) 4 4 4 4
1042 Structure Attribute (4)
1043 free-form- document- local- string
1045 (105) (106) (107) (108)
1049 Relation <= (2) 4 4 5 4
1050 Attribute >= (4) 4 4 5 4
1058 <sect3>Regular expressions
1061 Each term in a query is interpreted as a regular expression if
1062 the truncation value is either <bf/Regxp-1/ (102) or <bf/Regxp-2/ (103).
1063 Both query types follow the same syntax with the operands:
1065 <tag/x/ Matches the character <it/x/.
1066 <tag/./ Matches any character.
1067 <tag><tt/[/..<tt/]/</tag> Matches the set of characters specified;
1068 such as <tt/[abc]/ or <tt/[a-c]/.
1072 <tag/x*/ Matches <it/x/ zero or more times. Priority: high.
1073 <tag/x+/ Matches <it/x/ one or more times. Priority: high.
1074 <tag/x?/ Matches <it/x/ once or twice. Priority: high.
1075 <tag/xy/ Matches <it/x/, then <it/y/. Priority: medium.
1076 <tag/x|y/ Matches either <it/x/ or <it/y/. Priority: low.
1078 The order of evaluation may be changed by using parentheses.
1080 If the first character of the <bf/Regxp-2/ query is a plus character
1081 (<tt/+/) it marks the beginning of a section with non-standard
1082 specifiers. The next plus character marks the end of the section.
1083 Currently Zebra only supports one specifier, the error tolerance,
1084 which consists one digit.
1086 Since the plus operator is normally a suffix operator the addition to
1087 the query syntax doesn't violate the syntax for standard regular
1090 <sect3>Query examples
1092 Phrase search for <bf/information retrieval/ in the title-register:
1094 @attr 1=4 "information retrieval"
1097 Ranked search for the same thing:
1099 @attr 1=4 @attr 2=102 "Information retrieval"
1102 Phrase search with a regular expression:
1104 @attr 1=4 @attr 5=102 "informat.* retrieval"
1107 Ranked search with a regular expression:
1109 @attr 1=4 @attr 5=102 @attr 2=102 "informat.* retrieval"
1114 The present facility is supported in a standard fashion. The requested
1115 record syntax is matched against the ones supported by the profile of
1116 each record retrieved. If no record syntax is given, SUTRS is the
1117 default. The requested element set name, again, is matched against any
1118 provided by the relevant record profiles.
1123 The attribute combinations provided with the TermListAndStartPoint are
1124 processed in the same way as operands in a query (see above).
1125 Currently, only the term and the globalOccurrences are returned with
1126 the TermInfo structure.
1131 If a Close PDU is received, the server will respond with a Close PDU
1132 with reason=FINISHED, no matter which protocol version was negotiated
1133 during initialization. If the protocol version is 3 or more, the
1134 server will generate a Close PDU under certain circumstances,
1135 including a session timeout (60 minutes by default), and certain kinds of
1136 protocol errors. Once a Close PDU has been sent, the protocol
1137 association is considered broken, and the transport connection will be
1138 closed immediately upon receipt of further data, or following a short
1141 <sect>The Record Model
1144 The Zebra system is designed to support a wide range of data management
1145 applications. The system can be configured to handle virtually any
1146 kind of structured data. Each record in the system is associated with
1147 a <it/record schema/ which lends context to the data elements of the
1148 record. Any number of record schema can coexist in the system.
1149 Although it may be wise to use only a single schema within
1150 one database, the system poses no such restrictions.
1152 The record model described in this chapter applies to the fundamental
1153 record type <tt>grs</tt> as introduced in
1154 section <ref id="record-types" name="Record Types">.
1156 Records pass through three different states during processing in the
1160 <item>When records are accessed by the system, they are represented
1161 in their local, or native format. This might be SGML or HTML files,
1162 News or Mail archives, MARC records. If the system doesn't already
1163 know how to read the type of data you need to store, you can set up an
1164 input filter by preparing conversion rules based on regular
1165 expressions and a flexible scripting language (Tcl). The input filter
1166 produces as output an internal representation:
1168 <item>When records are processed by the system, they are represented
1169 in a tree-structure, constructed by tagged data elements hanging off a
1170 root node. The tagged elements may contain data or yet more tagged
1171 elements in a recursive structure. The system performs various
1172 actions on this tree structure (indexing, element selection, schema
1175 <item>Before transmitting records to the client, they are first
1176 converted from the internal structure to a form suitable for exchange
1177 over the network - according to the Z39.50 standard.
1180 <sect1>Local Representation<label id="local-representation">
1183 As mentioned earlier, Zebra places few restrictions on the type of
1184 data that you can index and manage. Generally, whatever the form of
1185 the data, it is parsed by an input filter specific to that format, and
1186 turned into an internal structure that Zebra knows how to handle. This
1187 process takes place whenever the record is accessed - for indexing and
1190 <sect2>Canonical Input Format
1193 Although input data can take any form, it is sometimes useful to
1194 describe the record processing capabilities of the system in terms of
1195 a single, canonical input format that gives access to the full
1196 spectrum of structure and flexibility in the system. In Zebra, this
1197 canonical format is an &dquot;SGML-like&dquot; syntax.
1199 To use the canonical format specify <tt>grs.sgml</tt> as the record
1202 Consider a record describing an information resource (such a record is
1203 sometimes known as a <it/locator record/). It might contain a field
1204 describing the distributor of the information resource, which might in
1205 turn be partitioned into various fields providing details about the
1206 distributor, like this:
1210 <Name> USGS/WRD &etago;Name>
1211 <Organization> USGS/WRD &etago;Organization>
1213 U.S. GEOLOGICAL SURVEY, 505 MARQUETTE, NW
1214 &etago;Street-Address>
1215 <City> ALBUQUERQUE &etago;City>
1216 <State> NM &etago;State>
1217 <Zip-Code> 87102 &etago;Zip-Code>
1218 <Country> USA &etago;Country>
1219 <Telephone> (505) 766-5560 &etago;Telephone>
1223 <it>NOTE: The indentation used above is used to illustrate how Zebra
1224 interprets the markup. The indentation, in itself, has no
1225 significance to the parser for the canonical input format, which
1226 discards superfluous whitespace.</it>
1228 The keywords surrounded by <...> are <it/tags/, while the
1229 sections of text in between are the <it/data elements/. A data element
1230 is characterized by its location in the tree that is made up by the
1231 nested elements. Each element is terminated by a closing tag -
1232 beginning with &etago;, and containing the same symbolic tag-name as
1233 the corresponding opening tag. The general closing tag - &etago;> -
1234 terminates the element started by the last opening tag. The
1235 structuring of elements is significant. The element <bf/Telephone/,
1236 for instance, may be indexed and presented to the client differently,
1237 depending on whether it appears inside the <bf/Distributor/ element,
1238 or some other, structured data element such a <bf/Supplier/ element.
1243 The first tag in a record describes the root node of the tree that
1244 makes up the total record. In the canonical input format, the root tag
1245 should contain the name of the schema that lends context to the
1246 elements of the record (see section <ref id="internal-representation"
1247 name="Internal Representation">). The following is a GILS record that
1248 contains only a single element (strictly speaking, that makes it an
1249 illegal GILS record, since the GILS profile includes several mandatory
1250 elements - Zebra does not validate the contents of a record against
1251 the Z39.50 profile, however - it merely attempts to match up elements
1252 of a local representation with the given schema):
1256 <title>Zen and the Art of Motorcycle Maintenance&etago;title>
1263 Zebra allows you to provide individual data elements in a number of
1264 <it/variant forms/. Examples of variant forms are textual data
1265 elements which might appear in different languages, and images which
1266 may appear in different formats or layouts. The variant system in
1268 essentially a representation of the variant mechanism of
1271 The following is an example of a title element which occurs in two
1272 different languages.
1276 <var lang lang "eng">
1277 Zen and the Art of Motorcycle Maintenance&etago;>
1278 <var lang lang "dan">
1279 Zen og Kunsten at Vedligeholde en Motorcykel&etago;>
1283 The syntax of the <it/variant element/ is <tt><<bf/var/ <it/class
1284 type value/></tt>. The available values for the <it/class/ and
1285 <it/type/ fields are given by the variant set that is associated with the
1286 current schema (see section <ref id="variant-set" name="Variant Set
1289 Variant elements are terminated by the general end-tag &etago;>, by
1290 the variant end-tag &etago;var>, by the appearance of another variant
1291 tag with the same <it/class/ and <it/value/ settings, or by the
1292 appearance of another, normal tag. In other words, the end-tags for
1293 the variants used in the example above could have been saved.
1295 Variant elements can be nested. The element
1299 <var lang lang "eng"><var body iana "text/plain">
1300 Zen and the Art of Motorcycle Maintenance
1304 Associates two variant components to the variant list for the title
1307 Given the nesting rules described above, we could write
1311 <var body iana "text/plain>
1312 <var lang lang "eng">
1313 Zen and the Art of Motorcycle Maintenance
1314 <var lang lang "dan">
1315 Zen og Kunsten at Vedligeholde en Motorcykel
1319 The title element above comes in two variants. Both have the IANA body
1320 type &dquot;text/plain&dquot;, but one is in English, and the other in
1321 Danish. The client, using the element selection mechanism of Z39.50,
1322 can retrieve information about the available variant forms of data
1323 elements, or it can select specific variants based on the requirements
1326 <sect2>Input Filters
1329 In order to handle general input formats, Zebra allows the
1330 operator to define filters which read individual records in their native format
1331 and produce an internal representation that the system can
1334 Input filters are ASCII files, generally with the suffix <tt/.flt/.
1335 The system looks for the files in the directories given in the
1336 <bf/profilePath/ setting in the <tt/zebra.cfg/ files. The record type
1337 for the filter is <tt>grs.regx.</tt><it>filter-filename</it>
1338 (fundamental type <tt>grs</tt>, file read type <tt>regx</tt>, argument
1339 <it>filter-filename</it>).
1341 Generally, an input filter consists of a sequence of rules, where each
1342 rule consists of a sequence of expressions, followed by an action. The
1343 expressions are evaluated against the contents of the input record,
1344 and the actions normally contribute to the generation of an internal
1345 representation of the record.
1347 An expression can be either of the following:
1350 <tag/INIT/The action associated with this expression is evaluated
1351 exactly once in the lifetime of the application, before any records
1352 are read. It can be used in conjunction with an action that
1353 initializes tables or other resources that are used in the processing
1356 <tag/BEGIN/Matches the beginning of the record. It can be used to
1357 initialize variables, etc. Typically, the <bf/BEGIN/ rule is also used
1358 to establish the root node of the record.
1360 <tag/END/Matches the end of the record - when all of the contents
1361 of the record has been processed.
1363 <tag>/pattern/</tag>Matches a string of characters from the input
1366 <tag/BODY/This keyword may only be used between two patterns. It
1367 matches everything between (not including) those patterns.
1369 <tag/FINISH/THe expression asssociated with this pattern is evaluated
1370 once, before the application terminates. It can be used to release
1371 system resources - typically ones allocated in the <bf/INIT/ step.
1375 An action is surrounded by curly braces ({...}), and consists of a
1376 sequence of statements. Statements may be separated by newlines or
1377 semicolons (;). Within actions, the strings that matched the
1378 expressions immediately preceding the action can be referred to as
1379 $0, $1, $2, etc.
1381 The available statements are:
1385 <tag>begin <it/type [parameter ... ]/</tag>Begin a new
1386 data element. The type is one of the following:
1388 <tag/record/Begin a new record. The followingparameter should be the
1389 name of the schema that describes the structure of the record, eg.
1390 <tt/gils/ or <tt/wais/ (see below). The <tt/begin record/ call should
1392 any other use of the <bf/begin/ statement.
1394 <tag/element/Begin a new tagged element. The parameter is the
1395 name of the tag. If the tag is not matched anywhere in the tagsets
1396 referenced by the current schema, it is treated as a local string
1399 <tag/variant/Begin a new node in a variant tree. The parameters are
1400 <it/class type value/.
1404 <tag/data/Create a data element. The concatenated arguments make
1405 up the value of the data element. The option <tt/-text/ signals that
1406 the layout (whitespace) of the data should be retained for
1407 transmission. The option <tt/-element/ <it/tag/ wraps the data up in
1408 the <it/tag/. The use of the <tt/-element/ option is equivalent to
1409 preceding the command with a <bf/begin element/ command, and following
1410 it with the <bf/end/ command.
1412 <tag>end <it/[type]/</tag>Close a tagged element. If no parameter is given,
1413 the last element on the stack is terminated. The first parameter, if
1414 any, is a type name, similar to the <bf/begin/ statement. For the
1415 <bf/element/ type, a tag name can be provided to terminate a specific tag.
1419 The following input filter reads a Usenet news file, producing a
1420 record in the WAIS schema. Note that the body of a news posting is
1421 separated from the list of headers by a blank line (or rather a
1422 sequence of two newline characters.
1425 BEGIN { begin record wais }
1427 /^From:/ BODY /$/ { data -element name $1 }
1428 /^Subject:/ BODY /$/ { data -element title $1 }
1429 /^Date:/ BODY /$/ { data -element lastModified $1 }
1431 begin element bodyOfDisplay
1432 begin variant body iana "text/plain"
1438 If Zebra is compiled with support for Tcl (Tool Command Language)
1439 enabled, the statements described above are supplemented with a complete
1440 scripting environment, including control structures (conditional
1441 expressions and loop constructs), and powerful string manipulation
1442 mechanisms for modifying the elements of a record. Tcl is a popular
1443 scripting environment, with several tutorials available both online
1446 <it>NOTE: Tcl support is not currently available, but will be
1447 included with one of the next alpha or beta releases.</it>
1449 <it>NOTE: Variant support is not currently available in the input
1450 filter, but will be included with one of the next alpha or beta
1453 <sect1>Internal Representation<label id="internal-representation">
1456 When records are manipulated by the system, they're represented in a
1457 tree-structure, with data elements at the leaf nodes, and tags or
1458 variant components at the non-leaf nodes. The root-node identifies the
1459 schema that lends context to the tagging and structuring of the
1460 record. Imagine a simple record, consisting of a 'title' element and
1461 an 'author' element:
1464 TITLE "Zen and the Art of Motorcycle Maintenance"
1466 AUTHOR "Robert Pirsig"
1469 A slightly more complex record would have the author element consist
1470 of two elements, a surname and a first name:
1473 TITLE "Zen and the Art of Motorcycle Maintenance"
1480 The root of the record will refer to the record schema that describes
1481 the structuring of this particular record. The schema defines the
1482 element tags (TITLE, FIRST-NAME, etc.) that may occur in the record, as
1483 well as the structuring (SURNAME should appear below AUTHOR, etc.). In
1484 addition, the schema establishes element set names that are used by
1485 the client to request a subset of the elements of a given record. The
1486 schema may also establish rules for converting the record to a
1487 different schema, by stating, for each element, a mapping to a
1490 <sect2>Tagged Elements
1493 A data element is characterized by its tag, and its position in the
1494 structure of the record. For instance, while the tag &dquot;telephone
1495 number&dquot; may be used different places in a record, we may need to
1496 distinguish between these occurrences, both for searching and
1497 presentation purposes. For instance, while the phone numbers for the
1498 &dquot;customer&dquot; and the &dquot;service provider&dquot; are both
1499 representatives for the same type of resource (a telephone number), it
1500 is essential that they be kept separate. The record schema provides
1501 the structure of the record, and names each data element (defined by
1502 the sequence of tags - the tag path - by which the element can be
1503 reached from the root of the record).
1508 The children of a tag node may be either more tag nodes, a data node
1509 (possibly accompanied by tag nodes),
1510 or a tree of variant nodes. The children of variant nodes are either
1511 more variant nodes or a data node (possibly accompanied by more
1512 variant nodes). Each leaf node, which is normally a
1513 data node, corresponds to a <it/variant form/ of the tagged element
1514 identified by the tag which parents the variant tree. The following
1515 title element occurs in two different languages:
1518 VARIANT LANG=ENG "War and Peace"
1520 VARIANT LANG=DAN "Krig og Fred"
1523 Which of the two elements are transmitted to the client by the server
1524 depends on the specifications provided by the client, if any.
1526 In practice, each variant node is associated with a triple of class,
1527 type, value, corresponding to the variant mechanism of Z39.50.
1529 <sect2>Data Elements
1532 Data nodes have no children (they are always leaf nodes in the record
1535 <it>NOTE: Documentation needs extension here about types of nodes - numerical,
1536 textual, etc., plus the various types of inclusion notes.</it>
1538 <sect1>Configuring Your Data Model<label id="data-model">
1541 The following sections describe the configuration files that govern
1542 the internal management of data records. The system searches for the files
1543 in the directories specified by the <bf/profilePath/ setting in the
1544 <tt/zebra.cfg/ file.
1546 <sect2>The Abstract Syntax
1549 The abstract syntax definition (also known as an Abstract Record
1550 Structure, or ARS) is the focal point of the
1551 record schema description. For a given schema, the ABS file may state any
1552 or all of the following:
1555 <item>The object identifier of the Z39.50 schema associated
1556 with the ARS, so that it can be referred to by the client.
1558 <item>The attribute set (which can possibly be a compound of multiple
1559 sets) which applies in the profile. This is used when indexing and
1560 searching the records belonging to the given profile.
1562 <item>The Tag set (again, this can consist of several different sets).
1563 This is used when reading the records from a file, to recognize the
1564 different tags, and when transmitting the record to the client -
1565 mapping the tags to their numerical representation, if they are
1568 <item>The variant set which is used in the profile. This provides a
1569 vocabulary for specifying the <it/forms/ of data that appear inside
1572 <item>Element set names, which are a shorthand way for the client to
1573 ask for a subset of the data elements contained in a record. Element
1574 set names, in the retrieval module, are mapped to <it/element
1575 specifications/, which contain information equivalent to the
1576 <it/Espec-1/ syntax of Z39.50.
1578 <item>Map tables, which may specify mappings to <it/other/ database
1579 profiles, if desired.
1581 <item>Possibly, a set of rules describing the mapping of elements to a
1582 MARC representation.
1584 <item>A list of element descriptions (this is the actual ARS of the
1585 schema, in Z39.50 terms), which lists the ways in which the various
1586 tags can be used and organized hierarchically.
1589 Several of the entries above simply refer to other files, which
1590 describe the given objects.
1592 <sect2>The Configuration Files
1595 This section describes the syntax and use of the various tables which
1596 are used by the retrieval module.
1598 The number of different file types may appear daunting at first, but
1599 each type corresponds fairly clearly to a single aspect of the Z39.50
1600 retrieval facilities. Further, the average database administrator,
1601 who is simply reusing an existing profile for which tables already
1602 exist, shouldn't have to worry too much about the contents of these tables.
1604 Generally, the files are simple ASCII files, which can be maintained
1605 using any text editor. Blank lines, and lines beginning with a (#) are
1606 ignored. Any characters on a line followed by a (#) are also ignored.
1608 lines contain <it/directives/, which provide some setting or value
1609 to the system. Generally, settings are characterized by a single
1610 keyword, identifying the setting, followed by a number of parameters.
1611 Some settings are repeatable (r), while others may occur only once in a
1612 file. Some settings are optional (o), whicle others again are
1615 <sect2>The Abstract Syntax (.abs) Files
1618 The name of this file type is slightly misleading in Z39.50 terms,
1619 since, apart from the actual abstract syntax of the profile, it also
1620 includes most of the other definitions that go into a database
1623 When a record in the canonical, SGML-like format is read from a file
1624 or from the database, the first tag of the file should reference the
1625 profile that governs the layout of the record. If the first tag of the
1626 record is, say, <tt><gils></tt>, the system will look for the profile
1627 definition in the file <tt/gils.abs/. Profile definitions are cached,
1628 so they only have to be read once during the lifespan of the current
1631 When writing your own input filters, the <bf/record-begin/ command
1632 introduces the profile, and should always be called first thing when
1633 introducing a new record.
1635 The file may contain the following directives:
1638 <tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or
1639 description for the profile. Mostly useful for diagnostic purposes.
1641 <tag>reference <it/OID-name/</tag> (m) The reference name of the OID for
1642 the profile. The reference names can be found in the <bf/util/
1645 <tag>attset <it/filename/</tag> (m) The attribute set that is used for
1646 indexing and searching records belonging to this profile.
1648 <tag>tagset <it/filename/</tag> (o) The tag set (if any) that describe
1649 that fields of the records.
1651 <tag>varset <it/filename/</tag> (o) The variant set used in the profile.
1653 <tag>maptab <it/filename/</tag> (o,r) This points to a
1654 conversion table that might be used if the client asks for the record
1655 in a different schema from the native one.
1657 <tag>marc <it/filename/</tag> (o) Points to a file containing parameters
1658 for representing the record contents in the ISO2709 syntax. Read the
1659 description of the MARC representation facility below.
1661 <tag>esetname <it/name filename/</tag> (o,r) Associates the
1662 given element set name with an element selection file. If an (@) is
1663 given in place of the filename, this corresponds to a null mapping for
1664 the given element set name.
1666 <tag>any <it/tags/</tag> (o) This directive specifies a list of
1667 attributes which should be appended to the attribute list given for each
1668 element. The effect is to make every single element in the abstract
1669 syntax searchable by way of the given attributes. This directive
1670 provides an efficient way of supporting free-text searching across all
1671 elements. However, it does increase the size of the index
1672 significantly. The attributes can be qualified with a structure, as in
1673 the <bf/elm/ directive below.
1675 <tag>elm <it/path name attributes/</tag> (o,r) Adds an element
1676 to the abstract record syntax of the schema. The <it/path/ follows the
1677 syntax which is suggested by the Z39.50 document - that is, a sequence
1678 of tags separated by slashes (/). Each tag is given as a
1679 comma-separated pair of tag type and -value surrounded by parenthesis.
1680 The <it/name/ is the name of the element, and the <it/attributes/
1681 specifies which attributes to use when indexing the element in a
1682 comma-separated list. A ! in
1683 place of the attribute name is equivalent to specifying an attribute
1684 name identical to the element name. A - in place of the attribute name
1685 specifies that no indexing is to take place for the given element. The
1686 attributes can be qualified with a &dquot;p&dquot; or &dquot;w&dquot;
1687 to specify either word or phrase (complete field) indexing.
1691 NOTE: The mechanism for controlling indexing is not adequate for
1692 complex databases, and will probably be moved into a separate
1693 configuration table eventually.
1696 The following is an excerpt from the abstract syntax file for the GILS
1701 reference GILS-schema
1706 maptab gils-usmarc.map
1710 esetname VARIANT gils-variant.est # for WAIS-compliance
1711 esetname B gils-b.est
1712 esetname G gils-g.est
1717 elm (1,14) localControlNumber Local-number
1718 elm (1,16) dateOfLastModification Date/time-last-modified
1720 elm (4,1) controlIdentifier Identifier-standard
1721 elm (2,6) abstract Abstract
1722 elm (4,51) purpose !
1723 elm (4,52) originator -
1724 elm (4,53) accessConstraints !
1725 elm (4,54) useConstraints !
1726 elm (4,70) availability -
1727 elm (4,70)/(4,90) distributor -
1728 elm (4,70)/(4,90)/(2,7) distributorName !
1729 elm (4,70)/(4,90)/(2,10 distributorOrganization !
1730 elm (4,70)/(4,90)/(4,2) distributorStreetAddress !
1731 elm (4,70)/(4,90)/(4,3) distributorCity !
1734 <sect2>The Attribute Set (.att) Files<label id="attset-files">
1737 This file type describes the <bf/Use/ elements of an attribute set.
1738 It contains the following directives.
1742 <tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or
1743 description for the attribute set. Mostly useful for diagnostic purposes.
1745 <tag>reference <it/OID-name/</tag> (m) The reference name of the OID for
1746 the attribute set. The reference names can be found in the <bf/util/
1749 <tag>ordinal <it/integer/</tag> (m) This value will be used to represent the
1750 attribute set in the index. Care should be taken that each attribute
1751 set has a unique ordinal value.
1753 <tag>include <it/filename/</tag> (o,r) This directive is used to
1754 include another attribute set as a part of the current one. This is
1755 used when a new attribute set is defined as an extension to another
1756 set. For instance, many new attribute sets are defined as extensions
1757 to the <bf/bib-1/ set. This is an important feature of the retrieval
1758 system of Z39.50, as it ensures the highest possible level of
1759 interoperability, as those access points of your database which are
1760 derived from the external set (say, bib-1) can be used even by clients
1761 who are unaware of the new set.
1763 <tag>att <it/att-value att-name [local-value]/</tag> (o,r) This
1764 repeatable directive introduces a new attribute to the set. The
1765 attribute value is stored in the index (unless a <it/local-value/ is
1766 given, in which case this is stored). The name is used to refer to the
1767 attribute from the <it/abstract syntax/. </descrip>
1769 This is an excerpt from the GILS attribute set definition. Notice how
1770 the file describing the <it/bib-1/ attribute set is referenced.
1774 reference GILS-attset
1778 att 2001 distributorName
1779 att 2002 indexTermsControlled
1781 att 2004 accessConstraints
1782 att 2005 useConstraints
1785 <sect2>The Tag Set (.tag) Files
1788 This file type defines the tagset of the profile, possibly by
1789 referencing other tag sets (most tag sets, for instance, will include
1790 tagsetG and tagsetM from the Z39.50 specification. The file may
1791 contain the following directives.
1794 <tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or
1795 description for the tag set. Mostly useful for diagnostic purposes.
1797 <tag>reference <it/OID-name/</tag> (o) The reference name of the OID for
1798 the tag set. The reference names can be found in the <bf/util/
1799 module of <bf/YAZ/. The directive is optional, since not all tag sets
1800 are registered outside of their schema.
1802 <tag>type <it/integer/</tag> (m) The type number of the tag within the schema
1805 <tag>include <it/filename/</tag> (o,r) This directive is used
1806 to include the definitions of other tag sets into the current one.
1808 <tag>tag <it/number names type/</tag> (o,r) Introduces a new
1809 tag to the set. The <it/number/ is the tag number as used in the protocol
1810 (there is currently no mechanism for specifying string tags at this
1811 point, but this would be quick work to add). The <it/names/ parameter
1812 is a list of names by which the tag should be recognized in the input
1813 file format. The names should be separated by slashes (/). The
1814 <it/type/ is th recommended datatype of the tag. It should be one of
1822 <item>generalizedtime
1830 The following is an excerpt from the TagsetG definition file.
1839 tag 3 publicationPlace string
1840 tag 4 publicationDate string
1841 tag 5 documentId string
1842 tag 6 abstract string
1844 tag 8 date generalizedtime
1845 tag 9 bodyOfDisplay string
1846 tag 10 organization string
1849 <sect2>The Variant Set (.var) Files<label id="variant-set">
1852 The variant set file is a straightforward representation of the
1853 variant set definitions associated with the protocol. At present, only
1854 the <it/Variant-1/ set is known.
1856 These are the directives allowed in the file.
1859 <tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or
1860 description for the variant set. Mostly useful for diagnostic purposes.
1862 <tag>reference <it/OID-name/</tag> (o) The reference name of the OID for
1863 the variant set, if one is required. The reference names can be found
1864 in the <bf/util/ module of <bf/YAZ/.
1866 <tag>class <it/integer class-name/</tag> (m,r) Introduces a new
1867 class to the variant set.
1869 <tag>type <it/integer type-name datatype/</tag> (m,r) Addes a
1870 new type to the current class (the one introduced by the most recent
1871 <bf/class/ directive). The type names belong to the same name space as
1872 the one used in the tag set definition file.
1875 The following is an excerpt from the file describing the variant set
1884 type 1 variantId octetstring
1889 type 2 z39.50 string
1893 <sect2>The Element Set (.est) Files
1896 The element set specification files describe a selection of a subset
1897 of the elements of a database record. The element selection mechanism
1898 is equivalent to the one supplied by the <it/Espec-1/ syntax of the
1899 Z39.50 specification. In fact, the internal representation of an
1900 element set specification is identical to the <it/Espec-1/ structure,
1901 and we'll refer you to the description of that structure for most of
1902 the detailed semantics of the directives below.
1905 NOTE: Not all of the Espec-1 functionality has been implemented yet.
1906 The fields that are mentioned below all work as expected, unless
1910 The directives available in the element set file are as follows:
1913 <tag>defaultVariantSetId <it/OID-name/</tag> (o) If variants are used in
1914 the following, this should provide the name of the variantset used
1915 (it's not currently possible to specify a different set in the
1916 individual variant request). In almost all cases (certainly all
1917 profiles known to us), the name <tt/Variant-1/ should be given here.
1919 <tag>defaultVariantRequest <it/variant-request/</tag> (o) This directive
1920 provides a default variant request for
1921 use when the individual element requests (see below) do not contain a
1922 variant request. Variant requests consist of a blank-separated list of
1923 variant components. A variant compont is a comma-separated,
1924 parenthesized triple of variant class, type, and value (the two former
1925 values being represented as integers). The value can currently only be
1926 entered as a string (this will change to depend on the definition of
1927 the variant in question). The special value (@) is interpreted as a
1928 null value, however.
1930 <tag>simpleElement <it/path ['variant' variant-request]/</tag>
1931 (o,r) This corresponds to a simple element request in <it/Espec-1/. The
1932 path consists of a sequence of tag-selectors, where each of these can
1936 <item>A simple tag, consisting of a comma-separated type-value pair in
1937 parenthesis, possibly followed by a colon (:) followed by an
1938 occurrences-specification (see below). The tag-value can be a number
1939 or a string. If the first character is an apostrophe ('), this forces
1940 the value to be interpreted as a string, even if it appears to be numerical.
1942 <item>A WildThing, represented as a question mark (?), possibly
1943 followed by a colon (:) followed by an occurrences specification (see
1946 <item>A WildPath, represented as an asterisk (*). Note that the last
1947 element of the path should not be a wildPath (wildpaths don't work in
1951 The occurrences-specification can be either the string <tt/all/, the
1952 string <tt/last/, or an explicit value-range. The value-range is
1953 represented as an integer (the starting point), possibly followed by a
1954 plus (+) and a second integer (the number of elements, default being
1957 The variant-request has the same syntax as the defaultVariantRequest
1958 above. Note that it may sometimes be useful to give an empty variant
1959 request, simply to disable the default for a specific set of fields
1960 (we aren't certain if this is proper <it/Espec-1/, but it works in
1961 this implementation).
1964 The following is an example of an element specification belonging to
1968 simpleelement (1,10)
1969 simpleelement (1,12)
1971 simpleelement (1,14)
1973 simpleelement (4,52)
1976 <sect2>The Schema Mapping (.map) Files<label id="schema-mapping">
1979 Sometimes, the client might want to receive a database record in
1980 a schema that differs from the native schema of the record. For
1981 instance, a client might only know how to process WAIS records, while
1982 the database record is represented in a more specific schema, such as
1983 GILS. In this module, a mapping of data to one of the MARC formats is
1984 also thought of as a schema mapping (mapping the elements of the
1985 record into fields consistent with the given MARC specification, prior
1986 to actually converting the data to the ISO2709). This use of the
1987 object identifier for USMARC as a schema identifier represents an
1988 overloading of the OID which might not be entirely proper. However,
1989 it represents the dual role of schema and record syntax which
1990 is assumed by the MARC family in Z39.50.
1993 NOTE: The schema-mapping functions are so far limited to a
1994 straightforward mapping of elements. This should be extended with
1995 mechanisms for conversions of the element contents, and conditional
1996 mappings of elements based on the record contents.
1999 These are the directives of the schema mapping file format:
2002 <tag>targetName <it/name/</tag> (m) A symbolic name for the target schema
2003 of the table. Useful mostly for diagnostic purposes.
2005 <tag>targetRef <it/OID-name/</tag> (m) An OID name for the target schema.
2006 This is used, for instance, by a server receiving a request to present
2007 a record in a different schema from the native one. The name, again,
2008 is found in the <bf/oid/ module of <bf/YAZ/.
2010 <tag>map <it/element-name target-path/</tag> (o,r) Adds
2011 an element mapping rule to the table.
2014 <sect2>The MARC (ISO2709) Representation (.mar) Files
2017 This file provides rules for representing a record in the ISO2709
2018 format. The rules pertain mostly to the values of the constant-length
2019 header of the record.
2021 <it>NOTE: This will be described better. We're in the process of
2022 re-evaluating and most likely changing the way that MARC records are
2023 handled by the system.</it>
2025 <sect1>Exchange Formats
2028 Converting records from the internal structure to en exchange format
2029 is largely an automatic process. Currently, the following exchange
2030 formats are supported:
2033 <item>GRS-1. The internal representation is based on GRS-1, so the
2034 conversion here is straightforward. The system will create
2035 applied variant and supported variant lists as required, if a record
2036 contains variant information.
2038 <item>SUTRS. Again, the mapping is fairly straighforward. Indentation
2039 is used to show the hierarchical structure of the record.
2041 <item>ISO2709-based formats (USMARC, etc.). Only records with a
2042 two-level structure (corresponding to fields and subfields) can be
2043 directly mapped to ISO2709. For records with a different structuring
2044 (eg., GILS), the representation in a structure like USMARC involves a
2045 schema-mapping (see section <ref id="schema-mapping" name="Schema
2046 Mapping">), to an &dquot;implied&dquot; USMARC schema (implied,
2047 because there is no formal schema which specifies the use of the
2048 USMARC fields outside of ISO2709). The resultant, two-level record is
2049 then mapped directly from the internal representation to ISO2709. See
2050 the GILS schema definition files for a detailed example of this
2053 <item>Explain. This representation is only available for records
2054 belonging to the Explain schema.
2061 Copyright © 1995,1996 Index Data.
2063 All rights reserved.
2065 Use and redistribution in source or binary form, with or without
2066 modification, of any or all of this software and documentation is
2067 permitted, provided that the following conditions are met:
2069 1. This copyright and permission notice appear with all copies of the
2070 software and its documentation. Notices of copyright or attribution
2071 which appear at the beginning of any file must remain unchanged.
2073 2. The names of Index Data or the individual authors may not be used to
2074 endorse or promote products derived from this software without specific
2075 prior written permission.
2077 3. Source code or binary versions of this software and its
2078 documentation may be used freely in not-for-profit applications. For
2079 profit applications - such as providing for-pay database services,
2080 marketing a product based in whole or in part on this software or its
2081 documentation, or generally distributing this software or its
2082 documentation under a different license - requires a commercial
2083 license from Index Data. The software may be installed and used for
2084 evaluation purposes in conjunction with a commercial application for a
2085 trial period of no more than 60 days.
2087 THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT WARRANTY OF ANY KIND,
2088 EXPRESS, IMPLIED, OR OTHERWISE, INCLUDING WITHOUT LIMITATION, ANY
2089 WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
2090 IN NO EVENT SHALL INDEX DATA BE LIABLE FOR ANY SPECIAL, INCIDENTAL,
2091 INDIRECT OR CONSEQUENTIAL DAMAGES OF ANY KIND, OR ANY DAMAGES
2092 WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER OR
2093 NOT ADVISED OF THE POSSIBILITY OF DAMAGE, AND ON ANY THEORY OF
2094 LIABILITY, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE
2097 <sect>About Index Data and the Zebra Server
2100 Index Data is a consulting and software-development enterprise that
2101 specialises in library and information management systems. Our
2102 interests and expertise span a broad range of related fields, and one
2103 of our primary, long-term objectives is the development of a powerful
2104 information management
2105 system with open network interfaces and hypermedia capabilities.
2107 We make this software available free of charge for not-for-profit
2108 purposes, as a service to the networking community, and to further
2109 the development and use of quality software for open network
2112 If you like this software, and would like to use all or part of it in
2113 a commercial product, or to provide a commercial database service,
2114 please contact us to discuss the details. We'll be happy to answer
2115 questions about the software, and about our services in general. If
2116 you have specific requirements to the software, we'll be glad to offer
2117 our advice - and if you need to adapt the software to a special
2118 purpose, our consulting services and expert knowledge of the software
2119 is available to you at favorable rates.
2124 DK-2200 København N&nl
2129 Phone: +45 3536 3672
2131 Email: info@index.ping.dk
2134 The <it>Random House College Dictionary</it>, 1975 edition
2135 offers this definition of the
2136 word &dquot;Zebra&dquot;:
2139 Zebra, n., any of several horselike, African mammals of the genus Equus,
2140 having a characteristic pattern of black or dark-brown stripes on
2141 a whitish background.