1 <!doctype linuxdoc system>
4 $Id: zebra.sgml,v 1.13 1996-01-03 16:21:58 quinn Exp $
8 <title>Zebra Server - Administrators's Guide and Reference
9 <author>Index Data, <tt/info@index.ping.dk/
10 <date>$Revision: 1.13 $
12 The Zebra information server combines a versatile fielded/free-text
13 search engine with a Z39.50-1995 frontend to provide a powerful and flexible
14 information management system. This document explains the procedure for
15 installing and configuring the system, and outlines the possibilities
16 for managing data and providing Z39.50
17 services with the software.
27 The Zebra system is a fielded free-text indexing and retrieval engine with a
28 Z39.50 frontend. You can use any commercial or freeware Z39.50 client
29 to access data stored in Zebra.
31 The Zebra server is our first step towards the development of a fully
32 configurable, open information system. Eventually, it will be paired
33 off with a powerful Z39.50 client to support complex information
34 management tasks within almost any application domain. We're making
35 the server available now because it's no fun to be in the open
36 information retrieval business all by yourself. We want to allow
37 people with interesting data to make their things
38 available in interesting ways, without having to start out
39 by implementing yet another protocol stack from scratch.
41 This document is an introduction to the Zebra system. It will tell you
42 how to compile the software, and how to prepare your first database.
43 It also explains how the server can be configured to give you the
44 functionality that you need.
46 If you find the software interesting, you should join the support
47 mailing-list by sending Email to <tt/zebra-request@index.ping.dk/.
52 This is a listof some of the most important features of the
58 Supports updating - records can be added and deleted without
59 rebuilding the index from scratch.
60 The update procedure is tolerant to crashes or hard interrupts
61 during register updating - registers can be reconstructed following a crash.
62 Registers can be safely updated even while users are accessing the server.
65 Supports large databases - files for indices, etc. can be
66 automatically partitioned over multiple disks.
69 Supports arbitrarily complex records - base input format is an
70 SGML-like syntax which allows nested (structured) data elements, as
71 well as variant forms of data.
74 Supports boolean queries as well as relevance-ranking (free-text)
75 searching. Right truncation and masking in terms are supported, as
76 well as full regular expressions.
79 Supports multiple concrete syntaxes
80 for record exchange (depending on the configuration): GRS-1, SUTRS,
81 ISO2709 (*MARC). Records can be mapped between record syntaxes and
90 Protocol facilities: Init, Search, Retrieve, Browse.
93 Piggy-backed presents are honored in the search-request.
96 Named result sets are supported.
99 Easily configured to support different application profiles, with
100 tables for attribute sets, tag sets, and abstract syntaxes.
101 Additional tables control facilities such as element mappings to
102 different schema (eg., GILS-to-USMARC).
105 Complex composition specifications using Espec-1 are partially
106 supported (simple element requests only).
109 Element Set Names are defined using the Espec-1 capability of the
110 system, and are given in configuration files as simple element
111 requests (and possibly variant requests).
114 Some variant support (not fully implemented yet).
117 Using the YAZ toolkit for the protocol implementation, the
118 server can utilise a plug-in XTI/mOSI implementation (not included) to
119 provide SR services over an OSI stack, as well as Z39.50 over TCP/IP.
128 This is an alfa-release of the software, to allow you to look at
129 it - try it out, and assess whether it can be of use to you. We expect
130 this version to be followed by a succession of beta-releases until we
131 arrive at a stable first version.
133 These are some of the plans that we have for the software in the near
134 and far future, approximately ordered after their relative importance.
136 asterisk will be implemented before the
142 *Allow the system to handle other input formats. Specifically
143 MARC records and general, structured ASCII records (such as mail/news
144 files) parameterized by regular expressions.
147 *Complete the support for variants. Finalize support for the WAIS
148 retrieval methodology.
151 *Finalize the data element <it/include/ facility to support multimedia
152 data elements in records.
155 *Port the system to Windows NT.
158 Add index and data compression to save disk space.
161 Add more sophisticated relevance ranking mechanisms. Add support for soundex
162 and stemming. Add relevance feedback support.
168 Add support for very large records by implementing segmentation and/or
172 Support the Item Update extended service of the protocol.
175 The Zebra search engine supports approximate string matching in the
176 index. We'd like to find a way to support and control this from RPN.
179 We want to add a management system that allows you to
180 control your databases and configuration tables from a graphical
181 interface. We'll probably use Tcl/Tk to stay platform-independent.
185 Programmers thrive on user feedback. If you are interested in a facility that
186 you don't see mentioned here, or if there's something you think we
187 could do better, please drop us a mail. If you think it's all really
188 neat, you're welcome to drop us a line saying that, too. You'll find
189 contact info at the end of this file.
191 <sect>Getting Started
193 <sect>Compiling the software
196 Zebra uses the YAZ package to implement Z39.50, so you
197 have to compile YAZ before going further. Specifically, Zebra uses
198 the YAZ header files in <tt>yaz/include/..</tt> and its public library
199 <tt>yaz/lib/libyaz.a</tt>.
201 As with YAZ, an ANSI C compiler is required in order to compile the Zebra
202 server system — <tt/gcc/ works fine if your own system doesn't
203 provide an adequate compiler.
205 Unpack the Zebra software. You might put Zebra in the same directory level
206 as YAZ, for example if YAZ is placed in ..<tt>/src/yaz-xxx</tt>, then
207 Zebra is placed in ..<tt>/src/zebra-yyy</tt>.
209 Edit the top-level <tt>Makefile</tt> in the Zebra directory in which
210 you specify the location of YAZ by setting make variables.
211 The <tt>OSILIB</tt> should be empty if YAZ wasn't compiled with
212 MOSI support. Some systems, such as Solaris, have separate socket
213 libraries and for those systems you need to specify the
214 <tt>NETLIB</tt> variable.
216 When you are done editing the <tt>Makefile</tt> type:
221 If successful, two executables have been created in the sub-directory
224 <tag><tt>zebrasrv</tt></tag> The Z39.50 server and search engine.
225 <tag><tt>zebraidx</tt></tag> The administrative tool for the search index.
231 This section will get you started quickly! We will try to index a few sample
232 GILS records that are included with the Zebra distribution. Go to the
233 <tt>test</tt> subdirectory. There you will find a configuration
234 file named <tt>zebra.cfg</tt> with the following contents:
236 # Where are the YAZ tables located.
237 profilePath: /usr/local/yaz
239 # Files that describe the attribute sets supported.
244 Now, edit the file and set <tt>profilePath</tt> to the path of the
245 YAZ profile tables (sub directory <tt>tab</tt> of YAZ).
247 The 48 test records are located in the sub directory <tt>records</tt>.
248 To index these, type:
250 $ ../index/zebraidx -t grs update records
253 In the command above the option <tt>-t</tt> specified the record
254 type — in this case <tt>grs</tt>. The word <tt>update</tt> followed
255 by a directory root updates all files below that directory node.
257 If your indexing command was successful, you are now ready to
258 fire up a server. To start a server on port 2100, type:
260 $ ../index/zebrasrv tcp:@:2100
263 The Zebra index that you've just made has one database called Default. It will
264 return either USMARC, GRS-1, or SUTRS depending on what your client asks
267 To test the server, you can use any Z39.50 client (1992 or later). For
268 instance, you can use the demo client that comes with YAZ: Just cd to
269 the <tt/client/ subdirectory of the YAZ distribution and type:
272 $ client tcp:localhost:2100
275 When the client has connected, you can type:
282 The default retrieval syntax for the client is USMARC. To try other
283 formats for the same record, try:
292 If you've made it this far, there's a reasonably good chance that
293 you've got through the compilation OK.
295 <sect><heading><label id="administrating">Administrating Zebra</>
298 Unlike many simpler retrieval systems, Zebra supports safe, incremental
299 updates to an existing index.
301 Normally, when Zebra modifies the index it reads a number of records
303 Depending on your specifications and on the contents of each record
304 one the following events take place for each record:
306 <tag>Insert</tag> The record is indexed as if it never occurred
307 before. Either the Zebra system doesn't know how to identify the record or
308 Zebra can identify the record but didn't find it to be already indexed.
309 <tag>Modify</tag> The record has already been indexed. In this case
310 either the contents of the record or the location (file) of the record
311 indicates that it has been indexed before.
312 <tag>Delete</tag> The record is deleted from the index. As in the
313 update-case it must be able to identify the record.
316 Please note that in both the modify- and delete- case the Zebra
317 indexer must be able to generate a unique key that identifies the record in
318 question (more on this below).
320 To administrate the Zebra retrieval system, you run the
321 <tt>zebraidx</tt> program. This program supports a number of options
322 which are preceded by a minus, and a few commands (not preceded by
325 Both the Zebra administrative tool and the Z39.50 server share a
326 set of index files and a global configuration file. The
327 name of the configuration file defaults to <tt>zebra.cfg</tt>.
328 The configuration file includes specifications on how to index
329 various kinds of records and where the other configuration files
330 are located. <tt>zebrasrv</tt> and <tt>zebraidx</tt> <em>must</em>
331 be run in the same directory where the configuration file if you do
332 not indicate the location of the configuration file by option
335 <sect1><heading><label id="record-types">Record Types</>
337 Indexing is a per-record process, in which
338 either insert/modify/delete will occur. Before a record is indexed
339 search keys are extracted from whatever might be the layout the
340 original record (sgml,html,text, etc..). The Zebra system
341 currently only supports SGML-like, structured records and unstructured text
343 To specify a particular extraction process, use either the
344 command line option <tt>-t</tt> or specify a
345 <tt>recordType</tt> setting in the configuration file.
347 <sect1><heading><label id="configuration-file">The Zebra Configuration File</>
349 The Zebra configuration file, read by <tt>zebraidx</tt> and
350 <tt>zebrasrv</tt> defaults to <tt>zebra.cfg</tt> unless specified
351 by <tt>-c</tt> option.
353 You can edit the configuration file with a normal text editor.
354 Parameter names and values are seperated by colons in the file. Lines
355 starting with a hash sign (<tt/#/) are treated as comments.
357 If you manage different sets of records that each share common
358 caracteristics, you can organize the configuration settings for each
359 type into &dquot;groups&dquot;.
360 When <tt>zebraidx</tt> is run and you wish to address a given group
361 you specify that group with the <tt>-g</tt> option. In this case
362 settings that have the group name as their prefix will be used
363 by <tt>zebraidx</tt> and not default values. The default values have no prefix.
365 The group is written before the option itself, separated by a dot (.).
366 For instance, to set the record type for group <tt/public/ to <tt/grs/
367 (the common format for structured records)
371 public.recordType: grs
374 To set the default value of the record type to <tt/text/ write:
380 The configuration settings are summarized below. They will be
381 explained further in the following sections.
384 <tag><it>group</it>recordType<it>name</it></tag>
385 Specifies how records with the file extension <it>name</it> should
386 be handled by the indexer. This option may also be specified
387 as a command line option (<tt>-t</tt>). Note that if you do not
388 specify a <it/name/, the setting applies to all files.
389 <tag><it>group</it>recordId</tag>
390 Specifies how the record is to be identified when updated.
391 <tag><it>group</it>database</tag>
392 Specifies the Z39.50 database name.
393 <tag><it>group</it>storeKeys</tag>
394 Specifies whether key information should be saved for a given
395 group of records. If you plan to update/delete this type of
396 records later this should be specified as 1; otherwise it
397 should be 0 (default).
398 <tag><it>group</it>storeData</tag>
399 Specifies whether the records should be stored internally
400 in the Zebra system files. If you want to maintain the raw records yourself,
401 this option should be false (0). If you want Zebra to take care of the records
402 for you, it should be true(1).
404 Specifies the location of the various files that Zebra uses to represent
406 <tag>tempSetPath</tag>
407 Specifies the directory that the server uses for temporary result sets.
408 If not specified <tt>/tmp</tt> will be used.
409 <tag>profilePath</tag>
410 Specifies the location of profile specification paths.
412 Specifies the filename(s) of attribute set files for use in
413 searching. At least the Bib-1 set should be loaded (<tt/bib1.att/).
414 The <tt/profilePath/ setting is used to search for attribute set
418 <sect1>Locating Records
420 The default behaviour of the Zebra system is to reference the
421 records from their original location, i.e. where they were found when you
424 If your input files are temporary - for example if you retrieve
425 your records from an outside source, or if they where temporarily mounted on a CD-ROM,
426 you may want Zebra to make an internal copy of them. To do this,
427 you specify 1 (true) in the <tt>storedata</tt> setting. When
428 the Z39.50 server retrieves the records they will be read from the
429 internal file structures of the system.
431 <sect1>Indexing with no Record IDs (Simple Indexing)
434 If you have a set of records that you <em/never/ wish to delete
435 or modify you may find &dquot;indexing without records IDs&dquot; convenient.
436 This indexing method uses less space than the other methods and
439 To use this method, you simply don't provide the <tt>recordId</tt> entry
440 for the group of files that you index. To add a set of records you use
441 <tt>zebraidx</tt> with the <tt>update</tt> command. The
442 <tt>update</tt> command will always add all of the records to the index
443 because Zebra doesn't know how to match the new set of records with
446 Consider a system in which you have a group of text files called
447 <tt>simple</tt>. That group of records should belong to a Z39.50 database
448 called <tt>textbase</tt>. The following <tt/zebra.cfg/ file will suffice:
451 profilePath: /usr/local/yaz
453 simple.recordType: text
454 simple.database: textbase
457 Since the existing records in an index can not be addressed by their
458 IDs, it is impossible to delete or modify records when using this method.
460 <sect1>Indexing with File Record IDs
463 If you have a set of external records that you wish to index you may
464 use the file key feature of the Zebra system. In short, the file key
465 methodology uses the paths of the files containing records as their
466 unique identifiers. To perform indexing of a directory with file keys,
467 again, you specify the top-level directory after the <tt>update</tt>
468 command. The command will recursively traverse the directories and
469 compare each with whatever have been indexed before in the same
470 directory. If a file is new (not in the previous version of the
471 directory) it is inserted into the registers; if a file was already
472 indexed and it has been modified since the last insertionm, the index
473 is also modified; if a file has been removed since the last visit, it
474 is deleted from the index.
476 The resulting system is easy to administer. To delete a record
477 you simply have to delete the corresponding file (say, with the
479 To force update of a given file, you may use the <tt>touch</tt>
480 command. And to add files create new files (or directories with files).
481 For your changes to take effect in the register you must run <tt>zebraidx</tt> with
482 the same directory root again.
484 To use this method, you must specify <tt>file</tt> as the value
485 of <tt>recordId</tt> in the configuration file. In addition, you
486 should set <tt>storeKeys</tt> to <tt>1</tt>, since the Zebra
487 indexer must save additional information about the keys to each record in order to
488 modify the indices correctly at a later time.
490 For example, to update group <tt>esdd</tt> records below
491 <tt>/home/grs</tt> you could type:
493 $ zebraidx -g esdd update /home/grs
496 The corresponding configuration file includes:
503 <em>Important note: You cannot start out with a group of records with simple
504 indexing (no record IDs as in the previous section) and then later
505 enable file record Ids. Zebra must know from the first time that you
507 the files should be indexed with file record IDs.
510 You cannot explicitly delete records when using this method (using the
511 <bf/delete/ command to <tt/zebraidx/. Instead
512 you have to delete the files from the file system (or remove them)
513 and then run <tt>zebraidx</tt> with the <bf/update/ command again.
515 <sect1>Indexing with General Record IDs
517 When using this method you construct an (almost) arbritrary, internal
518 record key based on the contents of the record itself and other system
519 information. If you have a group of records that associates an ID with
520 each record, this method is convenient. For example, the record may
521 contain a title or a ID-number - unique within the group. In either
522 case you specify the Z39.50 attribute set and use-attribute location
523 in which this information is stored, and the system looks at this
524 field to determine the identity of the record.
526 As before, the record ID is defined by the <tt>recordId</tt> setting
527 in the configuration file. The value of the record ID specification
528 consists of one or more tokens separated by whitespace. The resulting
530 represented in the index by concatenating the tokens and separating them by
533 There are three kinds of tokens:
535 <tag>Internal record info</tag> The token refers to a key that is
536 extracted from the record. The syntax of this token is
537 <tt/(/ <em/set/ <tt/,/ <em/use/ <tt/)/, where <em/set/ is the
538 attribute set ordinal number and <em/use/ is the use value of the attribute.
539 <tag>System variable</tag> The system variables are preceded by
540 <verb>$</verb> and immediately followed by the system variable name, which
543 <tag>group</tag> Group name.
544 <tag>database</tag> Current database specified.
545 <tag>type</tag> Record type.
547 <tag>Constant string</tag> A string used as part of the ID — surrounded
548 by single- or double quotes.
551 The sample GILS records that come with the Zebra distribution contain a
553 in the Control-Identifier field. This field is mapped to the Bib-1
554 use attribute 1007. To use this field as a record id, specify
555 <tt>(1,1007)</tt> as the value of the <tt>recordId</tt> in the
556 configuration file. If you have other record types that uses
557 the same field for a different purpose, you might add the record type (or group or database name)
558 to the record id of the gils records as well, to prevent matches
559 with other types of records. In this case the recordId might be
562 gils.recordId: $type (1,1007)
565 As for the file record id case described in the previous section
566 updating your system is simply a matter of running <tt>zebraidx</tt>
567 with the <tt>update</tt> command. However, the update with general
568 keys is considerably slower than with file record IDs, since all files
569 visited must be (re)read to find their IDs.
571 You may have noticed that when using the general record IDs
572 method, you can only add or modify existing records with the <tt>update</tt>
573 command. If you wish to delete records, you must use the,
574 <tt>delete</tt> command, with a directory as a parameter.
575 This will remove all records that match the files below that root
578 <sect1><heading><label id="register-location">Register Location</>
581 Normally, the index files that form dictionaries, inverted
582 files, record info, etc., are stored in the directory where you run
583 <tt>zebraidx</tt>. If you wish to store these, possibly large, files
584 somewhere else, you must add the <tt>register</tt> entry to the
585 configuration file. Furthermore, the Zebra system allows its file
587 span multiple file systems, which is useful if a very large number of
590 The value <tt>register</tt> of register is a sequence of tokens.
591 Each token takes the form:
593 <em>dir</em><tt>:</tt><em>size</em>.
595 The <em>dir</em> specifies a directory in which index files will be
596 stored and the <em>size</em> specifies the maximum size of all
597 files in that directory. The Zebra indexer system fills each directory
598 in the order specified and use the next specified directories as needed.
599 The <em>size</em> is an integer followed by a qualifier
600 code, <tt>M</tt> for megabytes, <tt>k</tt> for kilobytes.
602 For instance, if you have two spare disks :) and the first disk is mounted
603 on <tt>/d1</tt> and has 200 Mb of free space and the
604 second, mounted on <tt>/d2</tt> has 300 Mb, you could
605 put this entry in your configuration file:
607 register: /d1:200M /d2:300M
610 Note that Zebra does not verify that the amount of space specified is
611 actually available on the directory (file system) specified - it is
612 your responsibility to ensure that enough space is available, and that
613 other applications do not use the free space. In a large production system,
614 it is recommended that you allocate one or more filesystem exclusively
615 to the Zebra register files.
617 <sect1><heading><label id="shadow-registers">Safe Updating - Using Shadow Registers</>
622 The Zebra server supports updating of the index structures. That is,
623 you can add records to databases managed by Zebra without rebuilding
624 the entire index. Since this process involves modifying structured
625 files with various references between blocks of data in the files, the
626 update process is inherently sensitive to system crashes, or to
627 process interruptions: Anything but a successfully completed update
628 process will leave the register files in an unknown state, and you
629 will essentially have no recourse but to re-index everything, or to
630 restore the register files from a backup medium. Further, while the
631 update process is active, users cannot be allowed to access the
632 system, as the contents of the register files may change unpredictably.
634 You can solve these problems by enabling the shadow register system in
635 Zebra. During the updating procedure, <tt/zebraidx/ will temporarily
636 write changes to the involved files in a set of &dquot;shadow
637 files&dquot;, without modifying the files that are accessed by the
638 active server processes. If the update procedure is interrupted by a
639 system crash or a signal, you simply repeat the procedure - the
640 register files have not been changed or damaged, and the partially
641 written shadow files are automatically deleted before the new updating
644 At the end of the updating procedure (or in a separate operation, if
645 you so desire), the system enters a &dquot;commit mode&dquot;. First,
646 any active server processes are forced to access those blocks that
647 have been changed from the shadow files rather than from the main
648 register files; the unmodified blocks are still accessed at their
649 normal location (the shadow files are not a complete copy of the
650 register files - they only contain those parts that have actually been
651 modified). If the process is interrupted at any point during the
652 commit process, the server processes will continue to access the
653 shadow files until you can repeat the commit procedure and complete
654 the writing of data to the main register files. You can perform
655 multiple update operations to the registers before you commit the
656 changes to the system files, or you can execute the commit operation
657 at the end of each update operation. When the commit phase has
658 completed successfully, any running server processes are instructed to
659 switch their operations to the new, operational register, and the
660 temporary shadow files are deleted.
662 <sect2>How to Use Shadow Register Files
665 The first step is to allocate space on your system for the shadow
666 files. You do this by adding a <tt/shadow/ entry to the <tt/zebra.cfg/
667 file. The syntax of the <tt/shadow/ entry is exactly the same as for
668 the <tt/register/ entry (see section <ref name="Register Location"
669 id="register-location">). The location of the shadow area should be
670 <it/different/ from the location of the main register area (if you
671 have specified one - remember that the default register area is the
672 working directory of the server and indexing processes).
674 The following excerpt from a <tt/zebra.cfg/ file shows one example of
675 a setup that configures both the main register location and the shadow
676 file area. Note that two directories or partitions have been set aside
677 for the shadow file area. You can specify any number of directories
678 for each of the file areas.
683 shadow: /scratch1:100M /scratch2:200M
686 When shadow files are enabled, an extra command is available at the
687 <tt/zebraidx/ command line. In order to make changes to the system
688 take effect for the users, you'll have to submit a
689 &dquot;commit&dquot; command after a (sequence of) update
690 operation(s). You can ask the indexer to commit the changes
691 immediately after the update operation:
694 $ zebraidx update /d1/records update /d2/more-records commit
697 Or you can execute multiple updates before committing the changes:
700 $ zebraidx -g books update /d1/records update /d2/more-records
701 $ zebraidx -g fun update /d3/fun-records
705 If one of the update operations above had been interrupted, the commit
706 operation on the last line would fail: <tt/zebraidx/ will not let you
707 commit changes that would destroy the running register. You'll have to
708 rerun all of the update operations since your last commit operation,
709 before you can commit the new changes.
711 Similarly, if the commit operation fails, <tt/zebraidx/ will not let
712 you start a new update operation before you have successfully repeated
713 the commit operation. The server processes will keep accessing the
714 shadow files rather than the (possibly damaged) blocks of the main
715 register files until the commit operation has successfully completed.
717 You should be aware that update operations may take slightly longer
718 when the shadow register system is enabled, since more file access
719 operations are involved. Further, while the disk space required for
720 the shadow register data is modest for a small update operation, you
721 may prefer to disable the system if you are adding a very large number
722 of records to an already very large database (we use the terms
723 <it/large/ and <it/modest/ very loosely here, since every
724 application's perception of size is different). To update the system
725 without the use of the the shadow files, simply run <tt/zebraidx/ with
726 the <tt/-n/ option (note that you do not have to execute the
727 <bf/commit/ command of <tt/zebraidx/ when you temporarily disable the
728 use of the shadow registers in this fashion. Note also that, just as
729 when the shadow registers are not enabled, server processes will be
730 barred from accessing the main register while the update procedure
733 <sect>Running the Maintenance Interface (zebraidx)
736 The following is a complete reference to the command line interface to
737 the <tt/zebraidx/ application.
741 $ zebraidx [options] command [directory] ...
745 <tag>-t <it/type/</tag>Update all files as <it/type/. Currently, the
746 types supported are <tt/text/ and <tt/grs/<it/.filter/. If no
747 <it/filter/ is provided for the GRS (General Record Structure) type,
748 the canonical input format is assumed (see section <ref
749 id="local-representation" name="Local Representation">). Generally, it
750 is probably advisable to specify the record types in the
751 <tt/zebra.cfg/ file (see section <ref id="record-types" name="Record Types">).
753 <tag>-c <it/config-file/</tag>Read the configuration file
754 <it/config-file/ instead of <tt/zebra.cfg/.
756 <tag>-g <it/group/</tag>Update the files according to the group
757 settings for <it/group/ (see section <ref id="configuration-file"
758 name="The Zebra Configuration File">).
760 <tag>-d <it/database/</tag>The records located should be associated
761 with the database name <it/database/ for access through the Z39.50
764 <tag>-d <it/mbytes/</tag>Use <it/mbytes/ of megabytes before flushing
765 keys to background storage. This setting affects performance when
766 updating large databases.
768 <tag>-n</tag>Disable the use of shadow registers for this operation
769 (see section <ref id="shadow-registers" name="Robust Updating - Using
772 <tag>-v <it/level/</tag>Set the log level to <it/level/. <it/level/
773 should be one of <tt/none/, <tt/debug/, and <tt/all/.
779 <tag>Update <it/directory/</tag>Update the register with the files
780 contained in <it/directory/. If no directory is provided, a list of
781 files is read from <tt/stdin/. See section <ref
782 id="administrating" name="Administrating Zebra">.
784 <tag>Delete <it/directory/</tag>Remove the records corresponding to
785 the files found under <it/directory/ from the register.
787 <tag/Commit/Write the changes resulting from the last <bf/update/
788 commands to the register. This command is only available if the use of
789 shadow register files is enabled (see section <ref
790 id="shadow-registers" name="Robust Updating - Using Shadow
795 <sect>Running the Z39.50 Server (zebrasrv)
800 zebrasrv [options] [listener-address ...]
804 <tag>-a <it/APDU file/</tag> Specify a file for dumping PDUs (for diagnostic purposes).
805 The special name &dquot;-&dquot; sends output to <tt/stderr/.
807 <tag>-c <it/config-file/</tag> Read configuration information from <it/config-file/. The default configuration is <tt/./zebra.cfg/.
809 <tag/-S/Don't fork on connection requests. This can be useful for
810 symbolic-level debugging. The server can only accept a single
811 connection in this mode.
813 <tag/-s/Use the SR protocol.
815 <tag/-z/Use the Z39.50 protocol (default). These two options complement
816 eachother. You can use both multiple times on the same command
817 line, between listener-specifications (see below). This way, you
818 can set up the server to listen for connections in both protocols
819 concurrently, on different local ports.
821 <tag>-l <it/logfile/</tag>Specify an output file for the diagnostic
822 messages. The default is to write this information to <tt/stderr/.
824 <tag>-v <it/log-level/</tag>The log level. Use a comma-separated list of members of the set
825 {fatal,debug,warn,log,all,none}.
827 <tag>-u <it/username/</tag>Set user ID. Sets the real UID of the server process to that of the
828 given <it/username/. It's useful if you aren't comfortable with having the
829 server run as root, but you need to start it as such to bind a
833 A <it/listener-address/ consists of a transport mode followed by a
834 colon (:) followed by a listener address. The transport mode is
835 either <tt/osi/ or <tt/tcp/.
837 For TCP, an address has the form
840 hostname | IP-number [: portnumber]
843 The port number defaults to 210 (standard Z39.50 port).
845 For OSI (only available if the server is compiled with XTI/mOSI
846 support enabled), the address form is
849 [t-selector /] hostname | IP-number [: portnumber]
852 The transport selector is given as a string of hex digits (with an even
853 number of digits). The default port number is 102 (RFC1006 port).
861 osi:0402/dbserver.osiworld.com:3000
865 In both cases, the special hostname &dquot;@&dquot; is mapped to
866 the address INADDR_ANY, which causes the server to listen on any local
867 interface. To start the server listening on the registered ports for
868 Z39.50 and SR over OSI/RFC1006, and to drop root privileges once the
869 ports are bound, execute the server like this (from a root shell):
872 zebrasrv -u daemon tcp:@ -s osi:@
875 You can replace <tt/daemon/ with another user, eg. your own account, or
876 a dedicated IR server account.
878 The default behavior for <tt/zebrasrv/ is to establish a single TCP/IP
879 listener, for the Z39.50 protocol, on port 9999.
881 <sect>The Record Model
884 The Zebra system is designed to span a wide range of data management
885 applications. The system can be configured to handle virtually any
886 kind of structured data. Each record in the system is associated with
887 a <it/record schema/ which lends context to the data elements of the
888 record. Any number of record schema can coexist in the system.
889 Although it may be wise to use only a single schema within
890 one database, the system poses no such restrictions.
892 Records pass through three different states during processing in the
896 <item>When records are first entered into the system, they are represented
897 in their local, or native format. This might be SGML or HTML files,
898 News or Mail archives, MARC records. If the system doesn't already
899 know how to read the type of data you need to store, you can set up an
900 input filter by preparing conversion rules based on regular
901 expressions and a flexible scripting language (Tcl). The input filter
902 produces as output an internal representation:
904 <item>When records are processed by the system, they are represented
905 in a tree-structure, constructed by tagged data elements hanging off a
906 root node. The tagged elements may contain data or yet more tagged
907 elements in a recursive structure. The system performs various
908 actions on this tree structure (indexing, element selection, schema
911 <item>Before transmitting records to the client, they are first
912 converted from the internal structure to a form suitable for exchange
913 over the network - according to the Z39.50 standard.
916 <sect1><heading><label id="local-representation">Local Representation</>
919 As mentioned earlier, Zebra places few restrictions on the type of
920 data that you can index and manage. Generally, whatever the form of
921 the data, it is parsed by an input filter specific to that format, and
922 turned into an internal structure that Zebra knows how to handle. This
923 process takes place whenever the record is accessed - for indexing and
926 <sect2>Canonical Input Format
929 Although input data can take any form, it is sometimes useful to
930 describe the record processing capabilities of the system in terms of
931 a single, canonical input format that gives access to the full
932 spectrum of structure and flexibility in the system. In Zebra, this
933 canonical format is an &dquot;SGML-like&dquot; syntax.
935 Consider a record describing an information resource (such a record is
936 sometimes known as a <it/locator record/). It might contain a field
937 describing the distributor of the information resource, which might in
938 turn be partitioned into various fields providing details about the
939 distributor, like this:
943 <Name> USGS/WRD &etago;Name>
944 <Organization> USGS/WRD &etago;Organization>
946 U.S. GEOLOGICAL SURVEY, 505 MARQUETTE, NW
947 &etago;Street-Address>
948 <City> ALBUQUERQUE &etago;City>
949 <State> NM &etago;State>
950 <Zip-Code> 87102 &etago;Zip-Code>
951 <Country> USA &etago;Country>
952 <Telephone> (505) 766-5560 &etago;Telephone>
956 <it>NOTE: The indentation used above is used to illustrate how Zebra
957 interprets the expression. The indentation, in itself, has no
958 significance to the parser for the canonical input format, which
959 ignores all whitespace.</it>
961 The keywords surrounded by <...> are <it/tags/, while the
962 sections of text in between are the <it/data elements/. A data element
963 is characterized by its location in the tree that is made up by the
964 nested elements. Each element is terminated by a closing tag -
965 beginning with &etago;, and containing the same symbolic tag-name as
966 the corresponding opening tag. The general closing tag - &etago;> -
967 terminates the element started by the last opening tag. The
968 structuring of elements is significant. The element <bf/Telephone/,
969 for instance, may be indexed and presented to the client differently,
970 depending on whether it appears inside the <bf/Distributor/ element,
971 or some other data element.
976 The first tag in a record describes the root node of the tree that
977 makes up the total record. In the canonical input format, the root tag
978 should contain the name of the schema that lends context to the
979 elements of the record (see section <ref id="internal-representation"
980 name="Internal Representation">). The following is a GILS record that
981 contains only a single element (strictly speaking, that makes it an
982 illegal GILS record, since the GILS profile includes several mandatory
983 elements - Zebra does not validate the contents of a record against
984 the Z39.50 profile, however):
988 <title>Zen and the Art of Motorcycle Maintenance&etago;title>
995 Zebra allows you to provide individual data elements in a number of
996 <it/variant forms/. Examples of variant forms are textual data
997 elements which might appear in different languages, and images which
998 may appear in different formats or layouts. The variant system is
999 essentially a clean representation of the variant mechanism of
1002 The following is an example of a title element which occurs in two
1003 different languages.
1007 <var lang lang "eng">
1008 Zen and the Art of Motorcycle Maintenance&etago;>
1009 <var lang lang "dan">
1010 Zen og Kunsten at Vedligeholde en Motorcykel&etago;>
1014 The syntax of the <it/variant element/ is <tt><<bf/var/ <it/class
1015 type value/></tt>. The available values for the <it/class/ and
1016 <it/type/ fields are given by the variant set that is associated with the
1017 current schema (see section <ref id="variant-set" name="Variant Set
1020 Variant elements are terminated by the general end-tag &etago;>, by
1021 the variant end-tag &etago;var>, by the appearance of another variant
1022 tag with the same <it/class/ and <it/value/ settings, or by the
1023 appearance of another, normal tag. In other words, the end-tags for
1024 the variants used in the example above could have been saved.
1026 Variant elements can be nested. The element
1030 <var lang lang "eng"><var body iana "text/plain">
1031 Zen and the Art of Motorcycle Maintenance
1035 Associates two variant components to the variant list for the title
1036 element. Given the nesting rules described above, we could write
1040 <var body iana "text/plain>
1041 <var lang lang "eng">
1042 Zen and the Art of Motorcycle Maintenance
1043 <var lang lang "dan">
1044 Zen og Kunsten at Vedligeholde en Motorcykel
1048 The title element above comes in two variants. Both have the IANA body
1049 type &dquot;text/plain&dquot;, but one is in English, and the other in
1052 <sect2>Input Filters
1055 In order to handle general, text-based input formats, Zebra allows the
1056 operator to specify filters which read individual records in their native format
1057 and produce an internal representation that the system can
1060 Input filters are ASCII files, generally with the suffix <tt/.flt/.
1061 The system looks for the files in the directories given in the
1062 <bf/profilePath/ setting in the <tt/zebra.cfg/ file.
1064 Generally, an input filter consists of a sequence of rules, where each
1065 rule consists of a sequence of expressions, followed by an action. The
1066 expressions are evaluated against the contents of the input record,
1067 and the actions normally contribute to the generation of an internal
1068 representation of the record.
1070 An expression can be either of the following:
1073 <tag/INIT/The action associated with this expression is evaluated
1074 exactly once in the lifetime of the application, before any records
1075 are read. It can be used in conjunction with an action that
1076 initializes tables or other resources that are used in the processing
1079 <tag/BEGIN/Matches the beginning of the record. It can be used to
1080 initialize variables, etc. Typically, the <bf/BEGIN/ rule is also used
1081 to establish the root node of the record.
1083 <tag/END/Matches the end of the record - when all of the contents
1084 of the record has been processed.
1086 <tag>/pattern/</tag>Matches a string of characters from the input
1089 <tag/BODY/This keyword may only be used between two patterns. It
1090 matches everything between (not including) those patterns.
1092 <tag/FINISH/THe expression asssociated with this pattern is evaluated
1093 once, before the application terminates. It can be used to release
1094 system resources - typically ones allocated in the <bf/INIT/ step.
1098 An action is surrounded by curly braces ({...}), and consists of a
1099 sequence of statements. Statements may be separated by newlines or
1100 semicolons (;). Within actions, the strings that matched the
1101 expressions immediately preceding the action can be referred to as
1102 $0, $1, $2, etc.
1104 The available statements are:
1108 <tag>begin <it/type [parameter ... ]/</tag>Begin a new
1109 data element. The type is one of the following:
1111 <tag/record/Begin a new record. The parameter should be the
1112 name of the schema that describes the structure of the record, eg.
1113 <tt/gils/ or <tt/wais/. The <tt/begin record/ call should come before
1114 any other call to <bf/begin/.
1116 <tag/element/Begin a new tagged element. The parameter is the
1117 name of the tag. If the tag is not matched anywhere in the tagsets
1118 referenced by the current schema, it is treated as a local string
1121 <tag/variant/Begin a new node in a variant tree. The parameters are
1122 <it/class type value/.
1126 <tag/data/Create a data element under. The concatenated arguments make
1127 up the value of the data element. The option <tt/-text/ signals that
1128 the layout (whitespace) of the data should be retained for
1129 transmission. The option <tt/-element/ <it/tag/ wraps the data up in
1130 the <it/tag/. The use of the <tt/-element/ option is equivalent to
1131 preceding the command with a <bf/begin element/ command, and following
1132 it with the <bf/end/ command.
1134 <tag>end <it/[type]/</tag>Close a data element. If no parameter is given,
1135 the last element on the stack is terminated. The first parameter, if
1136 any, is a type name, similar to the <bf/begin/ statement. For the
1137 <bf/element/ type, a tag name can be provided to terminate a specific tag.
1141 The following input filter reads a Usenet news file, producing a
1142 record in the WAIS schema.
1145 BEGIN { begin record wais }
1147 /^From:/ BODY /$/ { data -element name $1 }
1148 /^Subject:/ BODY /$/ { data -element title $1 }
1149 /^Date:/ BODY /$/ { data -element lastModified $1 }
1151 begin element bodyOfDisplay
1152 begin variant body iana "text/plain"
1158 If Zebra is compiled with support for Tcl (Tool Command Language)
1159 enabled, the statements described above are supplemented with a complete
1160 scripting environment, including control structures (conditional
1161 expressions and loop constructs), and powerful string manipulation
1162 mechanisms for modifying the elements of a record. Tcl is a popular
1163 scripting environment, with several tutorials available both online
1166 <it>NOTE: Tcl support is not currently available, but will be
1167 included with the next release.</it>
1169 <sect1><heading><label id="internal-representation">Internal Representation</>
1172 When records are manipulated by the system, they're represented in a
1173 tree-structure, with data elements at the leaf nodes, and tags or
1174 variant components at the non-leaf nodes. The root-node identifies the
1175 schema that lends context to the tagging and structuring of the
1176 record. Imagine a simple record, consisting of a 'title' element and
1177 an 'author' element:
1180 TITLE "Zen and the Art of Motorcycle Maintenance"
1182 AUTHOR "Robert Pirsig"
1185 A slightly more complex record would have the author element consist
1186 of two elements, a surname and a first name:
1189 TITLE "Zen and the Art of Motorcycle Maintenance"
1196 The root of the record will refer to the record schema that describes
1197 the structuring of this particular record. The schema defines the
1198 element tags (TITLE, FIRST-NAME, etc.) that occur in the record, as
1199 well as the structuring (SURNAME should appear below AUTHOR, etc.). In
1200 addition, the schema establishes element set names that are used by
1201 the client to request a subset of the elements of a given record. The
1202 schema may also establish rules for converting the record to a
1203 different schema, by stating, for each element, a mapping to a
1206 The following sections describe the configuration files that govern
1207 the internal management of records. The system searches for the files
1208 in the directories specified by the <bf/profilePath/ setting in the
1209 <tt/zebra.cfg/ file.
1211 <sect2>The Abstract Syntax
1214 The abstract syntax definition (ARS) is the focal point of the
1215 record schema description. For a given schema, it may state any
1216 or all of the following:
1219 <item>The object identifier of the Z39.50 schema associated
1220 with the ARS, so that it can be referred to by the client.
1222 <item>The attribute set (which can possibly be a compound of multiple
1223 sets) which applies in the profile. This is used when indexing and
1224 searching the records belonging to the given profile.
1226 <item>The Tag set (again, this can consist of several different sets).
1227 This is used when reading the records from a file, to recognize the
1228 different tags, and when transmitting the record to the client -
1229 mapping the tags to their numerical representation, if they are
1232 <item>The variant set which is used in the profile. This provides a
1233 vocabulary for specifying the <it/forms/ of data that appear inside
1236 <item>Element set names, which are a shorthand way for the client to
1237 ask for a subset of the data elements contained in a record. Element
1238 set names, in the retrieval module, are mapped to <it/element
1239 specifications/, which contain information equivalent to the
1240 <it/Espec-1/ syntax of Z39.50.
1242 <item>Map tables, which may specify mappings to <it/other/ database
1243 profiles, if desired.
1245 <item>Possibly, a set of rules describing the mapping of elements to a
1246 MARC representation.
1248 <item>A list of element descriptions (this is the actual ARS of the
1249 schema, in Z39.50 terms), which lists the ways in which the various
1250 tags can be used and organized hierarchically.
1253 Several of the entries above simply refer to other files, which
1254 describe the given objects.
1256 <sect2>The Configuration Files
1259 This section describes the syntax and use of the various tables which
1260 are used by the retrieval module.
1262 The number of different file types may appear daunting at first, but
1263 each type corresponds fairly clearly to a single aspect of the Z39.50
1264 retrieval facilities. Further, the average database administrator
1265 who is simply reusing an existing profile for which tables already
1266 exist, shouldn't have to worry too much about the contents of these tables.
1268 Generally, the files are simple ASCII files, which can be maintained
1269 using any text editor. Blank lines, and lines beginning with a (#) are
1270 ignored. Any characters followed by a (#) are also ignored. All other
1271 lines contain <it/directives/, which establish some setting or value
1272 to the system. Generally, settings are characterized by a single
1273 keyword, identifying the setting, followed by a number of parameters.
1274 Some settings are repeatable (r), while others may occur only once in a
1275 file. Some settings are optional (o), whicle others again are
1278 <sect2>The Abstract Syntax (.abs) Files
1281 The name of this file type is slightly misleading in Z39.50 terms,
1282 since, apart from the actual abstract syntax of the profile, it also
1283 includes most of the other definitions that go into a database
1286 When a record in the canonical, SGML-like format is read from a file
1287 or from the database, the first tag of the file should reference the
1288 profile that governs the layout of the record. If the first tag of the
1289 record is, say, <tt><gils></tt>, the system will look for the profile
1290 definition in the file <tt/gils.abs/. Profile definitions are cached,
1291 so they only have to be read once during the lifespan of the current
1294 When writing your own input filters, the <bf/record-begin/ command
1295 introduces the profile, and should always be called first thing when
1296 introducing a new record.
1298 The file may contain the following directives:
1301 <tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or
1302 description for the profile. Mostly useful for diagnostic purposes.
1304 <tag>reference <it/OID-name/</tag> (m) The reference name of the OID for
1305 the profile. The reference names can be found in the <bf/util/
1308 <tag>attset <it/filename/</tag> (m) The attribute set that is used for
1309 indexing and searching records belonging to this profile.
1311 <tag>tagset <it/filename/</tag> (o) The tag set (if any) that describe
1312 that fields of the records.
1314 <tag>varset <it/filename/</tag> (o) The variant set used in the profile.
1316 <tag>maptab <it/filename/</tag> (o,r) This points to a
1317 conversion table that might be used if the client asks for the record
1318 in a different schema from the native one.
1320 <tag>marc <it/filename/</tag> (o) Points to a file containing parameters
1321 for representing the record contents in the ISO2709 syntax. Read the
1322 description of the MARC representation facility below.
1324 <tag>esetname <it/name filename/</tag> (o,r) Associates the
1325 given element set name with an element selection file. If an (@) is
1326 given in place of the filename, this corresponds to a null mapping for
1327 the given element set name.
1329 <tag>elm <it/path name attribute/</tag> (o,r) Adds an element
1330 to the abstract record syntax of the schema. The <it/path/ follows the
1331 syntax which is suggested by the Z39.50 document - that is, a sequence
1332 of tags separated by slashes (/). Each tag is given as a
1333 comma-separated pair of tag type and -value surrounded by parenthesis.
1334 The <it/name/ is the name of the element, and the <it/attribute/
1335 specifies what attribute to use when indexing the element. A ! in
1336 place of the attribute name is equivalent to specifying an attribute
1337 name identical to the element name. A - in place of the attribute name
1338 specifies that no indexing is to take place for the given element.
1342 NOTE: The mechanism for controlling indexing is not adequate for
1343 complex databases, and will probably be moved into a separate
1344 configuration table eventually.
1347 The following is an excerpt from the abstract syntax file for the GILS
1352 reference GILS-schema
1357 maptab gils-usmarc.map
1361 esetname VARIANT gils-variant.est # for WAIS-compliance
1362 esetname B gils-b.est
1363 esetname G gils-g.est
1368 elm (1,14) localControlNumber Local-number
1369 elm (1,16) dateOfLastModification Date/time-last-modified
1371 elm (4,1) controlIdentifier Identifier-standard
1372 elm (2,6) abstract Abstract
1373 elm (4,51) purpose !
1374 elm (4,52) originator -
1375 elm (4,53) accessConstraints !
1376 elm (4,54) useConstraints !
1377 elm (4,70) availability -
1378 elm (4,70)/(4,90) distributor -
1379 elm (4,70)/(4,90)/(2,7) distributorName !
1380 elm (4,70)/(4,90)/(2,10 distributorOrganization !
1381 elm (4,70)/(4,90)/(4,2) distributorStreetAddress !
1382 elm (4,70)/(4,90)/(4,3) distributorCity !
1385 <sect2>The Attribute Set (.att) Files
1388 This file type describes the <bf/Use/ elements of an attribute set.
1389 It contains the following directives.
1393 <tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or
1394 description for the attribute set. Mostly useful for diagnostic purposes.
1396 <tag>reference <it/OID-name/</tag> (m) The reference name of the OID for
1397 the attribute set. The reference names can be found in the <bf/util/
1400 <tag>ordinal <it/integer/</tag> (m) This value will be used to represent the
1401 attribute set in the index. Care should be taken that each attribute
1402 set has a unique ordinal value.
1404 <tag>include <it/filename/</tag> (o,r) This directive is used to
1405 include another attribute set as a part of the current one. This is
1406 used when a new attribute set is defined as an extension to another
1407 set. For instance, many new attribute sets are defined as extensions
1408 to the <bf/bib-1/ set. This is an important feature of the retrieval
1409 system of Z39.50, as it ensures the highest possible level of
1410 interoperability, as those access points of your database which are
1411 derived from the external set (say, bib-1) can be used even by clients
1412 who are unaware of the new set.
1414 <tag>att <it/att-value att-name [local-value]/</tag> (o,r) This
1415 repeatable directive introduces a new attribute to the set. The
1416 attribute value is stored in the index (unless a <it/local-value/ is
1417 given, in which case this is stored). The name is used to refer to the
1418 attribute from the <it/abstract syntax/. </descrip>
1420 This is an excerpt from the GILS attribute set definition. Notice how
1421 the file describing the <it/bib-1/ attribute set is referenced.
1425 reference GILS-attset
1429 att 2001 distributorName
1430 att 2002 indexTermsControlled
1432 att 2004 accessConstraints
1433 att 2005 useConstraints
1436 <sect2>The Tag Set (.tag) Files
1439 This file type defines the tagset of the profile, possibly by
1440 referencing other tag sets (most tag sets, for instance, will include
1441 tagsetG and tagsetM from the Z39.50 specification. The file may
1442 contain the following directives.
1445 <tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or
1446 description for the tag set. Mostly useful for diagnostic purposes.
1448 <tag>reference <it/OID-name/</tag> (o) The reference name of the OID for
1449 the tag set. The reference names can be found in the <bf/util/
1450 module of <bf/YAZ/. The directive is optional, since not all tag sets
1451 are registered outside of their schema.
1453 <tag>type <it/integer/</tag> (m) The type number of the tag within the schema
1456 <tag>include <it/filename/</tag> (o,r) This directive is used
1457 to include the definitions of other tag sets into the current one.
1459 <tag>tag <it/number names type/</tag> (o,r) Introduces a new
1460 tag to the set. The <it/number/ is the tag number as used in the protocol
1461 (there is currently no mechanism for specifying string tags at this
1462 point, but this would be quick work to add). The <it/names/ parameter
1463 is a list of names by which the tag should be recognized in the input
1464 file format. The names should be separated by slashes (/). The
1465 <it/type/ is th recommended datatype of the tag. It should be one of
1473 <item>generalizedtime
1481 The following is an excerpt from the TagsetG definition file.
1490 tag 3 publicationPlace string
1491 tag 4 publicationDate string
1492 tag 5 documentId string
1493 tag 6 abstract string
1495 tag 8 date generalizedtime
1496 tag 9 bodyOfDisplay string
1497 tag 10 organization string
1500 <sect2><heading><label id="variant-set">The Variant Set (.var) Files</>
1503 The variant set file is a straightforward representation of the
1504 variant set definitions associated with the protocol. At present, only
1505 the <it/Variant-1/ set is known.
1507 These are the directives allowed in the file.
1510 <tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or
1511 description for the variant set. Mostly useful for diagnostic purposes.
1513 <tag>reference <it/OID-name/</tag> (o) The reference name of the OID for
1514 the variant set, if one is required. The reference names can be found
1515 in the <bf/util/ module of <bf/YAZ/.
1517 <tag>class <it/integer class-name/</tag> (m,r) Introduces a new
1518 class to the variant set.
1520 <tag>type <it/integer type-name datatype/</tag> (m,r) Addes a
1521 new type to the current class (the one introduced by the most recent
1522 <bf/class/ directive). The type names belong to the same name space as
1523 the one used in the tag set definition file.
1526 The following is an excerpt from the file describing the variant set
1535 type 1 variantId octetstring
1540 type 2 z39.50 string
1544 <sect2>The Element Set (.est) Files
1547 The element set specification files describe a selection of a subset
1548 of the elements of a database record. The element selection mechanism
1549 is equivalent to the one supplied by the <it/Espec-1/ syntax of the
1550 Z39.50 specification. In fact, the internal representation of an
1551 element set specification is identical to the <it/Espec-1/ structure,
1552 and we'll refer you to the description of that structure for most of
1553 the detailed semantics of the directives below.
1556 NOTE: Not all of the Espec-1 functionality has been implemented yet.
1557 The fields that are mentioned below all work as expected, unless
1561 The directives available in the element set file are as follows:
1564 <tag>defaultVariantSetId <it/OID-name/</tag> (o) If variants are used in
1565 the following, this should provide the name of the variantset used
1566 (it's not currently possible to specify a different set in the
1567 individual variant request). In almost all cases (certainly all
1568 profiles known to us), the name <tt/Variant-1/ should be given here.
1570 <tag>defaultVariantRequest <it/variant-request/</tag> (o) This directive
1571 provides a default variant request for
1572 use when the individual element requests (see below) do not contain a
1573 variant request. Variant requests consist of a blank-separated list of
1574 variant components. A variant compont is a comma-separated,
1575 parenthesized triple of variant class, type, and value (the two former
1576 values being represented as integers). The value can currently only be
1577 entered as a string (this will change to depend on the definition of
1578 the variant in question). The special value (@) is interpreted as a
1579 null value, however.
1581 <tag>simpleElement <it/path ['variant' variant-request]/</tag>
1582 (o,r) This corresponds to a simple element request in <it/Espec-1/. The
1583 path consists of a sequence of tag-selectors, where each of these can
1587 <item>A simple tag, consisting of a comma-separated type-value pair in
1588 parenthesis, possibly followed by a colon (:) followed by an
1589 occurrences-specification (see below). The tag-value can be a number
1590 or a string. If the first character is an apostrophe ('), this forces
1591 the value to be interpreted as a string, even if it appears to be numerical.
1593 <item>A WildThing, represented as a question mark (?), possibly
1594 followed by a colon (:) followed by an occurrences specification (see
1597 <item>A WildPath, represented as an asterisk (*). Note that the last
1598 element of the path should not be a wildPath (wildpaths don't work in
1602 The occurrences-specification can be either the string <tt/all/, the
1603 string <tt/last/, or an explicit value-range. The value-range is
1604 represented as an integer (the starting point), possibly followed by a
1605 plus (+) and a second integer (the number of elements, default being
1608 The variant-request has the same syntax as the defaultVariantRequest
1609 above. Note that it may sometimes be useful to give an empty variant
1610 request, simply to disable the default for a specific set of fields
1611 (we aren't certain if this is proper <it/Espec-1/, but it works in
1612 this implementation).
1615 The following is an example of an element specification belonging to
1619 simpleelement (1,10)
1620 simpleelement (1,12)
1622 simpleelement (1,14)
1624 simpleelement (4,52)
1627 <sect2><heading><label id="schema-mapping">The Schema Mapping (.map) Files</>
1630 Sometimes, the client might want to receive a database record in
1631 a schema that differs from the native schema of the record. For
1632 instance, a client might only know how to process WAIS records, while
1633 the database record is represented in a more specific schema, such as
1634 GILS. In this module, a mapping of data to one of the MARC formats is
1635 also thought of as a schema mapping (mapping the elements of the
1636 record into fields consistent with the given MARC specification, prior
1637 to actually converting the data to the ISO2709). This use of the
1638 object identifier for USMARC as a schema identifier represents an
1639 overloading of the OID which might not be entirely proper. However,
1640 it represents the dual role of schema and record syntax which
1641 is assumed by the MARC family in Z39.50.
1644 NOTE: The schema-mapping functions are so far limited to a
1645 straightforward mapping of elements. This should be extended with
1646 mechanisms for conversions of the element contents, and conditional
1647 mappings of elements based on the record contents.
1650 These are the directives of the schema mapping file format:
1653 <tag>targetName <it/name/</tag> (m) A symbolic name for the target schema
1654 of the table. Useful mostly for diagnostic purposes.
1656 <tag>targetRef <it/OID-name/</tag> (m) An OID name for the target schema.
1657 This is used, for instance, by a server receiving a request to present
1658 a record in a different schema from the native one. The name, again,
1659 is found in the <bf/oid/ module of <bf/YAZ/.
1661 <tag>map <it/element-name target-path/</tag> (o,r) Adds
1662 an element mapping rule to the table.
1665 <sect2>The MARC (ISO2709) Representation (.mar) Files
1668 This file provides rules for representing a record in the ISO2709
1669 format. The rules pertain mostly to the values of the constant-length
1670 header of the record.
1672 <it>NOTE: This will be described better. We're in the process of
1673 re-evaluating and most likely changing the way that MARC records are
1674 handled by the system.</it>
1676 <sect1>Exchange Formats
1679 Converting records from the internal structure to en exchange format
1680 is largely an automatic process. Currently, the following exchange
1681 formats are supported:
1684 <item>GRS-1. The internal representation is based on GRS-1, so the
1685 conversion here is straightforward. The system will create
1686 applied variant and supported variant lists as required, if a record
1687 contains variant information.
1689 <item>SUTRS. Again, the mapping is fairly straighforward. Indentation
1690 is used to show the hierarchical structure of the record.
1692 <item>ISO2709-based formats (USMARC, etc.). Only records with a
1693 two-level structure (corresponding to fields and subfields) can be
1694 directly mapped to ISO2709. For records with a different structuring
1695 (eg., GILS), the representation in a structure like USMARC involves a
1696 schema-mapping (see section <ref id="schema-mapping" name="Schema
1697 Mapping">), to an &dquot;implied&dquot; USMARC schema (implied,
1698 because there is no formal schema which specifies the use of the
1699 USMARC fields outside of ISO2709). The resultant, two-level record is
1700 then mapped directly from the internal representation to ISO2709. See
1701 the GILS schema definition files for a detailed example of this
1704 <item>Explain. This representation is only available for records
1705 belonging to the Explain schema.
1712 Copyright © 1995, Index Data.
1714 All rights reserved.
1716 Use and redistribution in source or binary form, with or without
1717 modification, of any or all of this software and documentation is
1718 permitted, provided that the following conditions are met:
1720 1. This copyright and permission notice appear with all copies of the
1721 software and its documentation. Notices of copyright or attribution
1722 which appear at the beginning of any file must remain unchanged.
1724 2. The names of Index Data or the individual authors may not be used to
1725 endorse or promote products derived from this software without specific
1726 prior written permission.
1728 3. Source code or binary versions of this software and its
1729 documentation may be used freely in not-for-profit applications. For
1730 profit applications - such as providing for-pay database services,
1731 marketing a product based in whole or in part on this software or its
1732 documentation, or generally distributing this software or its
1733 documentation under a different license - requires a commercial
1734 license from Index Data. The software may be installed and used for
1735 evaluation purposes in conjunction with a commercial application for a
1736 trial period of no more than 60 days.
1738 THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT WARRANTY OF ANY KIND,
1739 EXPRESS, IMPLIED, OR OTHERWISE, INCLUDING WITHOUT LIMITATION, ANY
1740 WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
1741 IN NO EVENT SHALL INDEX DATA BE LIABLE FOR ANY SPECIAL, INCIDENTAL,
1742 INDIRECT OR CONSEQUENTIAL DAMAGES OF ANY KIND, OR ANY DAMAGES
1743 WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER OR
1744 NOT ADVISED OF THE POSSIBILITY OF DAMAGE, AND ON ANY THEORY OF
1745 LIABILITY, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE
1748 <sect>About Index Data and the Zebra Server
1751 Index Data is a consulting and software-development enterprise that
1752 specialises in library and information management systems. Our
1753 interests and expertise span a broad range of related fields, and one
1754 of our primary, long-term objectives is the development of a powerful
1755 information management
1756 system with open network interfaces and hypermedia capabilities.
1758 We make this software available free of charge for not-for-profit
1759 purposes, as a service to the networking community, and to further
1760 the development and use of quality software for open network
1763 If you like this software, and would like to use all or part of it in
1764 a commercial product, or to provide a commercial database service,
1765 please contact us to discuss the details. We'll be happy to answer
1766 questions about the software, and about our services in general. If
1767 you have specific requirements to the software, we'll be glad to offer
1768 our advice - and if you need to adapt the software to a special
1769 purpose, our consulting services and expert knowledge of the software
1770 is available to you at favorable rates.
1775 DK-2200 København N&nl
1780 Phone: +45 3536 3672
1782 Email: info@index.ping.dk
1785 The <it>Random House College Dictionary</it>, 1975 edition
1786 offers this definition of the
1787 word &dquot;Zebra&dquot;:
1790 Zebra, n., any of several horselike, African mammals of the genus Equus,
1791 having a characteristic pattern of black or dark-brown stripes on
1792 a whitish background.