1 <!doctype linuxdoc system>
4 $Id: zebra.sgml,v 1.43 1999-12-08 15:03:11 adam Exp $
8 <title>Zebra Server - Administrators's Guide and Reference
9 <author><htmlurl url="http://www.indexdata.dk/" name="Index Data">,
10 <tt><htmlurl url="mailto:info@indexdata.dk" name="info@indexdata.dk"></>
11 <date>$Revision: 1.43 $
15 The Zebra server combines a versatile fielded/free-text
16 indexing/search engine with a Z39.50-1995 frontend to provide a powerful and flexible
17 information mining tool. This document explains the procedure for
18 installing and configuring Zebra, and outlines the possibilities
19 for managing data and providing Z39.50
20 services with the software. Zebra is a free version of the Index Data Z'mbol
21 information system, and it excludes some functionality such as incremental
22 database updating and support for large databases.
32 Zebra is a fielded free-text indexing and retrieval engine with a
33 Z39.50 frontend. You can use any commercial or freeware Z39.50 client
34 to access data stored in Zebra.
36 Zebra server can be used at the core of a Z39.50-based information retrieval
37 framework. We're making
38 the server available now to allow researchers and small organisations to
39 share their information in the best possible way. We believe that Z39.50
40 currently represents one of the best ways of sharing information with others, and
41 we would like to encourage as many people as possible to do so.
42 This document is a guide to using Zebra. It will tell you
43 how to compile the software, and how to prepare your first database.
44 It also explains how the server can be configured to give you the
45 functionality that you need.
47 If you find the software interesting, you should join the support
48 mailing-list by sending email to <tt/zebra-request@indexdata.dk/.
50 If you are interested in running a commercial service, if you wish to run large
51 databases, or if you wish to make incremental updates to your databases even
52 while users are accessing your system, then you might be interested in the Z'mbol
53 Information Server which is available from <htmlurl
54 url="http://www.indexdata.dk/zmbol/" name="Index Data"> or Fretwell-Downing
55 Informatics. Z'mbol is a complete and supported package which offers many
56 exciting possibilities that we have not been able to fit into this package.
61 This is a list of some of the most important features of the
67 Supports arbitrarily complex records - base input format is an
68 XML-like syntax which allows nested (structured) data elements, as
69 well as variant forms of data.
72 Supports random storage formats. A system of input filters driven by
73 regular expressions allows you to easily process most ASCII-based
74 data formats. SGML/XML, ISO2709 (MARC), and raw text are also supported.
77 Supports boolean queries as well as relevance-ranking (free-text)
78 searching. Right truncation and masking in terms are supported, as
79 well as full regular expressions.
82 Supports multiple concrete syntaxes
83 for record exchange (depending on the configuration): GRS-1, SUTRS,
84 ISO2709 (*MARC), XML. Records can be mapped between record syntaxes and
88 Supports approximate matching in registers (ie. spelling mistakes,
91 <item> Supports a subset of the Z39.50 Explain Facility. Zebra's Explain database
92 is automatically updated when a set of records is loaded into Zebra.
102 Protocol facilities: Init, Search, Retrieve, Browse, Sort, Close, and Explain.
105 Piggy-backed presents are honored in the search-request.
108 Named result sets are supported.
111 Easily configured to support different application profiles, with
112 tables for attribute sets, tag sets, and abstract syntaxes.
113 Additional tables control facilities such as element mappings to
114 different schema (eg., GILS-to-USMARC).
117 Complex composition specifications using Espec-1 are partially
118 supported (simple element requests only).
121 Element Set Names are defined using the Espec-1 capability of the
122 system, and are given in configuration files as simple element
123 requests (and possibly variant requests).
126 Zebra runs on most Unix-like systems as well as Windows NT - a binary
127 distribution for Windows NT is forthcoming - so far, the installation
128 requires Microsoft Visual C++ to compile the system (we use version 6.0).
136 These are some of the plans that we have for the software in the near
137 and far future, approximately ordered after their relative importance.
139 asterisk will be implemented before the
145 *Complete the support for variants.
148 *Finalize the data element <it/include/ facility to support multimedia
149 data elements in records.
152 Add more sophisticated relevance ranking mechanisms. Add support for soundex
153 and stemming. Add relevance <it/feedback/ support.
156 Complete EXPLAIN support.
159 We want to add a management system that allows you to
160 control your databases and configuration tables from a graphical
161 interface. We'll probably use Tcl/Tk to stay platform-independent.
165 Programmers thrive on user feedback. If you are interested in a facility that
166 you don't see mentioned here, or if there's something you think we
167 could do better, please drop us a mail. If you think it's all really
168 neat, you're welcome to drop us a line saying that, too. You'll find
169 contact info at the end of this file.
171 <sect>Compiling the software
174 <bf><htmlurl url="http://www.indexdata.dk/yaz/" name="YAZ"></>
175 package in order to compile this software. We suggest you
176 unpack <bf/YAZ/ in the same directory as Zebra. Running
177 ./configure (UNIX Only) and running make (nmake on WIN32) is
178 in usully what it takes to compile YAZ.
182 An ANSI C compiler is required to compile the Zebra
183 server system — <tt/gcc/ works very well if your own system doesn't
184 provide an adequate compiler.
186 Unpack the distribution archive. The <tt>configure</tt> shell script
187 attempts to guess correct values for various system-dependent variables
188 used during compilation. It uses those values to create a 'Makefile' in
189 each directory of Zebra.
191 To run the configure script type:
196 The configure script attempts to use the C compiler specified by
197 the <tt>CC</tt> environment variable. If not set, GNU C
198 will be used if it is available. The <tt>CFLAGS</tt> environment variable
199 holds options to be passed to the C compiler. If you're using a
200 Bourne-compatible shell you may pass something like this:
202 CC=/opt/ccs/bin/cc CFLAGS=-O ./configure
205 To customize Zebra the configure script accepts a set of options. The
208 <tag><tt>--prefix </tt>path</tag> Specifies installation prefix. This is
209 only needed if you run <tt>make install</tt> later to perform a
210 "system" installation. The prefix is <tt>/usr/local</tt> if not
212 <tag><tt>--with-tclconfig </tt>path</tag> If Tcl is installed on
213 the system you can tell configure where Tcl's <tt>tclConfig.sh</tt>
214 installed. The <tt>tclConfig.sh</tt> include information about settings
215 required to link with Tcl's libraries. If you don't specify this
216 option, configure will see if Tcl's shell <tt>tclsh</tt> is in your
217 path and if it is, it will guess where the equivalent tclConfig.sh
218 is located. If tclsh is not found in your path and this option is not
219 given Zebra will not include Tcl support.
220 <tag><tt>--with-yazconfig </tt>path</tag> This options allows you to
221 specify the path of YAZ's <tt>yaz-config</tt>. Therefore this option
222 forces Zebra to use a particular version of YAZ. YAZ version 1.5 and
223 later creates a script <tt>yaz-config</tt> that includes information
224 on compiler settings needed to link with it.
227 When configured build the software by typing:
232 As an option you may type <tt>make depend</tt> to create
233 source file dependencies for the package. This is only needed,
234 however, if you modify the source code later.
236 If successful, two executables have been created in the sub-directory
239 <tag><tt>zebrasrv</tt></tag> The Z39.50 server and search engine.
240 <tag><tt>zebraidx</tt></tag> The administrative tool for the search index.
244 The next step is optional and is only needed if you wish to install
245 zebra in system directories such as /usr/bin, /usr/lib, etc.
247 To perform this step, type
252 The executables will be installed in prefix/bin, and profile
253 tables will be installed in prefix/lib/zebra/tab. Here prefix
254 represents the prefix as specified -- default being /usr/local.
259 Zebra is shipped with "makefiles" for the NMAKE tool that comes
262 Start an MS-DOS prompt and switch the sub directory <tt>WIN</tt> where
263 the file <tt>zebra.mak</tt> is located. Customize the installation
264 by editing the <tt>zebra.mak</tt> file (for example by using notepad).
266 The following summarises the most important settings in that
270 <tag><tt>YAZDIR</tt></tag> Specifies where YAZ is located.
271 <tag><tt>DEBUG</tt></tag> If set to 1, the software is
272 compiled with debugging libraries. If set to 0, the software
273 is compiled with release (non-debugging) libraries.
274 <tag>BZIP2</tag> A group of settings (<tt>BZIP2LIB</tt>,..)
275 that must be defined if BZIP2 compression support is desired.
278 When satisfied with the settings in the makefile type
283 If compilation was successful the executables <tt>zebraidx.exe</tt>
284 and <tt>zebrasrv.exe</tt> are put in the sub directory <tt>BIN</tt>.
288 In this section, we will test the system by indexing a small set of sample
289 GILS records that are included with the software distribution. Go to the
290 <tt>test/gils</tt> subdirectory of the distribution archive. There you will
292 file named <tt>zebra.cfg</tt> with the following contents:
294 # Where are the YAZ tables located.
295 profilePath: ../../../yaz/tab ../../tab
297 # Files that describe the attribute sets supported.
303 Now, edit the file and set <tt>profilePath</tt> to the path of the
304 YAZ profile tables (sub directory <tt>tab</tt> of the YAZ distribution
307 The 48 test records are located in the sub directory <tt>records</tt>.
308 To index these, type:
310 $ ../../bin/zebraidx -t grs.sgml update records
313 In the command above the option <tt>-t</tt> specified the record
314 type — in this case <tt>grs.sgml</tt>. The word <tt>update</tt> followed
315 by a directory root updates all files below that directory node.
317 If your indexing command was successful, you are now ready to
318 fire up a server. To start a server on port 2100, type:
320 $ ../../bin/zebrasrv tcp:@:2100
323 The Zebra index that you have just created has a single database
324 named <tt/Default/. The database contains records structured according to
325 the GILS profile, and the server will
326 return records in either either XML, USMARC, GRS-1, or SUTRS depending
327 on what your client asks for.
329 To test the server, you can use any Z39.50 client (1992 or later). For
330 instance, you can use the demo client that comes with YAZ: Just cd to
331 the <tt/client/ subdirectory of the YAZ distribution and type:
334 $ ./yaz-client tcp:localhost:2100
337 When the client has connected, you can type:
344 The default retrieval syntax for the client is USMARC. To try other
345 formats for the same record, try:
358 <it>NOTE: You may notice that more fields are returned when your
359 client requests SUTRS or GRS-1 records. When retrieving GILS records,
360 this is normal - not all of the GILS data elements have mappings in
361 the USMARC record format.</it>
363 If you've made it this far, there's a good chance that
364 you've got through the compilation OK.
366 <sect>Administrating Zebra<label id="administrating">
370 To administrate Zebra, you run the
371 <tt>zebraidx</tt> program. This program supports a number of options
372 which are preceded by a minus, and a few commands (not preceded by
375 Both the Zebra administrative tool and the Z39.50 server share a
376 set of index files and a global configuration file. The
377 name of the configuration file defaults to <tt>zebra.cfg</tt>.
378 The configuration file includes specifications on how to index
379 various kinds of records and where the other configuration files
380 are located. <tt>zebrasrv</tt> and <tt>zebraidx</tt> <em>must</em>
381 be run in the directory where the configuration file lives unless you
382 indicate the location of the configuration file by option
385 <sect1>Record Types<label id="record-types">
387 Indexing is a per-record process. Before a record is indexed search
388 keys are extracted from whatever might be the layout the original
389 record (sgml,html,text, etc..).
390 The Zebra system currently supports two fundamantal types of records:
391 structured and simple text.
392 To specify a particular extraction process, use either the
393 command line option <tt>-t</tt> or specify a
394 <tt>recordType</tt> setting in the configuration file.
396 <sect1>The Zebra Configuration File<label id="configuration-file">
398 The Zebra configuration file, read by <tt>zebraidx</tt> and
399 <tt>zebrasrv</tt> defaults to <tt>zebra.cfg</tt> unless specified
400 by <tt>-c</tt> option.
402 You can edit the configuration file with a normal text editor.
403 Parameter names and values are seperated by colons in the file. Lines
404 starting with a hash sign (<tt/#/) are treated as comments.
406 If you manage different sets of records that share common
407 characteristics, you can organize the configuration settings for each
408 type into &dquot;groups&dquot;.
409 When <tt>zebraidx</tt> is run and you wish to address a given group
410 you specify the group name with the <tt>-g</tt> option. In this case
411 settings that have the group name as their prefix will be used
412 by <tt>zebraidx</tt>. If no <tt/-g/ option is specified, the settings
413 with no prefix are used.
415 In the configuration file, the group name is placed before the option
416 name itself, separated by a dot (.). For instance, to set the record type
417 for group <tt/public/ to <tt/grs.sgml/ (the SGML-like format for structured
418 records) you would write:
421 public.recordType: grs.sgml
424 To set the default value of the record type to <tt/text/ write:
430 The available configuration settings are summarized below. They will be
431 explained further in the following sections.
434 <tag><it>group</it>.recordType[<it>.name</it>]</tag>
435 Specifies how records with the file extension <it>name</it> should
436 be handled by the indexer. This option may also be specified
437 as a command line option (<tt>-t</tt>). Note that if you do not
438 specify a <it/name/, the setting applies to all files. In general,
439 the record type specifier consists of the elements (each
440 element separated by dot), <it>fundamental-type</it>,
441 <it>file-read-type</it> and arguments. Currently, two
442 fundamental types exist, <tt>text</tt> and <tt>grs</tt>.
443 <tag><it>group</it>.recordId</tag>
444 Specifies how the records are to be identified when updated. See
445 section <ref id="locating-records" name="Locating Records">.
446 <tag><it>group</it>.database</tag>
447 Specifies the Z39.50 database name.
448 <tag><it>group</it>.storeKeys</tag>
449 Specifies whether key information should be saved for a given
450 group of records. If you plan to update/delete this type of
451 records later this should be specified as 1; otherwise it
452 should be 0 (default), to save register space. See section
453 <ref id="file-ids" name="Indexing With File Record IDs">.
454 <tag><it>group</it>.storeData</tag>
455 Specifies whether the records should be stored internally
456 in the Zebra system files. If you want to maintain the raw records yourself,
457 this option should be false (0). If you want Zebra to take care of the records
458 for you, it should be true(1).
460 Directory in which various lock files are stored.
462 Directory in which temporary files used during zebraidx' update
465 Specifies the directory that the server uses for temporary result sets.
466 If not specified <tt>/tmp</tt> will be used.
467 <tag>profilePath</tag>
468 Specifies the location of profile specification files.
470 Specifies the filename(s) of attribute set files for use in
471 searching. At least the Bib-1 set should be loaded (<tt/bib1.att/).
472 The <tt/profilePath/ setting is used to look for the specified files.
473 See section <ref id="attset-files" name="The Attribute Set Files">
475 Specifies size of internal memory to use for the zebraidx program. The
476 amount is given in megabytes - default is 4 (4 MB).
478 <sect1>Locating Records<label id="locating-records">
480 The default behaviour of the Zebra system is to reference the
481 records from their original location, i.e. where they were found when you
482 ran <tt/zebraidx/. That is, when a client wishes to retrieve a record
483 following a search operation, the files are accessed from the place
484 where you originally put them - if you remove the files (without
485 running <tt/zebraidx/ again, the client will receive a diagnostic
488 If your input files are not permanent - for example if you retrieve
489 your records from an outside source, or if they were temporarily
490 mounted on a CD-ROM drive,
491 you may want Zebra to make an internal copy of them. To do this,
492 you specify 1 (true) in the <tt>storeData</tt> setting. When
493 the Z39.50 server retrieves the records they will be read from the
494 internal file structures of the system.
496 <sect1>Indexing example
499 Consider a system in which you have a group of text files called
500 <tt>simple</tt>. That group of records should belong to a Z39.50 database
501 called <tt>textbase</tt>. The following <tt/zebra.cfg/ file will suffice:
504 profilePath: /usr/lib/yaz/tab:/usr/lib/zebra/tab
507 simple.recordType: text
508 simple.database: textbase
511 <sect>Running the Maintenance Interface (zebraidx)
514 The following is a complete reference to the command line interface to
515 the <tt/zebraidx/ application.
519 $ zebraidx [options] command [directory] ...
523 <tag>-t <it/type/</tag>Update all files as <it/type/. Currently, the
524 types supported are <tt/text/ and <tt/grs/<it/.subtype/. If no
525 <it/subtype/ is provided for the GRS (General Record Structure) type,
526 the canonical input format is assumed (see section <ref
527 id="local-representation" name="Local Representation">). Generally, it
528 is probably advisable to specify the record types in the
529 <tt/zebra.cfg/ file (see section <ref id="record-types" name="Record
530 Types">), to avoid confusion at subsequent updates.
532 <tag>-c <it/config-file/</tag>Read the configuration file
533 <it/config-file/ instead of <tt/zebra.cfg/.
535 <tag>-g <it/group/</tag>Update the files according to the group
536 settings for <it/group/ (see section <ref id="configuration-file"
537 name="The Zebra Configuration File">).
539 <tag>-d <it/database/</tag>The records located should be associated
540 with the database name <it/database/ for access through the Z39.50
543 <tag>-m <it/mbytes/</tag>Use <it/mbytes/ of megabytes before flushing
544 keys to background storage. This setting affects performance when
545 updating large databases.
547 <tag>-s</tag>Show analysis of the indexing process. The maintenance
548 program works in a read-only mode and doesn't change the state
549 of the index. This options is very useful when you wish to test a
552 <tag>-V</tag>Show Zebra version.
554 <tag>-v <it/level/</tag>Set the log level to <it/level/. <it/level/
555 should be one of <tt/none/, <tt/debug/, and <tt/all/.
561 <tag>Update <it/directory/</tag>Update the register with the files
562 contained in <it/directory/. If no directory is provided, a list of
563 files is read from <tt/stdin/. See section <ref
564 id="administrating" name="Administrating Zebra">.
568 <sect>The Z39.50 Server
570 <sect1>Running the Z39.50 Server (zebrasrv)
575 zebrasrv [options] [listener-address ...]
580 <tag>-a <it/APDU file/</tag> Specify a file for dumping PDUs (for diagnostic purposes).
581 The special name &dquot;-&dquot; sends output to <tt/stderr/.
583 <tag>-c <it/config-file/</tag> Read configuration information from <it/config-file/. The default configuration is <tt>./zebra.cfg</tt>.
585 <tag/-S/Don't fork on connection requests. This can be useful for
586 symbolic-level debugging. The server can only accept a single
587 connection in this mode.
589 <tag>-l <it/logfile/</tag>Specify an output file for the diagnostic
590 messages. The default is to write this information to <tt/stderr/.
592 <tag>-v <it/log-level/</tag>The log level. Use a comma-separated list of members of the set
593 {fatal,debug,warn,log,all,none}.
595 <tag>-u <it/username/</tag>Set user ID. Sets the real UID of the server process to that of the
596 given <it/username/. It's useful if you aren't comfortable with having the
597 server run as root, but you need to start it as such to bind a
600 <tag>-w <it/working-directory/</tag>Change working directory.
602 <tag>-i</tag>Run under the Internet superserver, <tt/inetd/. Make
603 sure you use the logfile option <tt/-l/ in conjunction with this
604 mode and specify the <tt/-l/ option before any other options.
606 <tag>-t <it/timeout/</tag>Set the idle session timeout (default 60 minutes).
608 <tag>-k <it/kilobytes/</tag>Set the (approximate) maximum size of
609 present response messages. Default is 1024 Kb (1 Mb).
612 A <it/listener-address/ consists of a transport mode followed by a
613 colon (:) followed by a listener address. The transport mode is
614 either <tt/osi/ or <tt/tcp/.
616 For TCP, an address has the form
619 hostname | IP-number [: portnumber]
622 The port number defaults to 210 (standard Z39.50 port).
624 The special hostname &dquot;@&dquot; is mapped to
625 the address INADDR_ANY, which causes the server to listen on any local
626 interface. To start the server listening on the registered port for
627 Z39.50, and to drop root privileges once the
628 port is bound, execute the server like this (from a root shell):
631 zebrasrv -u daemon tcp:@
634 You can replace <tt/daemon/ with another user, eg. your own account, or
635 a dedicated IR server account.
637 The default behavior for <tt/zebrasrv/ is to establish a single TCP/IP
638 listener, for the Z39.50 protocol, on port 9999.
640 <sect1>Z39.50 Protocol Support and Behavior
642 <sect2>Initialization
645 During initialization, the server will negotiate to version 3 of the
646 Z39.50 protocol (unless the client specifies a lower version), and the option bits for Search, Present, Scan,
647 NamedResultSets, and concurrentOperations will be set, if requested by
648 the client. The maximum PDU size is negotiated down to a maximum of
651 <sect2>Search<label id="search">
654 The supported query type are 1 and 101. All operators are currently
655 supported with the restriction that only proximity units of type "word" are
656 supported for the proximity operator.
657 Queries can be arbitrarily complex.
658 Named result sets are supported, and result sets can be used as operands
660 Searches may span multiple databases.
662 The server has full support for piggy-backed present requests (see
663 also the following section).
665 <bf/Use/ attributes are interpreted according to the attribute sets which
666 have been loaded in the <tt/zebra.cfg/ file, and are matched against
667 specific fields as specified in the <tt/.abs/ file which describes the
668 profile of the records which have been loaded. If no <bf/Use/
669 attribute is provided, a default of Bib-1 <bf/Any/ is assumed.
671 If a <bf/Structure/ attribute of <bf/Phrase/ is used in conjunction with a
672 <bf/Completeness/ attribute of <bf/Complete (Sub)field/, the term is
673 matched against the contents of the phrase (long word) register, if one
674 exists for the given <bf/Use/ attribute.
675 A phrase register is created for those fields in the <tt/.abs/
676 file that contains a <tt/p/-specifier.
678 If <bf/Structure/=<bf/Phrase/ is used in conjunction with
679 <bf/Incomplete Field/ - the default value for <bf/Completeness/, the
680 search is directed against the normal word registers, but if the term
681 contains multiple words, the term will only match if all of the words
682 are found immediately adjacent, and in the given order.
683 The word search is performed on those fields that are indexed as
684 type <tt/w/ in the <tt/.abs/ file.
686 If the <bf/Structure/ attribute is <bf/Word List/,
687 <bf/Free-form Text/, or <bf/Document Text/, the term is treated as a
688 natural-language, relevance-ranked query.
689 This search type uses the word register, i.e. those fields
690 that are indexed as type <tt/w/ in the <tt/.abs/ file.
692 If the <bf/Structure/ attribute is <bf/Numeric String/ the
693 term is treated as an integer. The search is performed on those
694 fields that are indexed as type <tt/n/ in the <tt/.abs/ file.
696 If the <bf/Structure/ attribute is <bf/URx/ the
697 term is treated as a URX (URL) entity. The search is performed on those
698 fields that are indexed as type <tt/u/ in the <tt/.abs/ file.
700 If the <bf/Structure/ attribute is <bf/Local Number/ the
701 term is treated as native Zebra Record Identifier.
703 If the <bf/Relation/ attribute is <bf/Equals/ (default), the term is
704 matched in a normal fashion (modulo truncation and processing of
705 individual words, if required). If <bf/Relation/ is <bf/Less Than/,
706 <bf/Less Than or Equal/, <bf/Greater than/, or <bf/Greater than or
707 Equal/, the term is assumed to be numerical, and a standard regular
708 expression is constructed to match the given expression. If
709 <bf/Relation/ is <bf/Relevance/, the standard natural-language query
710 processor is invoked.
712 For the <bf/Truncation/ attribute, <bf/No Truncation/ is the default.
713 <bf/Left Truncation/ is not supported. <bf/Process #/ is supported, as
714 is <bf/Regxp-1/. <bf/Regxp-2/ enables the fault-tolerant (fuzzy)
715 search. As a default, a single error (deletion, insertion,
716 replacement) is accepted when terms are matched against the register
719 <sect3>Regular expressions
722 Each term in a query is interpreted as a regular expression if
723 the truncation value is either <bf/Regxp-1/ (102) or <bf/Regxp-2/ (103).
724 Both query types follow the same syntax with the operands:
726 <tag/x/ Matches the character <it/x/.
727 <tag/./ Matches any character.
728 <tag><tt/[/..<tt/]/</tag> Matches the set of characters specified;
729 such as <tt/[abc]/ or <tt/[a-c]/.
733 <tag/x*/ Matches <it/x/ zero or more times. Priority: high.
734 <tag/x+/ Matches <it/x/ one or more times. Priority: high.
735 <tag/x?/ Matches <it/x/ once or twice. Priority: high.
736 <tag/xy/ Matches <it/x/, then <it/y/. Priority: medium.
737 <tag/x|y/ Matches either <it/x/ or <it/y/. Priority: low.
739 The order of evaluation may be changed by using parentheses.
741 If the first character of the <bf/Regxp-2/ query is a plus character
742 (<tt/+/) it marks the beginning of a section with non-standard
743 specifiers. The next plus character marks the end of the section.
744 Currently Zebra only supports one specifier, the error tolerance,
745 which consists one digit.
747 Since the plus operator is normally a suffix operator the addition to
748 the query syntax doesn't violate the syntax for standard regular
751 <sect3>Query examples
754 Phrase search for <bf/information retrieval/ in the title-register:
756 @attr 1=4 "information retrieval"
759 Ranked search for the same thing:
761 @attr 1=4 @attr 2=102 "Information retrieval"
764 Phrase search with a regular expression:
766 @attr 1=4 @attr 5=102 "informat.* retrieval"
769 Ranked search with a regular expression:
771 @attr 1=4 @attr 5=102 @attr 2=102 "informat.* retrieval"
774 In the GILS schema (<tt/gils.abs/), the west-bounding-coordinate is
775 indexed as type <tt/n/, and is therefore searched by specifying
776 <bf/structure/=<bf/Numeric String/.
777 To match all those records with west-bounding-coordinate greater
778 than -114 we use the following query:
780 @attr 4=109 @attr 2=5 @attr gils 1=2038 -114
785 The present facility is supported in a standard fashion. The requested
786 record syntax is matched against the ones supported by the profile of
787 each record retrieved. If no record syntax is given, SUTRS is the
788 default. The requested element set name, again, is matched against any
789 provided by the relevant record profiles.
794 The attribute combinations provided with the TermListAndStartPoint are
795 processed in the same way as operands in a query (see above).
796 Currently, only the term and the globalOccurrences are returned with
797 the TermInfo structure.
802 Z39.50 specifies three diffent types of sort criterias.
803 Of these Zebra supports the attribute specification type in which
804 case the use attribute specifies the "Sort register".
805 Sort registers are created for those fields that are of type "sort" in
806 the default.idx file.
807 The corresponding character mapping file in default.idx specifies the
808 ordinal of each character used in the actual sort.
810 Z39.50 allows the client to specify sorting on one or more input
811 result sets and one output result set.
812 Zebra supports sorting on one result set only which may or may not
813 be the same as the output result set.
818 If a Close PDU is received, the server will respond with a Close PDU
819 with reason=FINISHED, no matter which protocol version was negotiated
820 during initialization. If the protocol version is 3 or more, the
821 server will generate a Close PDU under certain circumstances,
822 including a session timeout (60 minutes by default), and certain kinds of
823 protocol errors. Once a Close PDU has been sent, the protocol
824 association is considered broken, and the transport connection will be
825 closed immediately upon receipt of further data, or following a short
828 <sect>The Record Model
831 Zebra is designed to support a wide range of data management
832 applications. The system can be configured to handle virtually any
833 kind of structured data. Each record in the system is associated with
834 a <it/record schema/ which lends context to the data elements of the
835 record. Any number of record schema can coexist in the system.
836 Although it may be wise to use only a single schema within
837 one database, the system poses no such restrictions.
839 The record model described in this chapter applies to the fundamental,
841 record type <tt>grs</tt> as introduced in
842 section <ref id="record-types" name="Record Types">.
844 Records pass through three different states during processing in the
848 <item>When records are accessed by the system, they are represented
849 in their local, or native format. This might be SGML or HTML files,
850 News or Mail archives, MARC records. If the system doesn't already
851 know how to read the type of data you need to store, you can set up an
852 input filter by preparing conversion rules based on regular
853 expressions and possibly augmented by a flexible scripting language (Tcl). The input filter
854 produces as output an internal representation:
856 <item>When records are processed by the system, they are represented
857 in a tree-structure, constructed by tagged data elements hanging off a
858 root node. The tagged elements may contain data or yet more tagged
859 elements in a recursive structure. The system performs various
860 actions on this tree structure (indexing, element selection, schema
863 <item>Before transmitting records to the client, they are first
864 converted from the internal structure to a form suitable for exchange
865 over the network - according to the Z39.50 standard.
868 <sect1>Local Representation<label id="local-representation">
871 As mentioned earlier, Zebra places few restrictions on the type of
872 data that you can index and manage. Generally, whatever the form of
873 the data, it is parsed by an input filter specific to that format, and
874 turned into an internal structure that Zebra knows how to handle. This
875 process takes place whenever the record is accessed - for indexing and
879 The RecordType parameter in the <tt/zebra.cfg/ file, or the <tt/-t/
880 option to the indexer tells Zebra how to process input records. Two
881 basic types of processing are available - raw text and structured
882 data. Raw text is just that, and it is selected by providing the
883 argument <bf/text/ to Zebra. Structured records are all handled
884 internally using the basic mechanisms described in the subsequent
885 sections. Zebra can read structured records in many different formats.
886 How this is done is governed by additional parameters after the
887 &dquot;grs&dquot; keyboard, separated by &dquot;.&dquot; characters.
889 Three basic subtypes to the <bf/grs/ type are currently available:
892 <tag>grs.sgml</tag>This is the canonical input format —
893 described below. It is a simple SGML-like syntax.
895 <tag>grs.regx.<it/filter/</tag>This enables a user-supplied input
896 filter. The mechanisms of these filters are described below.
898 <tag>grs.tcl.<it/filter/</tag>This enables a user-supplied input
899 filter with Tcl rules (only availble if zebra is compiled with Tcl
902 <tag>grs.marc.<it/abstract syntax/</tag>This allows Zebra to read
903 records in the ISO2709 (MARC) encoding standard. In this case, the
904 last paramemeter <it/abstract syntax/ names the .abs file (see below)
905 which describes the specific MARC structure of the input record as
906 well as the indexing rules.
909 <sect2>Canonical Input Format
912 Although input data can take any form, it is sometimes useful to
913 describe the record processing capabilities of the system in terms of
914 a single, canonical input format that gives access to the full
915 spectrum of structure and flexibility in the system. In Zebra, this
916 canonical format is an &dquot;SGML-like&dquot; syntax.
918 To use the canonical format specify <tt>grs.sgml</tt> as the record
921 Consider a record describing an information resource (such a record is
922 sometimes known as a <it/locator record/). It might contain a field
923 describing the distributor of the information resource, which might in
924 turn be partitioned into various fields providing details about the
925 distributor, like this:
929 <Name> USGS/WRD &etago;Name>
930 <Organization> USGS/WRD &etago;Organization>
932 U.S. GEOLOGICAL SURVEY, 505 MARQUETTE, NW
933 &etago;Street-Address>
934 <City> ALBUQUERQUE &etago;City>
935 <State> NM &etago;State>
936 <Zip-Code> 87102 &etago;Zip-Code>
937 <Country> USA &etago;Country>
938 <Telephone> (505) 766-5560 &etago;Telephone>
942 <it>NOTE: The indentation used above is used to illustrate how Zebra
943 interprets the markup. The indentation, in itself, has no
944 significance to the parser for the canonical input format, which
945 discards superfluous whitespace.</it>
947 The keywords surrounded by <...> are <it/tags/, while the
948 sections of text in between are the <it/data elements/. A data element
949 is characterized by its location in the tree that is made up by the
950 nested elements. Each element is terminated by a closing tag -
951 beginning with <tt/&etago;/, and containing the same symbolic tag-name as
952 the corresponding opening tag. The general closing tag - <tt/&etago;>/ -
953 terminates the element started by the last opening tag. The
954 structuring of elements is significant. The element <bf/Telephone/,
955 for instance, may be indexed and presented to the client differently,
956 depending on whether it appears inside the <bf/Distributor/ element,
957 or some other, structured data element such a <bf/Supplier/ element.
962 The first tag in a record describes the root node of the tree that
963 makes up the total record. In the canonical input format, the root tag
964 should contain the name of the schema that lends context to the
965 elements of the record (see section <ref id="internal-representation"
966 name="Internal Representation">). The following is a GILS record that
967 contains only a single element (strictly speaking, that makes it an
968 illegal GILS record, since the GILS profile includes several mandatory
969 elements - Zebra does not validate the contents of a record against
970 the Z39.50 profile, however - it merely attempts to match up elements
971 of a local representation with the given schema):
975 <title>Zen and the Art of Motorcycle Maintenance&etago;title>
982 Zebra allows you to provide individual data elements in a number of
983 <it/variant forms/. Examples of variant forms are textual data
984 elements which might appear in different languages, and images which
985 may appear in different formats or layouts. The variant system in
987 essentially a representation of the variant mechanism of
990 The following is an example of a title element which occurs in two
995 <var lang lang "eng">
996 Zen and the Art of Motorcycle Maintenance&etago;>
997 <var lang lang "dan">
998 Zen og Kunsten at Vedligeholde en Motorcykel&etago;>
1002 The syntax of the <it/variant element/ is <tt><<bf/var/ <it/class
1003 type value/></tt>. The available values for the <it/class/ and
1004 <it/type/ fields are given by the variant set that is associated with the
1005 current schema (see section <ref id="variant-set" name="Variant Set
1008 Variant elements are terminated by the general end-tag &etago;>, by
1009 the variant end-tag &etago;var>, by the appearance of another variant
1010 tag with the same <it/class/ and <it/value/ settings, or by the
1011 appearance of another, normal tag. In other words, the end-tags for
1012 the variants used in the example above could have been saved.
1014 Variant elements can be nested. The element
1018 <var lang lang "eng"><var body iana "text/plain">
1019 Zen and the Art of Motorcycle Maintenance
1023 Associates two variant components to the variant list for the title
1026 Given the nesting rules described above, we could write
1030 <var body iana "text/plain>
1031 <var lang lang "eng">
1032 Zen and the Art of Motorcycle Maintenance
1033 <var lang lang "dan">
1034 Zen og Kunsten at Vedligeholde en Motorcykel
1038 The title element above comes in two variants. Both have the IANA body
1039 type &dquot;text/plain&dquot;, but one is in English, and the other in
1040 Danish. The client, using the element selection mechanism of Z39.50,
1041 can retrieve information about the available variant forms of data
1042 elements, or it can select specific variants based on the requirements
1045 <sect2>Input Filters
1048 In order to handle general input formats, Zebra allows the
1049 operator to define filters which read individual records in their native format
1050 and produce an internal representation that the system can
1053 Input filters are ASCII files, generally with the suffix <tt/.flt/.
1054 The system looks for the files in the directories given in the
1055 <bf/profilePath/ setting in the <tt/zebra.cfg/ files. The record type
1056 for the filter is <tt>grs.regx.</tt><it>filter-filename</it>
1057 (fundamental type <tt>grs</tt>, file read type <tt>regx</tt>, argument
1058 <it>filter-filename</it>).
1060 Generally, an input filter consists of a sequence of rules, where each
1061 rule consists of a sequence of expressions, followed by an action. The
1062 expressions are evaluated against the contents of the input record,
1063 and the actions normally contribute to the generation of an internal
1064 representation of the record.
1066 An expression can be either of the following:
1069 <tag/INIT/The action associated with this expression is evaluated
1070 exactly once in the lifetime of the application, before any records
1071 are read. It can be used in conjunction with an action that
1072 initializes tables or other resources that are used in the processing
1075 <tag/BEGIN/Matches the beginning of the record. It can be used to
1076 initialize variables, etc. Typically, the <bf/BEGIN/ rule is also used
1077 to establish the root node of the record.
1079 <tag/END/Matches the end of the record - when all of the contents
1080 of the record has been processed.
1082 <tag>/pattern/</tag>Matches a string of characters from the input
1085 <tag/BODY/This keyword may only be used between two patterns. It
1086 matches everything between (not including) those patterns.
1088 <tag/FINISH/THe expression asssociated with this pattern is evaluated
1089 once, before the application terminates. It can be used to release
1090 system resources - typically ones allocated in the <bf/INIT/ step.
1094 An action is surrounded by curly braces ({...}), and consists of a
1095 sequence of statements. Statements may be separated by newlines or
1096 semicolons (;). Within actions, the strings that matched the
1097 expressions immediately preceding the action can be referred to as
1098 $0, $1, $2, etc.
1100 The available statements are:
1104 <tag>begin <it/type [parameter ... ]/</tag>Begin a new
1105 data element. The type is one of the following:
1107 <tag/record/Begin a new record. The followingparameter should be the
1108 name of the schema that describes the structure of the record, eg.
1109 <tt/gils/ or <tt/wais/ (see below). The <tt/begin record/ call should
1111 any other use of the <bf/begin/ statement.
1113 <tag/element/Begin a new tagged element. The parameter is the
1114 name of the tag. If the tag is not matched anywhere in the tagsets
1115 referenced by the current schema, it is treated as a local string
1118 <tag/variant/Begin a new node in a variant tree. The parameters are
1119 <it/class type value/.
1123 <tag/data/Create a data element. The concatenated arguments make
1124 up the value of the data element. The option <tt/-text/ signals that
1125 the layout (whitespace) of the data should be retained for
1126 transmission. The option <tt/-element/ <it/tag/ wraps the data up in
1127 the <it/tag/. The use of the <tt/-element/ option is equivalent to
1128 preceding the command with a <bf/begin element/ command, and following
1129 it with the <bf/end/ command.
1131 <tag>end <it/[type]/</tag>Close a tagged element. If no parameter is given,
1132 the last element on the stack is terminated. The first parameter, if
1133 any, is a type name, similar to the <bf/begin/ statement. For the
1134 <bf/element/ type, a tag name can be provided to terminate a specific tag.
1138 The following input filter reads a Usenet news file, producing a
1139 record in the WAIS schema. Note that the body of a news posting is
1140 separated from the list of headers by a blank line (or rather a
1141 sequence of two newline characters.
1144 BEGIN { begin record wais }
1146 /^From:/ BODY /$/ { data -element name $1 }
1147 /^Subject:/ BODY /$/ { data -element title $1 }
1148 /^Date:/ BODY /$/ { data -element lastModified $1 }
1150 begin element bodyOfDisplay
1151 begin variant body iana "text/plain"
1157 If Zebra is compiled with support for Tcl (Tool Command Language)
1158 enabled, the statements described above are supplemented with a complete
1159 scripting environment, including control structures (conditional
1160 expressions and loop constructs), and powerful string manipulation
1161 mechanisms for modifying the elements of a record. Tcl is a popular
1162 scripting environment, with several tutorials available both online
1165 <it>NOTE: Variant support is not currently available in the input
1166 filter, but will be included with one of the next
1169 <sect1>Internal Representation<label id="internal-representation">
1172 When records are manipulated by the system, they're represented in a
1173 tree-structure, with data elements at the leaf nodes, and tags or
1174 variant components at the non-leaf nodes. The root-node identifies the
1175 schema that lends context to the tagging and structuring of the
1176 record. Imagine a simple record, consisting of a 'title' element and
1177 an 'author' element:
1180 TITLE "Zen and the Art of Motorcycle Maintenance"
1182 AUTHOR "Robert Pirsig"
1185 A slightly more complex record would have the author element consist
1186 of two elements, a surname and a first name:
1189 TITLE "Zen and the Art of Motorcycle Maintenance"
1196 The root of the record will refer to the record schema that describes
1197 the structuring of this particular record. The schema defines the
1198 element tags (TITLE, FIRST-NAME, etc.) that may occur in the record, as
1199 well as the structuring (SURNAME should appear below AUTHOR, etc.). In
1200 addition, the schema establishes element set names that are used by
1201 the client to request a subset of the elements of a given record. The
1202 schema may also establish rules for converting the record to a
1203 different schema, by stating, for each element, a mapping to a
1206 <sect2>Tagged Elements
1209 A data element is characterized by its tag, and its position in the
1210 structure of the record. For instance, while the tag &dquot;telephone
1211 number&dquot; may be used different places in a record, we may need to
1212 distinguish between these occurrences, both for searching and
1213 presentation purposes. For instance, while the phone numbers for the
1214 &dquot;customer&dquot; and the &dquot;service provider&dquot; are both
1215 representatives for the same type of resource (a telephone number), it
1216 is essential that they be kept separate. The record schema provides
1217 the structure of the record, and names each data element (defined by
1218 the sequence of tags - the tag path - by which the element can be
1219 reached from the root of the record).
1224 The children of a tag node may be either more tag nodes, a data node
1225 (possibly accompanied by tag nodes),
1226 or a tree of variant nodes. The children of variant nodes are either
1227 more variant nodes or a data node (possibly accompanied by more
1228 variant nodes). Each leaf node, which is normally a
1229 data node, corresponds to a <it/variant form/ of the tagged element
1230 identified by the tag which parents the variant tree. The following
1231 title element occurs in two different languages:
1234 VARIANT LANG=ENG "War and Peace"
1236 VARIANT LANG=DAN "Krig og Fred"
1239 Which of the two elements are transmitted to the client by the server
1240 depends on the specifications provided by the client, if any.
1242 In practice, each variant node is associated with a triple of class,
1243 type, value, corresponding to the variant mechanism of Z39.50.
1245 <sect2>Data Elements
1248 Data nodes have no children (they are always leaf nodes in the record
1251 <it>NOTE: Documentation needs extension here about types of nodes - numerical,
1252 textual, etc., plus the various types of inclusion notes.</it>
1254 <sect1>Configuring Your Data Model<label id="data-model">
1257 The following sections describe the configuration files that govern
1258 the internal management of data records. The system searches for the files
1259 in the directories specified by the <bf/profilePath/ setting in the
1260 <tt/zebra.cfg/ file.
1262 <sect2>About Object Identifers
1264 When Object Identifiers (or OID's) need to be specified in the following
1265 a named OID reference or a raw OID reference may be used. For the named
1266 OID's refer to the source file <tt>util/oid.c</tt> from YAZ. The raw
1267 canonical OID's are specified in dot-notation (for example
1268 1.2.840.10003.3.1000.81.1).
1270 <sect2>The Abstract Syntax
1273 The abstract syntax definition (also known as an Abstract Record
1274 Structure, or ARS) is the focal point of the
1275 record schema description. For a given schema, the ABS file may state any
1276 or all of the following:
1279 <item>The object identifier of the Z39.50 schema associated
1280 with the ARS, so that it can be referred to by the client.
1282 <item>The attribute set (which can possibly be a compound of multiple
1283 sets) which applies in the profile. This is used when indexing and
1284 searching the records belonging to the given profile.
1286 <item>The Tag set (again, this can consist of several different sets).
1287 This is used when reading the records from a file, to recognize the
1288 different tags, and when transmitting the record to the client -
1289 mapping the tags to their numerical representation, if they are
1292 <item>The variant set which is used in the profile. This provides a
1293 vocabulary for specifying the <it/forms/ of data that appear inside
1296 <item>Element set names, which are a shorthand way for the client to
1297 ask for a subset of the data elements contained in a record. Element
1298 set names, in the retrieval module, are mapped to <it/element
1299 specifications/, which contain information equivalent to the
1300 <it/Espec-1/ syntax of Z39.50.
1302 <item>Map tables, which may specify mappings to <it/other/ database
1303 profiles, if desired.
1305 <item>Possibly, a set of rules describing the mapping of elements to a
1306 MARC representation.
1308 <item>A list of element descriptions (this is the actual ARS of the
1309 schema, in Z39.50 terms), which lists the ways in which the various
1310 tags can be used and organized hierarchically.
1313 Several of the entries above simply refer to other files, which
1314 describe the given objects.
1316 <sect2>The Configuration Files
1319 This section describes the syntax and use of the various tables which
1320 are used by the retrieval module.
1322 The number of different file types may appear daunting at first, but
1323 each type corresponds fairly clearly to a single aspect of the Z39.50
1324 retrieval facilities. Further, the average database administrator,
1325 who is simply reusing an existing profile for which tables already
1326 exist, shouldn't have to worry too much about the contents of these tables.
1328 Generally, the files are simple ASCII files, which can be maintained
1329 using any text editor. Blank lines, and lines beginning with a (#) are
1330 ignored. Any characters on a line followed by a (#) are also ignored.
1332 lines contain <it/directives/, which provide some setting or value
1333 to the system. Generally, settings are characterized by a single
1334 keyword, identifying the setting, followed by a number of parameters.
1335 Some settings are repeatable (r), while others may occur only once in a
1336 file. Some settings are optional (o), whicle others again are
1339 <sect2>The Abstract Syntax (.abs) Files
1342 The name of this file type is slightly misleading in Z39.50 terms,
1343 since, apart from the actual abstract syntax of the profile, it also
1344 includes most of the other definitions that go into a database
1347 When a record in the canonical, SGML-like format is read from a file
1348 or from the database, the first tag of the file should reference the
1349 profile that governs the layout of the record. If the first tag of the
1350 record is, say, <tt><gils></tt>, the system will look for the profile
1351 definition in the file <tt/gils.abs/. Profile definitions are cached,
1352 so they only have to be read once during the lifespan of the current
1355 When writing your own input filters, the <bf/record-begin/ command
1356 introduces the profile, and should always be called first thing when
1357 introducing a new record.
1359 The file may contain the following directives:
1362 <tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or
1363 description for the profile. Mostly useful for diagnostic purposes.
1365 <tag>reference <it/OID-name/</tag> (m) The OID for
1366 the profile (name or dotted-numerical list).
1368 <tag>attset <it/filename/</tag> (m) The attribute set that is used for
1369 indexing and searching records belonging to this profile.
1371 <tag>tagset <it/filename/ [<it/type/]</tag> (o) The tag
1372 set (if any) that describe that fields of the records. The type, which
1373 is optional, specifies the tag type. If not given, the type-specifier
1374 in the Tag Set files is used.
1376 <tag>varset <it/filename/</tag> (o) The variant set used in the profile.
1378 <tag>maptab <it/filename/</tag> (o,r) This points to a
1379 conversion table that might be used if the client asks for the record
1380 in a different schema from the native one.
1382 <tag>marc <it/filename/</tag> (o) Points to a file containing parameters
1383 for representing the record contents in the ISO2709 syntax. Read the
1384 description of the MARC representation facility below.
1386 <tag>esetname <it/name filename/</tag> (o,r) Associates the
1387 given element set name with an element selection file. If an (@) is
1388 given in place of the filename, this corresponds to a null mapping for
1389 the given element set name.
1391 <tag>any <it/tags/</tag> (o) This directive specifies a list of
1392 attributes which should be appended to the attribute list given for each
1393 element. The effect is to make every single element in the abstract
1394 syntax searchable by way of the given attributes. This directive
1395 provides an efficient way of supporting free-text searching across all
1396 elements. However, it does increase the size of the index
1397 significantly. The attributes can be qualified with a structure, as in
1398 the <bf/elm/ directive below.
1400 <tag>elm <it/path name attributes/</tag> (o,r) Adds an element
1401 to the abstract record syntax of the schema. The <it/path/ follows the
1402 syntax which is suggested by the Z39.50 document - that is, a sequence
1403 of tags separated by slashes (/). Each tag is given as a
1404 comma-separated pair of tag type and -value surrounded by parenthesis.
1405 The <it/name/ is the name of the element, and the <it/attributes/
1406 specifies which attributes to use when indexing the element in a
1407 comma-separated list. A ! in
1408 place of the attribute name is equivalent to specifying an attribute
1409 name identical to the element name. A - in place of the attribute name
1410 specifies that no indexing is to take place for the given element. The
1411 attributes can be qualified with <it/field types/ to specify which
1412 character set should govern the indexing procedure for that field. The
1413 same data element may be indexed into several different fields, using
1414 different character set definitions. See the section
1415 <ref id="field structure and character sets"
1416 name="Field Structure and Character Sets">.
1417 The default field type is &dquot;w&dquot; for
1421 The following is an excerpt from the abstract syntax file for the GILS
1426 reference GILS-schema
1431 maptab gils-usmarc.map
1435 esetname VARIANT gils-variant.est # for WAIS-compliance
1436 esetname B gils-b.est
1437 esetname G gils-g.est
1442 elm (1,14) localControlNumber Local-number
1443 elm (1,16) dateOfLastModification Date/time-last-modified
1444 elm (2,1) Title w:!,p:!
1445 elm (4,1) controlIdentifier Identifier-standard
1446 elm (2,6) abstract Abstract
1447 elm (4,51) purpose !
1448 elm (4,52) originator -
1449 elm (4,53) accessConstraints !
1450 elm (4,54) useConstraints !
1451 elm (4,70) availability -
1452 elm (4,70)/(4,90) distributor -
1453 elm (4,70)/(4,90)/(2,7) distributorName !
1454 elm (4,70)/(4,90)/(2,10 distributorOrganization !
1455 elm (4,70)/(4,90)/(4,2) distributorStreetAddress !
1456 elm (4,70)/(4,90)/(4,3) distributorCity !
1459 <sect2>The Attribute Set (.att) Files<label id="attset-files">
1462 This file type describes the <bf/Use/ elements of an attribute set.
1463 It contains the following directives.
1467 <tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or
1468 description for the attribute set. Mostly useful for diagnostic purposes.
1470 <tag>reference <it/OID-name/</tag> (m) The reference name of the OID for
1473 <tag>include <it/filename/</tag> (o,r) This directive is used to
1474 include another attribute set as a part of the current one. This is
1475 used when a new attribute set is defined as an extension to another
1476 set. For instance, many new attribute sets are defined as extensions
1477 to the <bf/bib-1/ set. This is an important feature of the retrieval
1478 system of Z39.50, as it ensures the highest possible level of
1479 interoperability, as those access points of your database which are
1480 derived from the external set (say, bib-1) can be used even by clients
1481 who are unaware of the new set.
1483 <tag>att <it/att-value att-name [local-value]/</tag> (o,r) This
1484 repeatable directive introduces a new attribute to the set. The
1485 attribute value is stored in the index (unless a <it/local-value/ is
1486 given, in which case this is stored). The name is used to refer to the
1487 attribute from the <it/abstract syntax/. </descrip>
1489 This is an excerpt from the GILS attribute set definition. Notice how
1490 the file describing the <it/bib-1/ attribute set is referenced.
1494 reference GILS-attset
1497 att 2001 distributorName
1498 att 2002 indexTermsControlled
1500 att 2004 accessConstraints
1501 att 2005 useConstraints
1504 <sect2>The Tag Set (.tag) Files
1507 This file type defines the tagset of the profile, possibly by
1508 referencing other tag sets (most tag sets, for instance, will include
1509 tagsetG and tagsetM from the Z39.50 specification. The file may
1510 contain the following directives.
1513 <tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or
1514 description for the tag set. Mostly useful for diagnostic purposes.
1516 <tag>reference <it/OID-name/</tag> (o) The reference name of the OID for
1517 the tag set. The directive is optional, since not all tag sets are
1518 registered outside of their schema.
1520 <tag>type <it/integer/</tag> (m) The type number of the tagset within the schema
1521 profile (note: this specification really should belong to the .abs
1522 file. This will be fixed in a future release).
1524 <tag>include <it/filename/</tag> (o,r) This directive is used
1525 to include the definitions of other tag sets into the current one.
1527 <tag>tag <it/number names type/</tag> (o,r) Introduces a new
1528 tag to the set. The <it/number/ is the tag number as used in the protocol
1529 (there is currently no mechanism for specifying string tags at this
1530 point, but this would be quick work to add). The <it/names/ parameter
1531 is a list of names by which the tag should be recognized in the input
1532 file format. The names should be separated by slashes (/). The
1533 <it/type/ is th recommended datatype of the tag. It should be one of
1541 <item>generalizedtime
1549 The following is an excerpt from the TagsetG definition file.
1558 tag 3 publicationPlace string
1559 tag 4 publicationDate string
1560 tag 5 documentId string
1561 tag 6 abstract string
1563 tag 8 date generalizedtime
1564 tag 9 bodyOfDisplay string
1565 tag 10 organization string
1568 <sect2>The Variant Set (.var) Files<label id="variant-set">
1571 The variant set file is a straightforward representation of the
1572 variant set definitions associated with the protocol. At present, only
1573 the <it/Variant-1/ set is known.
1575 These are the directives allowed in the file.
1578 <tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or
1579 description for the variant set. Mostly useful for diagnostic purposes.
1581 <tag>reference <it/OID-name/</tag> (o) The reference name of the OID for
1582 the variant set, if one is required.
1584 <tag>class <it/integer class-name/</tag> (m,r) Introduces a new
1585 class to the variant set.
1587 <tag>type <it/integer type-name datatype/</tag> (m,r) Addes a
1588 new type to the current class (the one introduced by the most recent
1589 <bf/class/ directive). The type names belong to the same name space as
1590 the one used in the tag set definition file.
1593 The following is an excerpt from the file describing the variant set
1602 type 1 variantId octetstring
1607 type 2 z39.50 string
1611 <sect2>The Element Set (.est) Files
1614 The element set specification files describe a selection of a subset
1615 of the elements of a database record. The element selection mechanism
1616 is equivalent to the one supplied by the <it/Espec-1/ syntax of the
1617 Z39.50 specification. In fact, the internal representation of an
1618 element set specification is identical to the <it/Espec-1/ structure,
1619 and we'll refer you to the description of that structure for most of
1620 the detailed semantics of the directives below.
1623 NOTE: Not all of the Espec-1 functionality has been implemented yet.
1624 The fields that are mentioned below all work as expected, unless
1628 The directives available in the element set file are as follows:
1631 <tag>defaultVariantSetId <it/OID-name/</tag> (o) If variants are used in
1632 the following, this should provide the name of the variantset used
1633 (it's not currently possible to specify a different set in the
1634 individual variant request). In almost all cases (certainly all
1635 profiles known to us), the name <tt/Variant-1/ should be given here.
1637 <tag>defaultVariantRequest <it/variant-request/</tag> (o) This directive
1638 provides a default variant request for
1639 use when the individual element requests (see below) do not contain a
1640 variant request. Variant requests consist of a blank-separated list of
1641 variant components. A variant compont is a comma-separated,
1642 parenthesized triple of variant class, type, and value (the two former
1643 values being represented as integers). The value can currently only be
1644 entered as a string (this will change to depend on the definition of
1645 the variant in question). The special value (@) is interpreted as a
1646 null value, however.
1648 <tag>simpleElement <it/path ['variant' variant-request]/</tag>
1649 (o,r) This corresponds to a simple element request in <it/Espec-1/. The
1650 path consists of a sequence of tag-selectors, where each of these can
1654 <item>A simple tag, consisting of a comma-separated type-value pair in
1655 parenthesis, possibly followed by a colon (:) followed by an
1656 occurrences-specification (see below). The tag-value can be a number
1657 or a string. If the first character is an apostrophe ('), this forces
1658 the value to be interpreted as a string, even if it appears to be numerical.
1660 <item>A WildThing, represented as a question mark (?), possibly
1661 followed by a colon (:) followed by an occurrences specification (see
1664 <item>A WildPath, represented as an asterisk (*). Note that the last
1665 element of the path should not be a wildPath (wildpaths don't work in
1669 The occurrences-specification can be either the string <tt/all/, the
1670 string <tt/last/, or an explicit value-range. The value-range is
1671 represented as an integer (the starting point), possibly followed by a
1672 plus (+) and a second integer (the number of elements, default being
1675 The variant-request has the same syntax as the defaultVariantRequest
1676 above. Note that it may sometimes be useful to give an empty variant
1677 request, simply to disable the default for a specific set of fields
1678 (we aren't certain if this is proper <it/Espec-1/, but it works in
1679 this implementation).
1682 The following is an example of an element specification belonging to
1686 simpleelement (1,10)
1687 simpleelement (1,12)
1689 simpleelement (1,14)
1691 simpleelement (4,52)
1694 <sect2>The Schema Mapping (.map) Files<label id="schema-mapping">
1697 Sometimes, the client might want to receive a database record in
1698 a schema that differs from the native schema of the record. For
1699 instance, a client might only know how to process WAIS records, while
1700 the database record is represented in a more specific schema, such as
1701 GILS. In this module, a mapping of data to one of the MARC formats is
1702 also thought of as a schema mapping (mapping the elements of the
1703 record into fields consistent with the given MARC specification, prior
1704 to actually converting the data to the ISO2709). This use of the
1705 object identifier for USMARC as a schema identifier represents an
1706 overloading of the OID which might not be entirely proper. However,
1707 it represents the dual role of schema and record syntax which
1708 is assumed by the MARC family in Z39.50.
1711 NOTE: The schema-mapping functions are so far limited to a
1712 straightforward mapping of elements. This should be extended with
1713 mechanisms for conversions of the element contents, and conditional
1714 mappings of elements based on the record contents.
1717 These are the directives of the schema mapping file format:
1720 <tag>targetName <it/name/</tag> (m) A symbolic name for the target schema
1721 of the table. Useful mostly for diagnostic purposes.
1723 <tag>targetRef <it/OID-name/</tag> (m) An OID name for the target schema.
1724 This is used, for instance, by a server receiving a request to present
1725 a record in a different schema from the native one.
1727 <tag>map <it/element-name target-path/</tag> (o,r) Adds
1728 an element mapping rule to the table.
1731 <sect2>The MARC (ISO2709) Representation (.mar) Files
1734 This file provides rules for representing a record in the ISO2709
1735 format. The rules pertain mostly to the values of the constant-length
1736 header of the record.
1738 <it>NOTE: This will be described better. We're in the process of
1739 re-evaluating and most likely changing the way that MARC records are
1740 handled by the system.</it>
1742 <sect2>Field Structure and Character Sets
1743 <label id="field structure and character sets">
1746 In order to provide a flexible approach to national character set
1747 handling, Zebra allows the administrator to configure the set up the
1748 system to handle any 8-bit character set — including sets that
1749 require multi-octet diacritics or other multi-octet characters. The
1750 definition of a character set includes a specification of the
1751 permissible values, their sort order (this affects the display in the
1752 SCAN function), and relationships between upper- and lowercase
1753 characters. Finally, the definition includes the specification of
1754 space characters for the set.
1756 The operator can define different character sets for different fields,
1757 typical examples being standard text fields, numerical fields, and
1758 special-purpose fields such as WWW-style linkages (URx).
1760 The field types, and hence character sets, are associated with data
1761 elements by the .abs files (see above). The file <tt/default.idx/
1762 provides the association between field type codes (as used in the .abs
1763 files) and the character map files (with the .chr suffix). The format
1764 of the .idx file is as follows
1767 <tag>index <it/field type code/</tag>This directive introduces a new
1768 search index code. The argument is a one-character code to be used in the
1769 .abs files to select this particular index type. An index, roughly,
1770 corresponds to a particular structure attribute during search. Refer
1771 to section <ref id="search" name="Search">.
1773 <tag>sort <it/field code type/</tag>This directive introduces a
1774 sort index. The argument is a one-character code to be used in the
1775 .abs fie to select this particular index type. The corresponding
1776 use attribute must be used in the sort request to refer to this
1777 particular sort index. The corresponding character map (see below)
1778 is used in the sort process.
1780 <tag>completeness <it/boolean/</tag>This directive enables or disables
1781 complete field indexing. The value of the <it/boolean/ should be 0
1782 (disable) or 1. If completeness is enabled, the index entry will
1783 contain the complete contents of the field (up to a limit), with words
1784 (non-space characters) separated by single space characters
1785 (normalized to &dquot; &dquot; on display). When completeness is
1786 disabled, each word is indexed as a separate entry. Complete subfield
1787 indexing is most useful for fields which are typically browsed (eg.
1788 titles, authors, or subjects), or instances where a match on a
1789 complete subfield is essential (eg. exact title searching). For fields
1790 where completeness is disabled, the search engine will interpret a
1791 search containing space characters as a word proximity search.
1793 <tag>charmap <it/filename/</tag> This is the filename of the character
1794 map to be used for this index for field type.
1797 The contents of the character map files are structured as follows:
1800 <tag>lowercase <it/value-set/</tag>This directive introduces the basic
1801 value set of the field type. The format is an ordered list (without
1802 spaces) of the characters which may occur in &dquot;words&dquot; of
1803 the given type. The order of the entries in the list determines the
1804 sort order of the index. In addition to single characters, the
1805 following combinations are legal:
1808 <item>Backslashes may be used to introduce three-digit octal, or
1809 two-digit hex representations of single characters (preceded by <tt/x/).
1810 In addition, the combinations
1811 \\, \\r, \\n, \\t, \\s (space — remember that real space-characters
1812 may ot occur in the value definition), and \\ are recognised,
1813 with their usual interpretation.
1815 <item>Curly braces {} may be used to enclose ranges of single
1816 characters (possibly using the escape convention described in the
1817 preceding point), eg. {a-z} to entroduce the standard range of ASCII
1818 characters. Note that the interpretation of such a range depends on
1819 the concrete representation in your local, physical character set.
1821 <item>Paranthesises () may be used to enclose multi-byte characters -
1822 eg. diacritics or special national combinations (eg. Spanish
1823 &dquot;ll&dquot;). When found in the input stream (or a search term),
1824 these characters are viewed and sorted as a single character, with a
1825 sorting value depending on the position of the group in the value
1829 <tag>uppercase <it/value-set/</tag>This directive introduces the
1830 upper-case equivalencis to the value set (if any). The number and
1831 order of the entries in the list should be the same as in the
1832 <tt/lowercase/ directive.
1834 <tag>space <it/value-set/</tag>This directive introduces the character
1835 which separate words in the input stream. Depending on the
1836 completeness mode of the field in question, these characters either
1837 terminate an index entry, or delimit individual &dquot;words&dquot; in
1838 the input stream. The order of the elements is not significant —
1839 otherwise the representation is the same as for the <tt/upercase/ and
1840 <tt/lowercase/ directives.
1842 <tag>map <it/value-set/ <it/target/</tag>This directive introduces a
1843 mapping between each of the members of the value-set on the left to
1844 the character on the right. The character on the right must occur in
1845 the value set (the <tt/lowercase/ directive) of the character set, but
1846 it may be a paranthesis-enclosed multi-octet character. This directive
1847 may be used to map diacritics to their base characters, or to map
1848 HTML-style character-representations to their natural form, etc.
1851 <sect1>Exchange Formats
1854 Converting records from the internal structure to en exchange format
1855 is largely an automatic process. Currently, the following exchange
1856 formats are supported:
1859 <item>GRS-1. The internal representation is based on GRS-1, so the
1860 conversion here is straightforward. The system will create
1861 applied variant and supported variant lists as required, if a record
1862 contains variant information.
1864 <item>SUTRS. Again, the mapping is fairly straighforward. Indentation
1865 is used to show the hierarchical structure of the record. All
1866 &dquot;GRS&dquot; type records support both the GRS-1 and SUTRS
1869 <item>ISO2709-based formats (USMARC, etc.). Only records with a
1870 two-level structure (corresponding to fields and subfields) can be
1871 directly mapped to ISO2709. For records with a different structuring
1872 (eg., GILS), the representation in a structure like USMARC involves a
1873 schema-mapping (see section <ref id="schema-mapping" name="Schema
1874 Mapping">), to an &dquot;implied&dquot; USMARC schema (implied,
1875 because there is no formal schema which specifies the use of the
1876 USMARC fields outside of ISO2709). The resultant, two-level record is
1877 then mapped directly from the internal representation to ISO2709. See
1878 the GILS schema definition files for a detailed example of this
1881 <item>Explain. This representation is only available for records
1882 belonging to the Explain schema.
1884 <item>Summary. This ASN-1 based structure is only available for records
1885 belonging to the Summary schema - or schema which provide a mapping
1886 to this schema (see the description of the schema mapping facility
1889 <item>SOIF. Support for this syntax is experimental, and is currently
1890 keyed to a private Index Data OID (1.2.840.10003.5.1000.81.2). All
1891 abstract syntaxes can be mapped to the SOIF format, although nested
1892 elements are represented by concatenation of the tag names at each
1895 <item>XML. The use of XML as a transfer syntax in Z39.50 is not yet widely established
1896 so the use of it here must be characterised as somewhat experimental. The
1897 tag-names used are taken from the tag-set in use, except for local string tags
1898 where the tag itself is passed through unchanged.
1906 Copyright (c) 1995-1999 Index Data ApS.
1908 All rights reserved.
1910 Use and redistribution in source or binary form, with or without
1911 modification, of any or all of this software and documentation is
1912 permitted, provided that the following Conditions 1 to 6 set out below
1915 1. Unless prior specific written permission is obtained this copyright
1916 and permission notice appear with all copies of the software and its
1917 documentation. Notices of copyright or attribution which appear at the
1918 beginning of any file must remain unchanged.
1920 2. The names of Index Data or the individual authors may not be used
1921 to endorse or promote products derived from this software without
1922 specific prior written permission.
1924 3. Source code or binary versions of this software and its documentation
1925 may be used freely in not for profit applications limited to databases
1926 of 100,000 records maximum. Other applications - such as publishing over
1927 100,000 records, providing for-pay services, distributing a product based
1928 in whole or in part on this software or its documentation, or generally
1929 distributing this software or its documentation under a different license
1930 require a commercial license from Index Data.
1932 4. The software may be installed and used for evaluation purposes in
1933 conjunction with such commercially licensed applications for a trial
1934 period no longer than 60 days.
1936 5. Unless a prior specific written agreement is obtained THIS SOFTWARE
1937 IS PROVIDED "AS IS" AND WITHOUT WARRANTY OF ANY KIND, EXPRESS, IMPLIED,
1938 OR OTHERWISE, INCLUDING WITHOUT LIMITATION, ANY WARRANTY OF
1939 MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. IN NO EVENT SHALL
1940 INDEX DATA BE LIABLE FOR ANY SPECIAL, INCIDENTAL, INDIRECT OR
1941 CONSEQUENTIAL DAMAGES OF ANY KIND, OR ANY DAMAGES WHATSOEVER RESULTING
1942 FROM LOSS OF USE, DATA OR PROFITS, WHETHER OR NOT ADVISED OF THE
1943 POSSIBILITY OF DAMAGE, AND ON ANY THEORY OF LIABILITY, ARISING OUT OF
1944 OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
1946 6. Commercial licenses and support agreements for Zebra and related
1947 Index Data products such as Z'bol (c) - and written agreements
1948 relating to these Conditions may be obtained only from Index Data
1949 or its appointed agents as follows:
1951 Index Data: www.indexdata.dk
1952 Fretwell-Downing Informatics: www.fdgroup.co.uk
1953 Fretwell-Downing Informatics USA: www.fdi.com
1955 <sect>About Index Data and the Zebra Server
1958 Index Data is a consulting and software-development enterprise that
1959 specialises in information management and retrieval applications. Our
1960 interests and expertise span a broad range of related fields, and one
1961 of our primary, long-term objectives is the development of a powerful
1962 information management
1963 system with open network interfaces and hypermedia capabilities. Zebra is an
1964 important component in this strategy.
1966 We make this software available free of charge for not-for-profit
1967 purposes, as a service to the networking community, and to further
1968 the development and use of quality software for open network
1969 communication. We encourage your comments and questions if you have ideas, things
1970 you would like to see in future versions, or things you would like to
1973 If you like this software, and would like to use all or part of it in
1974 a commercial product, or to provide a commercial database service,
1975 please contact us. The Z'mbol Information System represents the commercial
1976 variant of Zebra. It includes full support; additional functionality and
1977 performance-boosting features, and it has what we think is a very exciting
1983 DK-2200 Copenhagen N
1988 Phone: +45 3536 3672
1990 Email: info@indexdata.dk
1993 The <it>Random House College Dictionary</it>, 1975 edition
1994 offers this definition of the
1995 word &dquot;Zebra&dquot;:
1998 Zebra, n., any of several horselike, African mammals of the genus Equus,
1999 having a characteristic pattern of black or dark-brown stripes on
2000 a whitish background.