1 <!doctype linuxdoc system>
4 $Id: zebra.sgml,v 1.46 2000-02-10 10:23:34 adam Exp $
8 <title>Zebra Server - Administrators's Guide and Reference
9 <author><htmlurl url="http://www.indexdata.dk/" name="Index Data">,
10 <tt><htmlurl url="mailto:info@indexdata.dk" name="info@indexdata.dk"></>
11 <date>$Revision: 1.46 $
15 The Zebra server combines a versatile fielded/free-text
16 indexing/search engine with a Z39.50-1995 frontend to provide a powerful and flexible
17 information mining tool. This document explains the procedure for
18 installing and configuring Zebra, and outlines the possibilities
19 for managing data and providing Z39.50
20 services with the software. Zebra is a free version of the Index Data Z'mbol
21 information system, and it excludes some functionality such as incremental
22 database updating and support for large databases.
32 Zebra is a fielded free-text indexing and retrieval engine with a
33 Z39.50 frontend. You can use any commercial or freeware Z39.50 client
34 to access data stored in Zebra.
36 Zebra server can be used at the core of a Z39.50-based information retrieval
37 framework. We're making
38 the server available now to allow researchers and small organisations to
39 share their information in the best possible way. We believe that Z39.50
40 currently represents one of the best ways of sharing information with others, and
41 we would like to encourage as many people as possible to do so.
42 This document is a guide to using Zebra. It will tell you
43 how to compile the software, and how to prepare your first database.
44 It also explains how the server can be configured to give you the
45 functionality that you need.
47 If you find the software interesting, you should join the support
48 mailing-list by sending email to <tt/zebra-request@indexdata.dk/.
50 If you are interested in running a commercial service, if you wish to run large
51 databases, or if you wish to make incremental updates to your databases even
52 while users are accessing your system, then you might be interested in the Z'mbol
53 Information Server which is available from <htmlurl
54 url="http://www.indexdata.dk/zmbol/" name="Index Data"> or Fretwell-Downing
55 Informatics. Z'mbol is a complete and supported package which offers many
56 exciting possibilities that we have not been able to fit into this package.
61 This is a list of some of the most important features of the
67 Supports arbitrarily complex records - base input format is an
68 XML-like syntax which allows nested (structured) data elements, as
69 well as variant forms of data.
72 Supports random storage formats. A system of input filters driven by
73 regular expressions allows you to easily process most ASCII-based
74 data formats. SGML/XML, ISO2709 (MARC), and raw text are also supported.
77 Supports boolean queries as well as relevance-ranking (free-text)
78 searching. Right truncation and masking in terms are supported, as
79 well as full regular expressions.
82 Supports multiple concrete syntaxes
83 for record exchange (depending on the configuration): GRS-1, SUTRS,
84 ISO2709 (*MARC), XML. Records can be mapped between record syntaxes and
88 Supports approximate matching in registers (ie. spelling mistakes,
91 <item> Supports a subset of the Z39.50 Explain Facility. Zebra's Explain database
92 is automatically updated when a set of records is loaded into Zebra.
102 Protocol facilities: Init, Search, Retrieve, Browse, Sort, Close, and Explain.
105 Piggy-backed presents are honored in the search-request.
108 Named result sets are supported.
111 Easily configured to support different application profiles, with
112 tables for attribute sets, tag sets, and abstract syntaxes.
113 Additional tables control facilities such as element mappings to
114 different schema (eg., GILS-to-USMARC).
117 Complex composition specifications using Espec-1 are partially
118 supported (simple element requests only).
121 Element Set Names are defined using the Espec-1 capability of the
122 system, and are given in configuration files as simple element
123 requests (and possibly variant requests).
126 Zebra runs on most Unix-like systems as well as Windows NT - a binary
127 distribution for Windows NT is forthcoming - so far, the installation
128 requires Microsoft Visual C++ to compile the system (we use version 6.0).
136 These are some of the plans that we have for the software in the near
137 and far future, approximately ordered after their relative importance.
139 asterisk will be implemented before the
145 *Complete the support for variants.
148 *Finalize the data element <it/include/ facility to support multimedia
149 data elements in records.
152 Add more sophisticated relevance ranking mechanisms. Add support for soundex
153 and stemming. Add relevance <it/feedback/ support.
156 Complete EXPLAIN support.
159 We want to add a management system that allows you to
160 control your databases and configuration tables from a graphical
161 interface. We'll probably use Tcl/Tk to stay platform-independent.
165 Programmers thrive on user feedback. If you are interested in a facility that
166 you don't see mentioned here, or if there's something you think we
167 could do better, please drop us a mail. If you think it's all really
168 neat, you're welcome to drop us a line saying that, too. You'll find
169 contact info at the end of this file.
171 <sect>Compiling the software
174 <bf><htmlurl url="http://www.indexdata.dk/yaz/" name="YAZ"></>
175 package in order to compile this software. We suggest you
176 unpack <bf/YAZ/ in the same directory as Zebra. Running
177 ./configure (UNIX Only) and running make (nmake on WIN32) is
178 in usully what it takes to compile YAZ.
182 An ANSI C compiler is required to compile the Zebra
183 server system — <tt/gcc/ works very well if your own system doesn't
184 provide an adequate compiler.
186 Unpack the distribution archive. The <tt>configure</tt> shell script
187 attempts to guess correct values for various system-dependent variables
188 used during compilation. It uses those values to create a 'Makefile' in
189 each directory of Zebra.
191 To run the configure script type:
196 The configure script attempts to use the C compiler specified by
197 the <tt>CC</tt> environment variable. If not set, GNU C
198 will be used if it is available. The <tt>CFLAGS</tt> environment variable
199 holds options to be passed to the C compiler. If you're using a
200 Bourne-compatible shell you may pass something like this:
202 CC=/opt/ccs/bin/cc CFLAGS=-O ./configure
205 To customize Zebra the configure script accepts a set of options. The
208 <tag><tt>--prefix </tt>path</tag> Specifies installation prefix. This is
209 only needed if you run <tt>make install</tt> later to perform a
210 "system" installation. The prefix is <tt>/usr/local</tt> if not
212 <tag><tt>--with-tclconfig </tt>path</tag> If Tcl is installed on
213 the system you can tell configure where Tcl's <tt>tclConfig.sh</tt>
214 installed. The <tt>tclConfig.sh</tt> include information about settings
215 required to link with Tcl's libraries. If you don't specify this
216 option, configure will see if Tcl's shell <tt>tclsh</tt> is in your
217 path and if it is, it will guess where the equivalent tclConfig.sh
218 is located. If tclsh is not found in your path and this option is not
219 given Zebra will not include Tcl support.
220 <tag><tt>--with-yazconfig </tt>path</tag> This options allows you to
221 specify the path of YAZ's <tt>yaz-config</tt>. Therefore this option
222 forces Zebra to use a particular version of YAZ. YAZ version 1.5 and
223 later creates a script <tt>yaz-config</tt> that includes information
224 on compiler settings needed to link with it.
227 When configured build the software by typing:
232 As an option you may type <tt>make depend</tt> to create
233 source file dependencies for the package. This is only needed,
234 however, if you modify the source code later.
236 If successful, two executables have been created in the sub-directory
239 <tag><tt>zebrasrv</tt></tag> The Z39.50 server and search engine.
240 <tag><tt>zebraidx</tt></tag> The administrative tool for the search index.
244 The next step is optional and is only needed if you wish to install
245 zebra in system directories such as /usr/bin, /usr/lib, etc.
247 To perform this step, type
252 The executables will be installed in prefix/bin, and profile
253 tables will be installed in prefix/lib/zebra/tab. Here prefix
254 represents the prefix as specified -- default being /usr/local.
259 Zebra is shipped with "makefiles" for the NMAKE tool that comes
262 Start an MS-DOS prompt and switch the sub directory <tt>WIN</tt> where
263 the file <tt>makefile</tt> is located. Customize the installation
264 by editing the <tt>makefile</tt> file (for example by using wordpad).
266 The following summarises the most important settings in that file.
269 <tag><tt>YAZDIR</tt></tag> Specifies where YAZ is located.
270 <tag><tt>DEBUG</tt></tag> If set to 1, the software is
271 compiled with debugging libraries. If set to 0, the software
272 is compiled with release (non-debugging) libraries.
273 <tag>BZIP2</tag> A group of settings (<tt>BZIP2LIB</tt>,..)
274 that must be defined if BZIP2 compression support is desired.
277 When satisfied with the settings in the makefile type
282 If compilation was successful the executables <tt>zebraidx.exe</tt>
283 and <tt>zebrasrv.exe</tt> are put in the sub directory <tt>BIN</tt>.
287 In this section, we will test the system by indexing a small set of sample
288 GILS records that are included with the software distribution. Go to the
289 <tt>test/gils</tt> subdirectory of the distribution archive. There you will
291 file named <tt>zebra.cfg</tt> with the following contents:
293 # Where the schema files, attribute files, etc. are located.
294 profilePath: .:../../tab:../../../yaz/tab
296 # Files that describe the attribute sets supported.
302 Now, edit the file and set <tt>profilePath</tt> to the path of the
303 YAZ profile tables (sub directory <tt>tab</tt> of the YAZ distribution
306 The 48 test records are located in the sub directory <tt>records</tt>.
307 To index these, type:
309 $ ../../bin/zebraidx -t grs.sgml update records
312 In the command above the option <tt>-t</tt> specified the record
313 type — in this case <tt>grs.sgml</tt>. The word <tt>update</tt> followed
314 by a directory root updates all files below that directory node.
316 If your indexing command was successful, you are now ready to
317 fire up a server. To start a server on port 2100, type:
319 $ ../../bin/zebrasrv tcp:@:2100
322 The Zebra index that you have just created has a single database
323 named <tt/Default/. The database contains records structured according to
324 the GILS profile, and the server will
325 return records in either either XML, USMARC, GRS-1, or SUTRS depending
326 on what your client asks for.
328 To test the server, you can use any Z39.50 client (1992 or later). For
329 instance, you can use the demo client that comes with YAZ: Just cd to
330 the <tt/client/ subdirectory of the YAZ distribution and type:
333 $ ./yaz-client tcp:localhost:2100
336 When the client has connected, you can type:
343 The default retrieval syntax for the client is USMARC. To try other
344 formats for the same record, try:
357 <it>NOTE: You may notice that more fields are returned when your
358 client requests SUTRS or GRS-1 records. When retrieving GILS records,
359 this is normal - not all of the GILS data elements have mappings in
360 the USMARC record format.</it>
362 If you've made it this far, there's a good chance that
363 you've got through the compilation OK.
365 <sect>Administrating Zebra<label id="administrating">
369 To administrate Zebra, you run the
370 <tt>zebraidx</tt> program. This program supports a number of options
371 which are preceded by a minus, and a few commands (not preceded by
374 Both the Zebra administrative tool and the Z39.50 server share a
375 set of index files and a global configuration file. The
376 name of the configuration file defaults to <tt>zebra.cfg</tt>.
377 The configuration file includes specifications on how to index
378 various kinds of records and where the other configuration files
379 are located. <tt>zebrasrv</tt> and <tt>zebraidx</tt> <em>must</em>
380 be run in the directory where the configuration file lives unless you
381 indicate the location of the configuration file by option
384 <sect1>Record Types<label id="record-types">
386 Indexing is a per-record process. Before a record is indexed search
387 keys are extracted from whatever might be the layout the original
388 record (sgml,html,text, etc..).
389 The Zebra system currently supports two fundamantal types of records:
390 structured and simple text.
391 To specify a particular extraction process, use either the
392 command line option <tt>-t</tt> or specify a
393 <tt>recordType</tt> setting in the configuration file.
395 <sect1>The Zebra Configuration File<label id="configuration-file">
397 The Zebra configuration file, read by <tt>zebraidx</tt> and
398 <tt>zebrasrv</tt> defaults to <tt>zebra.cfg</tt> unless specified
399 by <tt>-c</tt> option.
401 You can edit the configuration file with a normal text editor.
402 Parameter names and values are seperated by colons in the file. Lines
403 starting with a hash sign (<tt/#/) are treated as comments.
405 If you manage different sets of records that share common
406 characteristics, you can organize the configuration settings for each
407 type into &dquot;groups&dquot;.
408 When <tt>zebraidx</tt> is run and you wish to address a given group
409 you specify the group name with the <tt>-g</tt> option. In this case
410 settings that have the group name as their prefix will be used
411 by <tt>zebraidx</tt>. If no <tt/-g/ option is specified, the settings
412 with no prefix are used.
414 In the configuration file, the group name is placed before the option
415 name itself, separated by a dot (.). For instance, to set the record type
416 for group <tt/public/ to <tt/grs.sgml/ (the SGML-like format for structured
417 records) you would write:
420 public.recordType: grs.sgml
423 To set the default value of the record type to <tt/text/ write:
429 The available configuration settings are summarized below. They will be
430 explained further in the following sections.
433 <tag><it>group</it>.recordType[<it>.name</it>]</tag>
434 Specifies how records with the file extension <it>name</it> should
435 be handled by the indexer. This option may also be specified
436 as a command line option (<tt>-t</tt>). Note that if you do not
437 specify a <it/name/, the setting applies to all files. In general,
438 the record type specifier consists of the elements (each
439 element separated by dot), <it>fundamental-type</it>,
440 <it>file-read-type</it> and arguments. Currently, two
441 fundamental types exist, <tt>text</tt> and <tt>grs</tt>.
442 <tag><it>group</it>.recordId</tag>
443 Specifies how the records are to be identified when updated. See
444 section <ref id="locating-records" name="Locating Records">.
445 <tag><it>group</it>.database</tag>
446 Specifies the Z39.50 database name.
447 <tag><it>group</it>.storeKeys</tag>
448 Specifies whether key information should be saved for a given
449 group of records. If you plan to update/delete this type of
450 records later this should be specified as 1; otherwise it
451 should be 0 (default), to save register space.
452 <tag><it>group</it>.storeData</tag>
453 Specifies whether the records should be stored internally
454 in the Zebra system files. If you want to maintain the raw records yourself,
455 this option should be false (0). If you want Zebra to take care of the records
456 for you, it should be true(1).
458 Directory in which various lock files are stored.
460 Directory in which temporary files used during zebraidx' update
463 Specifies the directory that the server uses for temporary result sets.
464 If not specified <tt>/tmp</tt> will be used.
465 <tag>profilePath</tag>
466 Specifies the location of profile specification files.
468 Specifies the filename(s) of attribute set files for use in
469 searching. At least the Bib-1 set should be loaded (<tt/bib1.att/).
470 The <tt/profilePath/ setting is used to look for the specified files.
471 See section <ref id="attset-files" name="The Attribute Set Files">
473 Specifies size of internal memory to use for the zebraidx program. The
474 amount is given in megabytes - default is 4 (4 MB).
476 <sect1>Locating Records<label id="locating-records">
478 The default behaviour of the Zebra system is to reference the
479 records from their original location, i.e. where they were found when you
480 ran <tt/zebraidx/. That is, when a client wishes to retrieve a record
481 following a search operation, the files are accessed from the place
482 where you originally put them - if you remove the files (without
483 running <tt/zebraidx/ again, the client will receive a diagnostic
486 If your input files are not permanent - for example if you retrieve
487 your records from an outside source, or if they were temporarily
488 mounted on a CD-ROM drive,
489 you may want Zebra to make an internal copy of them. To do this,
490 you specify 1 (true) in the <tt>storeData</tt> setting. When
491 the Z39.50 server retrieves the records they will be read from the
492 internal file structures of the system.
494 <sect1>Indexing example
497 Consider a system in which you have a group of text files called
498 <tt>simple</tt>. That group of records should belong to a Z39.50 database
499 called <tt>textbase</tt>. The following <tt/zebra.cfg/ file will suffice:
502 profilePath: /usr/lib/yaz/tab:/usr/lib/zebra/tab
505 simple.recordType: text
506 simple.database: textbase
509 <sect>Running the Maintenance Interface (zebraidx)
512 The following is a complete reference to the command line interface to
513 the <tt/zebraidx/ application.
517 $ zebraidx [options] command [directory] ...
521 <tag>-t <it/type/</tag>Update all files as <it/type/. Currently, the
522 types supported are <tt/text/ and <tt/grs/<it/.subtype/. If no
523 <it/subtype/ is provided for the GRS (General Record Structure) type,
524 the canonical input format is assumed (see section <ref
525 id="local-representation" name="Local Representation">). Generally, it
526 is probably advisable to specify the record types in the
527 <tt/zebra.cfg/ file (see section <ref id="record-types" name="Record
528 Types">), to avoid confusion at subsequent updates.
530 <tag>-c <it/config-file/</tag>Read the configuration file
531 <it/config-file/ instead of <tt/zebra.cfg/.
533 <tag>-g <it/group/</tag>Update the files according to the group
534 settings for <it/group/ (see section <ref id="configuration-file"
535 name="The Zebra Configuration File">).
537 <tag>-d <it/database/</tag>The records located should be associated
538 with the database name <it/database/ for access through the Z39.50
541 <tag>-m <it/mbytes/</tag>Use <it/mbytes/ of megabytes before flushing
542 keys to background storage. This setting affects performance when
543 updating large databases.
545 <tag>-s</tag>Show analysis of the indexing process. The maintenance
546 program works in a read-only mode and doesn't change the state
547 of the index. This options is very useful when you wish to test a
550 <tag>-V</tag>Show Zebra version.
552 <tag>-v <it/level/</tag>Set the log level to <it/level/. <it/level/
553 should be one of <tt/none/, <tt/debug/, and <tt/all/.
559 <tag>Update <it/directory/</tag>Update the register with the files
560 contained in <it/directory/. If no directory is provided, a list of
561 files is read from <tt/stdin/. See section <ref
562 id="administrating" name="Administrating Zebra">.
566 <sect>The Z39.50 Server
568 <sect1>Running the Z39.50 Server (zebrasrv)
573 zebrasrv [options] [listener-address ...]
578 <tag>-a <it/APDU file/</tag> Specify a file for dumping PDUs (for diagnostic purposes).
579 The special name &dquot;-&dquot; sends output to <tt/stderr/.
581 <tag>-c <it/config-file/</tag> Read configuration information from <it/config-file/. The default configuration is <tt>./zebra.cfg</tt>.
583 <tag/-S/Don't fork on connection requests. This can be useful for
584 symbolic-level debugging. The server can only accept a single
585 connection in this mode.
587 <tag>-l <it/logfile/</tag>Specify an output file for the diagnostic
588 messages. The default is to write this information to <tt/stderr/.
590 <tag>-v <it/log-level/</tag>The log level. Use a comma-separated list of members of the set
591 {fatal,debug,warn,log,all,none}.
593 <tag>-u <it/username/</tag>Set user ID. Sets the real UID of the server process to that of the
594 given <it/username/. It's useful if you aren't comfortable with having the
595 server run as root, but you need to start it as such to bind a
598 <tag>-w <it/working-directory/</tag>Change working directory.
600 <tag>-i</tag>Run under the Internet superserver, <tt/inetd/. Make
601 sure you use the logfile option <tt/-l/ in conjunction with this
602 mode and specify the <tt/-l/ option before any other options.
604 <tag>-t <it/timeout/</tag>Set the idle session timeout (default 60 minutes).
606 <tag>-k <it/kilobytes/</tag>Set the (approximate) maximum size of
607 present response messages. Default is 1024 Kb (1 Mb).
610 A <it/listener-address/ consists of a transport mode followed by a
611 colon (:) followed by a listener address. The transport mode is
612 either <tt/osi/ or <tt/tcp/.
614 For TCP, an address has the form
617 hostname | IP-number [: portnumber]
620 The port number defaults to 210 (standard Z39.50 port).
622 The special hostname &dquot;@&dquot; is mapped to
623 the address INADDR_ANY, which causes the server to listen on any local
624 interface. To start the server listening on the registered port for
625 Z39.50, and to drop root privileges once the
626 port is bound, execute the server like this (from a root shell):
629 zebrasrv -u daemon tcp:@
632 You can replace <tt/daemon/ with another user, eg. your own account, or
633 a dedicated IR server account.
635 The default behavior for <tt/zebrasrv/ is to establish a single TCP/IP
636 listener, for the Z39.50 protocol, on port 9999.
638 <sect1>Z39.50 Protocol Support and Behavior
640 <sect2>Initialization
643 During initialization, the server will negotiate to version 3 of the
644 Z39.50 protocol (unless the client specifies a lower version), and the option bits for Search, Present, Scan,
645 NamedResultSets, and concurrentOperations will be set, if requested by
646 the client. The maximum PDU size is negotiated down to a maximum of
649 <sect2>Search<label id="search">
652 The supported query type are 1 and 101. All operators are currently
653 supported with the restriction that only proximity units of type "word" are
654 supported for the proximity operator.
655 Queries can be arbitrarily complex.
656 Named result sets are supported, and result sets can be used as operands
658 Searches may span multiple databases.
660 The server has full support for piggy-backed present requests (see
661 also the following section).
663 <bf/Use/ attributes are interpreted according to the attribute sets which
664 have been loaded in the <tt/zebra.cfg/ file, and are matched against
665 specific fields as specified in the <tt/.abs/ file which describes the
666 profile of the records which have been loaded. If no <bf/Use/
667 attribute is provided, a default of Bib-1 <bf/Any/ is assumed.
669 If a <bf/Structure/ attribute of <bf/Phrase/ is used in conjunction with a
670 <bf/Completeness/ attribute of <bf/Complete (Sub)field/, the term is
671 matched against the contents of the phrase (long word) register, if one
672 exists for the given <bf/Use/ attribute.
673 A phrase register is created for those fields in the <tt/.abs/
674 file that contains a <tt/p/-specifier.
676 If <bf/Structure/=<bf/Phrase/ is used in conjunction with
677 <bf/Incomplete Field/ - the default value for <bf/Completeness/, the
678 search is directed against the normal word registers, but if the term
679 contains multiple words, the term will only match if all of the words
680 are found immediately adjacent, and in the given order.
681 The word search is performed on those fields that are indexed as
682 type <tt/w/ in the <tt/.abs/ file.
684 If the <bf/Structure/ attribute is <bf/Word List/,
685 <bf/Free-form Text/, or <bf/Document Text/, the term is treated as a
686 natural-language, relevance-ranked query.
687 This search type uses the word register, i.e. those fields
688 that are indexed as type <tt/w/ in the <tt/.abs/ file.
690 If the <bf/Structure/ attribute is <bf/Numeric String/ the
691 term is treated as an integer. The search is performed on those
692 fields that are indexed as type <tt/n/ in the <tt/.abs/ file.
694 If the <bf/Structure/ attribute is <bf/URx/ the
695 term is treated as a URX (URL) entity. The search is performed on those
696 fields that are indexed as type <tt/u/ in the <tt/.abs/ file.
698 If the <bf/Structure/ attribute is <bf/Local Number/ the
699 term is treated as native Zebra Record Identifier.
701 If the <bf/Relation/ attribute is <bf/Equals/ (default), the term is
702 matched in a normal fashion (modulo truncation and processing of
703 individual words, if required). If <bf/Relation/ is <bf/Less Than/,
704 <bf/Less Than or Equal/, <bf/Greater than/, or <bf/Greater than or
705 Equal/, the term is assumed to be numerical, and a standard regular
706 expression is constructed to match the given expression. If
707 <bf/Relation/ is <bf/Relevance/, the standard natural-language query
708 processor is invoked.
710 For the <bf/Truncation/ attribute, <bf/No Truncation/ is the default.
711 <bf/Left Truncation/ is not supported. <bf/Process #/ is supported, as
712 is <bf/Regxp-1/. <bf/Regxp-2/ enables the fault-tolerant (fuzzy)
713 search. As a default, a single error (deletion, insertion,
714 replacement) is accepted when terms are matched against the register
717 <sect3>Regular expressions
720 Each term in a query is interpreted as a regular expression if
721 the truncation value is either <bf/Regxp-1/ (102) or <bf/Regxp-2/ (103).
722 Both query types follow the same syntax with the operands:
724 <tag/x/ Matches the character <it/x/.
725 <tag/./ Matches any character.
726 <tag><tt/[/..<tt/]/</tag> Matches the set of characters specified;
727 such as <tt/[abc]/ or <tt/[a-c]/.
731 <tag/x*/ Matches <it/x/ zero or more times. Priority: high.
732 <tag/x+/ Matches <it/x/ one or more times. Priority: high.
733 <tag/x?/ Matches <it/x/ once or twice. Priority: high.
734 <tag/xy/ Matches <it/x/, then <it/y/. Priority: medium.
735 <tag/x|y/ Matches either <it/x/ or <it/y/. Priority: low.
737 The order of evaluation may be changed by using parentheses.
739 If the first character of the <bf/Regxp-2/ query is a plus character
740 (<tt/+/) it marks the beginning of a section with non-standard
741 specifiers. The next plus character marks the end of the section.
742 Currently Zebra only supports one specifier, the error tolerance,
743 which consists one digit.
745 Since the plus operator is normally a suffix operator the addition to
746 the query syntax doesn't violate the syntax for standard regular
749 <sect3>Query examples
752 Phrase search for <bf/information retrieval/ in the title-register:
754 @attr 1=4 "information retrieval"
757 Ranked search for the same thing:
759 @attr 1=4 @attr 2=102 "Information retrieval"
762 Phrase search with a regular expression:
764 @attr 1=4 @attr 5=102 "informat.* retrieval"
767 Ranked search with a regular expression:
769 @attr 1=4 @attr 5=102 @attr 2=102 "informat.* retrieval"
772 In the GILS schema (<tt/gils.abs/), the west-bounding-coordinate is
773 indexed as type <tt/n/, and is therefore searched by specifying
774 <bf/structure/=<bf/Numeric String/.
775 To match all those records with west-bounding-coordinate greater
776 than -114 we use the following query:
778 @attr 4=109 @attr 2=5 @attr gils 1=2038 -114
783 The present facility is supported in a standard fashion. The requested
784 record syntax is matched against the ones supported by the profile of
785 each record retrieved. If no record syntax is given, SUTRS is the
786 default. The requested element set name, again, is matched against any
787 provided by the relevant record profiles.
792 The attribute combinations provided with the TermListAndStartPoint are
793 processed in the same way as operands in a query (see above).
794 Currently, only the term and the globalOccurrences are returned with
795 the TermInfo structure.
800 Z39.50 specifies three diffent types of sort criterias.
801 Of these Zebra supports the attribute specification type in which
802 case the use attribute specifies the "Sort register".
803 Sort registers are created for those fields that are of type "sort" in
804 the default.idx file.
805 The corresponding character mapping file in default.idx specifies the
806 ordinal of each character used in the actual sort.
808 Z39.50 allows the client to specify sorting on one or more input
809 result sets and one output result set.
810 Zebra supports sorting on one result set only which may or may not
811 be the same as the output result set.
816 If a Close PDU is received, the server will respond with a Close PDU
817 with reason=FINISHED, no matter which protocol version was negotiated
818 during initialization. If the protocol version is 3 or more, the
819 server will generate a Close PDU under certain circumstances,
820 including a session timeout (60 minutes by default), and certain kinds of
821 protocol errors. Once a Close PDU has been sent, the protocol
822 association is considered broken, and the transport connection will be
823 closed immediately upon receipt of further data, or following a short
826 <sect>The Record Model
829 Zebra is designed to support a wide range of data management
830 applications. The system can be configured to handle virtually any
831 kind of structured data. Each record in the system is associated with
832 a <it/record schema/ which lends context to the data elements of the
833 record. Any number of record schema can coexist in the system.
834 Although it may be wise to use only a single schema within
835 one database, the system poses no such restrictions.
837 The record model described in this chapter applies to the fundamental,
839 record type <tt>grs</tt> as introduced in
840 section <ref id="record-types" name="Record Types">.
842 Records pass through three different states during processing in the
846 <item>When records are accessed by the system, they are represented
847 in their local, or native format. This might be SGML or HTML files,
848 News or Mail archives, MARC records. If the system doesn't already
849 know how to read the type of data you need to store, you can set up an
850 input filter by preparing conversion rules based on regular
851 expressions and possibly augmented by a flexible scripting language (Tcl). The input filter
852 produces as output an internal representation:
854 <item>When records are processed by the system, they are represented
855 in a tree-structure, constructed by tagged data elements hanging off a
856 root node. The tagged elements may contain data or yet more tagged
857 elements in a recursive structure. The system performs various
858 actions on this tree structure (indexing, element selection, schema
861 <item>Before transmitting records to the client, they are first
862 converted from the internal structure to a form suitable for exchange
863 over the network - according to the Z39.50 standard.
866 <sect1>Local Representation<label id="local-representation">
869 As mentioned earlier, Zebra places few restrictions on the type of
870 data that you can index and manage. Generally, whatever the form of
871 the data, it is parsed by an input filter specific to that format, and
872 turned into an internal structure that Zebra knows how to handle. This
873 process takes place whenever the record is accessed - for indexing and
877 The RecordType parameter in the <tt/zebra.cfg/ file, or the <tt/-t/
878 option to the indexer tells Zebra how to process input records. Two
879 basic types of processing are available - raw text and structured
880 data. Raw text is just that, and it is selected by providing the
881 argument <bf/text/ to Zebra. Structured records are all handled
882 internally using the basic mechanisms described in the subsequent
883 sections. Zebra can read structured records in many different formats.
884 How this is done is governed by additional parameters after the
885 &dquot;grs&dquot; keyboard, separated by &dquot;.&dquot; characters.
887 Three basic subtypes to the <bf/grs/ type are currently available:
890 <tag>grs.sgml</tag>This is the canonical input format —
891 described below. It is a simple SGML-like syntax.
893 <tag>grs.regx.<it/filter/</tag>This enables a user-supplied input
894 filter. The mechanisms of these filters are described below.
896 <tag>grs.tcl.<it/filter/</tag>This enables a user-supplied input
897 filter with Tcl rules (only availble if zebra is compiled with Tcl
900 <tag>grs.marc.<it/abstract syntax/</tag>This allows Zebra to read
901 records in the ISO2709 (MARC) encoding standard. In this case, the
902 last paramemeter <it/abstract syntax/ names the .abs file (see below)
903 which describes the specific MARC structure of the input record as
904 well as the indexing rules.
907 <sect2>Canonical Input Format
910 Although input data can take any form, it is sometimes useful to
911 describe the record processing capabilities of the system in terms of
912 a single, canonical input format that gives access to the full
913 spectrum of structure and flexibility in the system. In Zebra, this
914 canonical format is an &dquot;SGML-like&dquot; syntax.
916 To use the canonical format specify <tt>grs.sgml</tt> as the record
919 Consider a record describing an information resource (such a record is
920 sometimes known as a <it/locator record/). It might contain a field
921 describing the distributor of the information resource, which might in
922 turn be partitioned into various fields providing details about the
923 distributor, like this:
927 <Name> USGS/WRD &etago;Name>
928 <Organization> USGS/WRD &etago;Organization>
930 U.S. GEOLOGICAL SURVEY, 505 MARQUETTE, NW
931 &etago;Street-Address>
932 <City> ALBUQUERQUE &etago;City>
933 <State> NM &etago;State>
934 <Zip-Code> 87102 &etago;Zip-Code>
935 <Country> USA &etago;Country>
936 <Telephone> (505) 766-5560 &etago;Telephone>
940 <it>NOTE: The indentation used above is used to illustrate how Zebra
941 interprets the markup. The indentation, in itself, has no
942 significance to the parser for the canonical input format, which
943 discards superfluous whitespace.</it>
945 The keywords surrounded by <...> are <it/tags/, while the
946 sections of text in between are the <it/data elements/. A data element
947 is characterized by its location in the tree that is made up by the
948 nested elements. Each element is terminated by a closing tag -
949 beginning with <tt/&etago;/, and containing the same symbolic tag-name as
950 the corresponding opening tag. The general closing tag - <tt/&etago;>/ -
951 terminates the element started by the last opening tag. The
952 structuring of elements is significant. The element <bf/Telephone/,
953 for instance, may be indexed and presented to the client differently,
954 depending on whether it appears inside the <bf/Distributor/ element,
955 or some other, structured data element such a <bf/Supplier/ element.
960 The first tag in a record describes the root node of the tree that
961 makes up the total record. In the canonical input format, the root tag
962 should contain the name of the schema that lends context to the
963 elements of the record (see section <ref id="internal-representation"
964 name="Internal Representation">). The following is a GILS record that
965 contains only a single element (strictly speaking, that makes it an
966 illegal GILS record, since the GILS profile includes several mandatory
967 elements - Zebra does not validate the contents of a record against
968 the Z39.50 profile, however - it merely attempts to match up elements
969 of a local representation with the given schema):
973 <title>Zen and the Art of Motorcycle Maintenance&etago;title>
980 Zebra allows you to provide individual data elements in a number of
981 <it/variant forms/. Examples of variant forms are textual data
982 elements which might appear in different languages, and images which
983 may appear in different formats or layouts. The variant system in
985 essentially a representation of the variant mechanism of
988 The following is an example of a title element which occurs in two
993 <var lang lang "eng">
994 Zen and the Art of Motorcycle Maintenance&etago;>
995 <var lang lang "dan">
996 Zen og Kunsten at Vedligeholde en Motorcykel&etago;>
1000 The syntax of the <it/variant element/ is <tt><<bf/var/ <it/class
1001 type value/></tt>. The available values for the <it/class/ and
1002 <it/type/ fields are given by the variant set that is associated with the
1003 current schema (see section <ref id="variant-set" name="Variant Set
1006 Variant elements are terminated by the general end-tag &etago;>, by
1007 the variant end-tag &etago;var>, by the appearance of another variant
1008 tag with the same <it/class/ and <it/value/ settings, or by the
1009 appearance of another, normal tag. In other words, the end-tags for
1010 the variants used in the example above could have been saved.
1012 Variant elements can be nested. The element
1016 <var lang lang "eng"><var body iana "text/plain">
1017 Zen and the Art of Motorcycle Maintenance
1021 Associates two variant components to the variant list for the title
1024 Given the nesting rules described above, we could write
1028 <var body iana "text/plain>
1029 <var lang lang "eng">
1030 Zen and the Art of Motorcycle Maintenance
1031 <var lang lang "dan">
1032 Zen og Kunsten at Vedligeholde en Motorcykel
1036 The title element above comes in two variants. Both have the IANA body
1037 type &dquot;text/plain&dquot;, but one is in English, and the other in
1038 Danish. The client, using the element selection mechanism of Z39.50,
1039 can retrieve information about the available variant forms of data
1040 elements, or it can select specific variants based on the requirements
1043 <sect2>Input Filters
1046 In order to handle general input formats, Zebra allows the
1047 operator to define filters which read individual records in their native format
1048 and produce an internal representation that the system can
1051 Input filters are ASCII files, generally with the suffix <tt/.flt/.
1052 The system looks for the files in the directories given in the
1053 <bf/profilePath/ setting in the <tt/zebra.cfg/ files. The record type
1054 for the filter is <tt>grs.regx.</tt><it>filter-filename</it>
1055 (fundamental type <tt>grs</tt>, file read type <tt>regx</tt>, argument
1056 <it>filter-filename</it>).
1058 Generally, an input filter consists of a sequence of rules, where each
1059 rule consists of a sequence of expressions, followed by an action. The
1060 expressions are evaluated against the contents of the input record,
1061 and the actions normally contribute to the generation of an internal
1062 representation of the record.
1064 An expression can be either of the following:
1067 <tag/INIT/The action associated with this expression is evaluated
1068 exactly once in the lifetime of the application, before any records
1069 are read. It can be used in conjunction with an action that
1070 initializes tables or other resources that are used in the processing
1073 <tag/BEGIN/Matches the beginning of the record. It can be used to
1074 initialize variables, etc. Typically, the <bf/BEGIN/ rule is also used
1075 to establish the root node of the record.
1077 <tag/END/Matches the end of the record - when all of the contents
1078 of the record has been processed.
1080 <tag>/pattern/</tag>Matches a string of characters from the input
1083 <tag/BODY/This keyword may only be used between two patterns. It
1084 matches everything between (not including) those patterns.
1086 <tag/FINISH/THe expression asssociated with this pattern is evaluated
1087 once, before the application terminates. It can be used to release
1088 system resources - typically ones allocated in the <bf/INIT/ step.
1092 An action is surrounded by curly braces ({...}), and consists of a
1093 sequence of statements. Statements may be separated by newlines or
1094 semicolons (;). Within actions, the strings that matched the
1095 expressions immediately preceding the action can be referred to as
1096 $0, $1, $2, etc.
1098 The available statements are:
1102 <tag>begin <it/type [parameter ... ]/</tag>Begin a new
1103 data element. The type is one of the following:
1105 <tag/record/Begin a new record. The followingparameter should be the
1106 name of the schema that describes the structure of the record, eg.
1107 <tt/gils/ or <tt/wais/ (see below). The <tt/begin record/ call should
1109 any other use of the <bf/begin/ statement.
1111 <tag/element/Begin a new tagged element. The parameter is the
1112 name of the tag. If the tag is not matched anywhere in the tagsets
1113 referenced by the current schema, it is treated as a local string
1116 <tag/variant/Begin a new node in a variant tree. The parameters are
1117 <it/class type value/.
1121 <tag/data/Create a data element. The concatenated arguments make
1122 up the value of the data element. The option <tt/-text/ signals that
1123 the layout (whitespace) of the data should be retained for
1124 transmission. The option <tt/-element/ <it/tag/ wraps the data up in
1125 the <it/tag/. The use of the <tt/-element/ option is equivalent to
1126 preceding the command with a <bf/begin element/ command, and following
1127 it with the <bf/end/ command.
1129 <tag>end <it/[type]/</tag>Close a tagged element. If no parameter is given,
1130 the last element on the stack is terminated. The first parameter, if
1131 any, is a type name, similar to the <bf/begin/ statement. For the
1132 <bf/element/ type, a tag name can be provided to terminate a specific tag.
1136 The following input filter reads a Usenet news file, producing a
1137 record in the WAIS schema. Note that the body of a news posting is
1138 separated from the list of headers by a blank line (or rather a
1139 sequence of two newline characters.
1142 BEGIN { begin record wais }
1144 /^From:/ BODY /$/ { data -element name $1 }
1145 /^Subject:/ BODY /$/ { data -element title $1 }
1146 /^Date:/ BODY /$/ { data -element lastModified $1 }
1148 begin element bodyOfDisplay
1149 begin variant body iana "text/plain"
1155 If Zebra is compiled with support for Tcl (Tool Command Language)
1156 enabled, the statements described above are supplemented with a complete
1157 scripting environment, including control structures (conditional
1158 expressions and loop constructs), and powerful string manipulation
1159 mechanisms for modifying the elements of a record. Tcl is a popular
1160 scripting environment, with several tutorials available both online
1163 <it>NOTE: Variant support is not currently available in the input
1164 filter, but will be included with one of the next
1167 <sect1>Internal Representation<label id="internal-representation">
1170 When records are manipulated by the system, they're represented in a
1171 tree-structure, with data elements at the leaf nodes, and tags or
1172 variant components at the non-leaf nodes. The root-node identifies the
1173 schema that lends context to the tagging and structuring of the
1174 record. Imagine a simple record, consisting of a 'title' element and
1175 an 'author' element:
1178 TITLE "Zen and the Art of Motorcycle Maintenance"
1180 AUTHOR "Robert Pirsig"
1183 A slightly more complex record would have the author element consist
1184 of two elements, a surname and a first name:
1187 TITLE "Zen and the Art of Motorcycle Maintenance"
1194 The root of the record will refer to the record schema that describes
1195 the structuring of this particular record. The schema defines the
1196 element tags (TITLE, FIRST-NAME, etc.) that may occur in the record, as
1197 well as the structuring (SURNAME should appear below AUTHOR, etc.). In
1198 addition, the schema establishes element set names that are used by
1199 the client to request a subset of the elements of a given record. The
1200 schema may also establish rules for converting the record to a
1201 different schema, by stating, for each element, a mapping to a
1204 <sect2>Tagged Elements
1207 A data element is characterized by its tag, and its position in the
1208 structure of the record. For instance, while the tag &dquot;telephone
1209 number&dquot; may be used different places in a record, we may need to
1210 distinguish between these occurrences, both for searching and
1211 presentation purposes. For instance, while the phone numbers for the
1212 &dquot;customer&dquot; and the &dquot;service provider&dquot; are both
1213 representatives for the same type of resource (a telephone number), it
1214 is essential that they be kept separate. The record schema provides
1215 the structure of the record, and names each data element (defined by
1216 the sequence of tags - the tag path - by which the element can be
1217 reached from the root of the record).
1222 The children of a tag node may be either more tag nodes, a data node
1223 (possibly accompanied by tag nodes),
1224 or a tree of variant nodes. The children of variant nodes are either
1225 more variant nodes or a data node (possibly accompanied by more
1226 variant nodes). Each leaf node, which is normally a
1227 data node, corresponds to a <it/variant form/ of the tagged element
1228 identified by the tag which parents the variant tree. The following
1229 title element occurs in two different languages:
1232 VARIANT LANG=ENG "War and Peace"
1234 VARIANT LANG=DAN "Krig og Fred"
1237 Which of the two elements are transmitted to the client by the server
1238 depends on the specifications provided by the client, if any.
1240 In practice, each variant node is associated with a triple of class,
1241 type, value, corresponding to the variant mechanism of Z39.50.
1243 <sect2>Data Elements
1246 Data nodes have no children (they are always leaf nodes in the record
1249 <it>NOTE: Documentation needs extension here about types of nodes - numerical,
1250 textual, etc., plus the various types of inclusion notes.</it>
1252 <sect1>Configuring Your Data Model<label id="data-model">
1255 The following sections describe the configuration files that govern
1256 the internal management of data records. The system searches for the files
1257 in the directories specified by the <bf/profilePath/ setting in the
1258 <tt/zebra.cfg/ file.
1260 <sect2>About Object Identifers
1262 When Object Identifiers (or OID's) need to be specified in the following
1263 a named OID reference or a raw OID reference may be used. For the named
1264 OID's refer to the source file <tt>util/oid.c</tt> from YAZ. The raw
1265 canonical OID's are specified in dot-notation (for example
1266 1.2.840.10003.3.1000.81.1).
1268 <sect2>The Abstract Syntax
1271 The abstract syntax definition (also known as an Abstract Record
1272 Structure, or ARS) is the focal point of the
1273 record schema description. For a given schema, the ABS file may state any
1274 or all of the following:
1277 <item>The object identifier of the Z39.50 schema associated
1278 with the ARS, so that it can be referred to by the client.
1280 <item>The attribute set (which can possibly be a compound of multiple
1281 sets) which applies in the profile. This is used when indexing and
1282 searching the records belonging to the given profile.
1284 <item>The Tag set (again, this can consist of several different sets).
1285 This is used when reading the records from a file, to recognize the
1286 different tags, and when transmitting the record to the client -
1287 mapping the tags to their numerical representation, if they are
1290 <item>The variant set which is used in the profile. This provides a
1291 vocabulary for specifying the <it/forms/ of data that appear inside
1294 <item>Element set names, which are a shorthand way for the client to
1295 ask for a subset of the data elements contained in a record. Element
1296 set names, in the retrieval module, are mapped to <it/element
1297 specifications/, which contain information equivalent to the
1298 <it/Espec-1/ syntax of Z39.50.
1300 <item>Map tables, which may specify mappings to <it/other/ database
1301 profiles, if desired.
1303 <item>Possibly, a set of rules describing the mapping of elements to a
1304 MARC representation.
1306 <item>A list of element descriptions (this is the actual ARS of the
1307 schema, in Z39.50 terms), which lists the ways in which the various
1308 tags can be used and organized hierarchically.
1311 Several of the entries above simply refer to other files, which
1312 describe the given objects.
1314 <sect2>The Configuration Files
1317 This section describes the syntax and use of the various tables which
1318 are used by the retrieval module.
1320 The number of different file types may appear daunting at first, but
1321 each type corresponds fairly clearly to a single aspect of the Z39.50
1322 retrieval facilities. Further, the average database administrator,
1323 who is simply reusing an existing profile for which tables already
1324 exist, shouldn't have to worry too much about the contents of these tables.
1326 Generally, the files are simple ASCII files, which can be maintained
1327 using any text editor. Blank lines, and lines beginning with a (#) are
1328 ignored. Any characters on a line followed by a (#) are also ignored.
1330 lines contain <it/directives/, which provide some setting or value
1331 to the system. Generally, settings are characterized by a single
1332 keyword, identifying the setting, followed by a number of parameters.
1333 Some settings are repeatable (r), while others may occur only once in a
1334 file. Some settings are optional (o), whicle others again are
1337 <sect2>The Abstract Syntax (.abs) Files
1340 The name of this file type is slightly misleading in Z39.50 terms,
1341 since, apart from the actual abstract syntax of the profile, it also
1342 includes most of the other definitions that go into a database
1345 When a record in the canonical, SGML-like format is read from a file
1346 or from the database, the first tag of the file should reference the
1347 profile that governs the layout of the record. If the first tag of the
1348 record is, say, <tt><gils></tt>, the system will look for the profile
1349 definition in the file <tt/gils.abs/. Profile definitions are cached,
1350 so they only have to be read once during the lifespan of the current
1353 When writing your own input filters, the <bf/record-begin/ command
1354 introduces the profile, and should always be called first thing when
1355 introducing a new record.
1357 The file may contain the following directives:
1360 <tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or
1361 description for the profile. Mostly useful for diagnostic purposes.
1363 <tag>reference <it/OID-name/</tag> (m) The OID for
1364 the profile (name or dotted-numerical list).
1366 <tag>attset <it/filename/</tag> (m) The attribute set that is used for
1367 indexing and searching records belonging to this profile.
1369 <tag>tagset <it/filename/ [<it/type/]</tag> (o) The tag
1370 set (if any) that describe that fields of the records. The type, which
1371 is optional, specifies the tag type. If not given, the type-specifier
1372 in the Tag Set files is used.
1374 <tag>varset <it/filename/</tag> (o) The variant set used in the profile.
1376 <tag>maptab <it/filename/</tag> (o,r) This points to a
1377 conversion table that might be used if the client asks for the record
1378 in a different schema from the native one.
1380 <tag>marc <it/filename/</tag> (o) Points to a file containing parameters
1381 for representing the record contents in the ISO2709 syntax. Read the
1382 description of the MARC representation facility below.
1384 <tag>esetname <it/name filename/</tag> (o,r) Associates the
1385 given element set name with an element selection file. If an (@) is
1386 given in place of the filename, this corresponds to a null mapping for
1387 the given element set name.
1389 <tag>any <it/tags/</tag> (o) This directive specifies a list of
1390 attributes which should be appended to the attribute list given for each
1391 element. The effect is to make every single element in the abstract
1392 syntax searchable by way of the given attributes. This directive
1393 provides an efficient way of supporting free-text searching across all
1394 elements. However, it does increase the size of the index
1395 significantly. The attributes can be qualified with a structure, as in
1396 the <bf/elm/ directive below.
1398 <tag>elm <it/path name attributes/</tag> (o,r) Adds an element
1399 to the abstract record syntax of the schema. The <it/path/ follows the
1400 syntax which is suggested by the Z39.50 document - that is, a sequence
1401 of tags separated by slashes (/). Each tag is given as a
1402 comma-separated pair of tag type and -value surrounded by parenthesis.
1403 The <it/name/ is the name of the element, and the <it/attributes/
1404 specifies which attributes to use when indexing the element in a
1405 comma-separated list. A ! in
1406 place of the attribute name is equivalent to specifying an attribute
1407 name identical to the element name. A - in place of the attribute name
1408 specifies that no indexing is to take place for the given element. The
1409 attributes can be qualified with <it/field types/ to specify which
1410 character set should govern the indexing procedure for that field. The
1411 same data element may be indexed into several different fields, using
1412 different character set definitions. See the section
1413 <ref id="field structure and character sets"
1414 name="Field Structure and Character Sets">.
1415 The default field type is &dquot;w&dquot; for
1419 The following is an excerpt from the abstract syntax file for the GILS
1424 reference GILS-schema
1429 maptab gils-usmarc.map
1433 esetname VARIANT gils-variant.est # for WAIS-compliance
1434 esetname B gils-b.est
1435 esetname G gils-g.est
1440 elm (1,14) localControlNumber Local-number
1441 elm (1,16) dateOfLastModification Date/time-last-modified
1442 elm (2,1) Title w:!,p:!
1443 elm (4,1) controlIdentifier Identifier-standard
1444 elm (2,6) abstract Abstract
1445 elm (4,51) purpose !
1446 elm (4,52) originator -
1447 elm (4,53) accessConstraints !
1448 elm (4,54) useConstraints !
1449 elm (4,70) availability -
1450 elm (4,70)/(4,90) distributor -
1451 elm (4,70)/(4,90)/(2,7) distributorName !
1452 elm (4,70)/(4,90)/(2,10 distributorOrganization !
1453 elm (4,70)/(4,90)/(4,2) distributorStreetAddress !
1454 elm (4,70)/(4,90)/(4,3) distributorCity !
1457 <sect2>The Attribute Set (.att) Files<label id="attset-files">
1460 This file type describes the <bf/Use/ elements of an attribute set.
1461 It contains the following directives.
1465 <tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or
1466 description for the attribute set. Mostly useful for diagnostic purposes.
1468 <tag>reference <it/OID-name/</tag> (m) The reference name of the OID for
1471 <tag>include <it/filename/</tag> (o,r) This directive is used to
1472 include another attribute set as a part of the current one. This is
1473 used when a new attribute set is defined as an extension to another
1474 set. For instance, many new attribute sets are defined as extensions
1475 to the <bf/bib-1/ set. This is an important feature of the retrieval
1476 system of Z39.50, as it ensures the highest possible level of
1477 interoperability, as those access points of your database which are
1478 derived from the external set (say, bib-1) can be used even by clients
1479 who are unaware of the new set.
1481 <tag>att <it/att-value att-name [local-value]/</tag> (o,r) This
1482 repeatable directive introduces a new attribute to the set. The
1483 attribute value is stored in the index (unless a <it/local-value/ is
1484 given, in which case this is stored). The name is used to refer to the
1485 attribute from the <it/abstract syntax/. </descrip>
1487 This is an excerpt from the GILS attribute set definition. Notice how
1488 the file describing the <it/bib-1/ attribute set is referenced.
1492 reference GILS-attset
1495 att 2001 distributorName
1496 att 2002 indexTermsControlled
1498 att 2004 accessConstraints
1499 att 2005 useConstraints
1502 <sect2>The Tag Set (.tag) Files
1505 This file type defines the tagset of the profile, possibly by
1506 referencing other tag sets (most tag sets, for instance, will include
1507 tagsetG and tagsetM from the Z39.50 specification. The file may
1508 contain the following directives.
1511 <tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or
1512 description for the tag set. Mostly useful for diagnostic purposes.
1514 <tag>reference <it/OID-name/</tag> (o) The reference name of the OID for
1515 the tag set. The directive is optional, since not all tag sets are
1516 registered outside of their schema.
1518 <tag>type <it/integer/</tag> (m) The type number of the tagset within the schema
1519 profile (note: this specification really should belong to the .abs
1520 file. This will be fixed in a future release).
1522 <tag>include <it/filename/</tag> (o,r) This directive is used
1523 to include the definitions of other tag sets into the current one.
1525 <tag>tag <it/number names type/</tag> (o,r) Introduces a new
1526 tag to the set. The <it/number/ is the tag number as used in the protocol
1527 (there is currently no mechanism for specifying string tags at this
1528 point, but this would be quick work to add). The <it/names/ parameter
1529 is a list of names by which the tag should be recognized in the input
1530 file format. The names should be separated by slashes (/). The
1531 <it/type/ is th recommended datatype of the tag. It should be one of
1539 <item>generalizedtime
1547 The following is an excerpt from the TagsetG definition file.
1556 tag 3 publicationPlace string
1557 tag 4 publicationDate string
1558 tag 5 documentId string
1559 tag 6 abstract string
1561 tag 8 date generalizedtime
1562 tag 9 bodyOfDisplay string
1563 tag 10 organization string
1566 <sect2>The Variant Set (.var) Files<label id="variant-set">
1569 The variant set file is a straightforward representation of the
1570 variant set definitions associated with the protocol. At present, only
1571 the <it/Variant-1/ set is known.
1573 These are the directives allowed in the file.
1576 <tag>name <it/symbolic-name/</tag> (m) This provides a shorthand name or
1577 description for the variant set. Mostly useful for diagnostic purposes.
1579 <tag>reference <it/OID-name/</tag> (o) The reference name of the OID for
1580 the variant set, if one is required.
1582 <tag>class <it/integer class-name/</tag> (m,r) Introduces a new
1583 class to the variant set.
1585 <tag>type <it/integer type-name datatype/</tag> (m,r) Addes a
1586 new type to the current class (the one introduced by the most recent
1587 <bf/class/ directive). The type names belong to the same name space as
1588 the one used in the tag set definition file.
1591 The following is an excerpt from the file describing the variant set
1600 type 1 variantId octetstring
1605 type 2 z39.50 string
1609 <sect2>The Element Set (.est) Files
1612 The element set specification files describe a selection of a subset
1613 of the elements of a database record. The element selection mechanism
1614 is equivalent to the one supplied by the <it/Espec-1/ syntax of the
1615 Z39.50 specification. In fact, the internal representation of an
1616 element set specification is identical to the <it/Espec-1/ structure,
1617 and we'll refer you to the description of that structure for most of
1618 the detailed semantics of the directives below.
1621 NOTE: Not all of the Espec-1 functionality has been implemented yet.
1622 The fields that are mentioned below all work as expected, unless
1626 The directives available in the element set file are as follows:
1629 <tag>defaultVariantSetId <it/OID-name/</tag> (o) If variants are used in
1630 the following, this should provide the name of the variantset used
1631 (it's not currently possible to specify a different set in the
1632 individual variant request). In almost all cases (certainly all
1633 profiles known to us), the name <tt/Variant-1/ should be given here.
1635 <tag>defaultVariantRequest <it/variant-request/</tag> (o) This directive
1636 provides a default variant request for
1637 use when the individual element requests (see below) do not contain a
1638 variant request. Variant requests consist of a blank-separated list of
1639 variant components. A variant compont is a comma-separated,
1640 parenthesized triple of variant class, type, and value (the two former
1641 values being represented as integers). The value can currently only be
1642 entered as a string (this will change to depend on the definition of
1643 the variant in question). The special value (@) is interpreted as a
1644 null value, however.
1646 <tag>simpleElement <it/path ['variant' variant-request]/</tag>
1647 (o,r) This corresponds to a simple element request in <it/Espec-1/. The
1648 path consists of a sequence of tag-selectors, where each of these can
1652 <item>A simple tag, consisting of a comma-separated type-value pair in
1653 parenthesis, possibly followed by a colon (:) followed by an
1654 occurrences-specification (see below). The tag-value can be a number
1655 or a string. If the first character is an apostrophe ('), this forces
1656 the value to be interpreted as a string, even if it appears to be numerical.
1658 <item>A WildThing, represented as a question mark (?), possibly
1659 followed by a colon (:) followed by an occurrences specification (see
1662 <item>A WildPath, represented as an asterisk (*). Note that the last
1663 element of the path should not be a wildPath (wildpaths don't work in
1667 The occurrences-specification can be either the string <tt/all/, the
1668 string <tt/last/, or an explicit value-range. The value-range is
1669 represented as an integer (the starting point), possibly followed by a
1670 plus (+) and a second integer (the number of elements, default being
1673 The variant-request has the same syntax as the defaultVariantRequest
1674 above. Note that it may sometimes be useful to give an empty variant
1675 request, simply to disable the default for a specific set of fields
1676 (we aren't certain if this is proper <it/Espec-1/, but it works in
1677 this implementation).
1680 The following is an example of an element specification belonging to
1684 simpleelement (1,10)
1685 simpleelement (1,12)
1687 simpleelement (1,14)
1689 simpleelement (4,52)
1692 <sect2>The Schema Mapping (.map) Files<label id="schema-mapping">
1695 Sometimes, the client might want to receive a database record in
1696 a schema that differs from the native schema of the record. For
1697 instance, a client might only know how to process WAIS records, while
1698 the database record is represented in a more specific schema, such as
1699 GILS. In this module, a mapping of data to one of the MARC formats is
1700 also thought of as a schema mapping (mapping the elements of the
1701 record into fields consistent with the given MARC specification, prior
1702 to actually converting the data to the ISO2709). This use of the
1703 object identifier for USMARC as a schema identifier represents an
1704 overloading of the OID which might not be entirely proper. However,
1705 it represents the dual role of schema and record syntax which
1706 is assumed by the MARC family in Z39.50.
1709 NOTE: The schema-mapping functions are so far limited to a
1710 straightforward mapping of elements. This should be extended with
1711 mechanisms for conversions of the element contents, and conditional
1712 mappings of elements based on the record contents.
1715 These are the directives of the schema mapping file format:
1718 <tag>targetName <it/name/</tag> (m) A symbolic name for the target schema
1719 of the table. Useful mostly for diagnostic purposes.
1721 <tag>targetRef <it/OID-name/</tag> (m) An OID name for the target schema.
1722 This is used, for instance, by a server receiving a request to present
1723 a record in a different schema from the native one.
1725 <tag>map <it/element-name target-path/</tag> (o,r) Adds
1726 an element mapping rule to the table.
1729 <sect2>The MARC (ISO2709) Representation (.mar) Files
1732 This file provides rules for representing a record in the ISO2709
1733 format. The rules pertain mostly to the values of the constant-length
1734 header of the record.
1736 <it>NOTE: This will be described better. We're in the process of
1737 re-evaluating and most likely changing the way that MARC records are
1738 handled by the system.</it>
1740 <sect2>Field Structure and Character Sets
1741 <label id="field structure and character sets">
1744 In order to provide a flexible approach to national character set
1745 handling, Zebra allows the administrator to configure the set up the
1746 system to handle any 8-bit character set — including sets that
1747 require multi-octet diacritics or other multi-octet characters. The
1748 definition of a character set includes a specification of the
1749 permissible values, their sort order (this affects the display in the
1750 SCAN function), and relationships between upper- and lowercase
1751 characters. Finally, the definition includes the specification of
1752 space characters for the set.
1754 The operator can define different character sets for different fields,
1755 typical examples being standard text fields, numerical fields, and
1756 special-purpose fields such as WWW-style linkages (URx).
1758 The field types, and hence character sets, are associated with data
1759 elements by the .abs files (see above). The file <tt/default.idx/
1760 provides the association between field type codes (as used in the .abs
1761 files) and the character map files (with the .chr suffix). The format
1762 of the .idx file is as follows
1765 <tag>index <it/field type code/</tag>This directive introduces a new
1766 search index code. The argument is a one-character code to be used in the
1767 .abs files to select this particular index type. An index, roughly,
1768 corresponds to a particular structure attribute during search. Refer
1769 to section <ref id="search" name="Search">.
1771 <tag>sort <it/field code type/</tag>This directive introduces a
1772 sort index. The argument is a one-character code to be used in the
1773 .abs fie to select this particular index type. The corresponding
1774 use attribute must be used in the sort request to refer to this
1775 particular sort index. The corresponding character map (see below)
1776 is used in the sort process.
1778 <tag>completeness <it/boolean/</tag>This directive enables or disables
1779 complete field indexing. The value of the <it/boolean/ should be 0
1780 (disable) or 1. If completeness is enabled, the index entry will
1781 contain the complete contents of the field (up to a limit), with words
1782 (non-space characters) separated by single space characters
1783 (normalized to &dquot; &dquot; on display). When completeness is
1784 disabled, each word is indexed as a separate entry. Complete subfield
1785 indexing is most useful for fields which are typically browsed (eg.
1786 titles, authors, or subjects), or instances where a match on a
1787 complete subfield is essential (eg. exact title searching). For fields
1788 where completeness is disabled, the search engine will interpret a
1789 search containing space characters as a word proximity search.
1791 <tag>charmap <it/filename/</tag> This is the filename of the character
1792 map to be used for this index for field type.
1795 The contents of the character map files are structured as follows:
1798 <tag>lowercase <it/value-set/</tag>This directive introduces the basic
1799 value set of the field type. The format is an ordered list (without
1800 spaces) of the characters which may occur in &dquot;words&dquot; of
1801 the given type. The order of the entries in the list determines the
1802 sort order of the index. In addition to single characters, the
1803 following combinations are legal:
1806 <item>Backslashes may be used to introduce three-digit octal, or
1807 two-digit hex representations of single characters (preceded by <tt/x/).
1808 In addition, the combinations
1809 \\, \\r, \\n, \\t, \\s (space — remember that real space-characters
1810 may ot occur in the value definition), and \\ are recognised,
1811 with their usual interpretation.
1813 <item>Curly braces {} may be used to enclose ranges of single
1814 characters (possibly using the escape convention described in the
1815 preceding point), eg. {a-z} to entroduce the standard range of ASCII
1816 characters. Note that the interpretation of such a range depends on
1817 the concrete representation in your local, physical character set.
1819 <item>Paranthesises () may be used to enclose multi-byte characters -
1820 eg. diacritics or special national combinations (eg. Spanish
1821 &dquot;ll&dquot;). When found in the input stream (or a search term),
1822 these characters are viewed and sorted as a single character, with a
1823 sorting value depending on the position of the group in the value
1827 <tag>uppercase <it/value-set/</tag>This directive introduces the
1828 upper-case equivalencis to the value set (if any). The number and
1829 order of the entries in the list should be the same as in the
1830 <tt/lowercase/ directive.
1832 <tag>space <it/value-set/</tag>This directive introduces the character
1833 which separate words in the input stream. Depending on the
1834 completeness mode of the field in question, these characters either
1835 terminate an index entry, or delimit individual &dquot;words&dquot; in
1836 the input stream. The order of the elements is not significant —
1837 otherwise the representation is the same as for the <tt/upercase/ and
1838 <tt/lowercase/ directives.
1840 <tag>map <it/value-set/ <it/target/</tag>This directive introduces a
1841 mapping between each of the members of the value-set on the left to
1842 the character on the right. The character on the right must occur in
1843 the value set (the <tt/lowercase/ directive) of the character set, but
1844 it may be a paranthesis-enclosed multi-octet character. This directive
1845 may be used to map diacritics to their base characters, or to map
1846 HTML-style character-representations to their natural form, etc.
1849 <sect1>Exchange Formats
1852 Converting records from the internal structure to en exchange format
1853 is largely an automatic process. Currently, the following exchange
1854 formats are supported:
1857 <item>GRS-1. The internal representation is based on GRS-1, so the
1858 conversion here is straightforward. The system will create
1859 applied variant and supported variant lists as required, if a record
1860 contains variant information.
1862 <item>SUTRS. Again, the mapping is fairly straighforward. Indentation
1863 is used to show the hierarchical structure of the record. All
1864 &dquot;GRS&dquot; type records support both the GRS-1 and SUTRS
1867 <item>ISO2709-based formats (USMARC, etc.). Only records with a
1868 two-level structure (corresponding to fields and subfields) can be
1869 directly mapped to ISO2709. For records with a different structuring
1870 (eg., GILS), the representation in a structure like USMARC involves a
1871 schema-mapping (see section <ref id="schema-mapping" name="Schema
1872 Mapping">), to an &dquot;implied&dquot; USMARC schema (implied,
1873 because there is no formal schema which specifies the use of the
1874 USMARC fields outside of ISO2709). The resultant, two-level record is
1875 then mapped directly from the internal representation to ISO2709. See
1876 the GILS schema definition files for a detailed example of this
1879 <item>Explain. This representation is only available for records
1880 belonging to the Explain schema.
1882 <item>Summary. This ASN-1 based structure is only available for records
1883 belonging to the Summary schema - or schema which provide a mapping
1884 to this schema (see the description of the schema mapping facility
1887 <item>SOIF. Support for this syntax is experimental, and is currently
1888 keyed to a private Index Data OID (1.2.840.10003.5.1000.81.2). All
1889 abstract syntaxes can be mapped to the SOIF format, although nested
1890 elements are represented by concatenation of the tag names at each
1893 <item>XML. The use of XML as a transfer syntax in Z39.50 is not yet widely established
1894 so the use of it here must be characterised as somewhat experimental. The
1895 tag-names used are taken from the tag-set in use, except for local string tags
1896 where the tag itself is passed through unchanged.
1904 Copyright (c) 1995-2000 Index Data ApS.
1906 All rights reserved.
1908 Use and redistribution in source or binary form, with or without
1909 modification, of any or all of this software and documentation is
1910 permitted, provided that the following Conditions 1 to 6 set out below
1913 1. Unless prior specific written permission is obtained this copyright
1914 and permission notice appear with all copies of the software and its
1915 documentation. Notices of copyright or attribution which appear at the
1916 beginning of any file must remain unchanged.
1918 2. The names of Index Data or the individual authors may not be used
1919 to endorse or promote products derived from this software without
1920 specific prior written permission.
1922 3. Source code or binary versions of this software and its documentation
1923 may be used freely in not for profit applications limited to databases
1924 of 100,000 records maximum. Other applications - such as publishing over
1925 100,000 records, providing for-pay services, distributing a product based
1926 in whole or in part on this software or its documentation, or generally
1927 distributing this software or its documentation under a different license
1928 require a commercial license from Index Data.
1930 4. The software may be installed and used for evaluation purposes in
1931 conjunction with such commercially licensed applications for a trial
1932 period no longer than 60 days.
1934 5. Unless a prior specific written agreement is obtained THIS SOFTWARE
1935 IS PROVIDED "AS IS" AND WITHOUT WARRANTY OF ANY KIND, EXPRESS, IMPLIED,
1936 OR OTHERWISE, INCLUDING WITHOUT LIMITATION, ANY WARRANTY OF
1937 MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. IN NO EVENT SHALL
1938 INDEX DATA BE LIABLE FOR ANY SPECIAL, INCIDENTAL, INDIRECT OR
1939 CONSEQUENTIAL DAMAGES OF ANY KIND, OR ANY DAMAGES WHATSOEVER RESULTING
1940 FROM LOSS OF USE, DATA OR PROFITS, WHETHER OR NOT ADVISED OF THE
1941 POSSIBILITY OF DAMAGE, AND ON ANY THEORY OF LIABILITY, ARISING OUT OF
1942 OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
1944 6. Commercial licenses and support agreements for Zebra and related
1945 Index Data products such as Z'bol (c) - and written agreements
1946 relating to these Conditions may be obtained only from Index Data
1947 or its appointed agents as follows:
1949 Index Data: www.indexdata.dk
1950 Fretwell-Downing Informatics: www.fdgroup.co.uk
1951 Fretwell-Downing Informatics USA: www.fdi.com
1953 <sect>About Index Data and the Zebra Server
1956 Index Data is a consulting and software-development enterprise that
1957 specialises in information management and retrieval applications. Our
1958 interests and expertise span a broad range of related fields, and one
1959 of our primary, long-term objectives is the development of a powerful
1960 information management
1961 system with open network interfaces and hypermedia capabilities. Zebra is an
1962 important component in this strategy.
1964 We make this software available free of charge for not-for-profit
1965 purposes, as a service to the networking community, and to further
1966 the development and use of quality software for open network
1967 communication. We encourage your comments and questions if you have ideas, things
1968 you would like to see in future versions, or things you would like to
1971 If you like this software, and would like to use all or part of it in
1972 a commercial product, or to provide a commercial database service,
1973 please contact us. The Z'mbol Information System represents the commercial
1974 variant of Zebra. It includes full support; additional functionality and
1975 performance-boosting features, and it has what we think is a very exciting
1981 DK-2200 Copenhagen N
1986 Phone: +45 3536 3672
1988 Email: info@indexdata.dk
1991 The <it>Random House College Dictionary</it>, 1975 edition
1992 offers this definition of the
1993 word &dquot;Zebra&dquot;:
1996 Zebra, n., any of several horselike, African mammals of the genus Equus,
1997 having a characteristic pattern of black or dark-brown stripes on
1998 a whitish background.