1 <!doctype linuxdoc system>
4 $Id: zebra.sgml,v 1.5 1995-11-28 16:34:43 quinn Exp $
8 <title>Zebra Server - Administrators's Guide and Reference
9 <author>Index Data, <tt/info@index.ping.dk/
10 <date>$Revision: 1.5 $
12 The Zebra information server combines a versatile fielded/free-text
13 search engine with a Z39.50-1995 frontend to provide a powerful and flexible
14 information management system. This document explains the procedure for
15 installing and configuring the system, and outlines the possibilities
16 for managing data and providing Z39.50
17 services using the software.
27 The Zebra system is a fielded free-text indexing and retrieval engine with a
28 Z39.50 frontend. You can use any commercial or freeware Z39.50 client
29 to access data stored in Zebra.
31 The Zebra server is our first step towards the development of a fully
32 configurable, open information system. Eventually, it will be paired
33 off with a powerful Z39.50 client to support complex information
34 management tasks within almost any application domain. We're making
35 the server available now because it's no fun to be in the open
36 information retrieval business all by yourself. We want to allow
37 people with interesting data to make their things
38 available in interesting ways, without having to start out
39 by implementing yet another protocol stack from scratch.
41 This document is an introduction to the Zebra system. It will tell you
42 how to compile the software, and how to prepare your first database.
43 It also explains how the server can be configured to give you the
44 functionality that you need.
46 You should read <it/Specifying and Using Application (Database)
47 Profiles/, which is bundled with the YAZ documentation, to learn how
48 records are formatted, and how you can configure Zebra to handle
49 different types of Z39.50 application profiles.
54 This is a listing of some of the most important features of the
60 Supports updating - records can be added and deleted without
64 Supports large databases - files for indices, etc. can be
65 automatically partitioned over multiple disks.
68 Supports arbitrarily complex records - base input format is an
69 SGML-like syntax which allows nested (structured) data elements, as
70 well as variant forms of data.
73 Supports boolean queries as well as relevance-ranking (free-text)
74 searching. Right truncation and masking in terms are supported, as
75 well as full regular expressions.
78 Supports multiple concrete syntaxes
79 for record exchange (depending on the configuration): GRS-1, SUTRS,
80 ISO2709 (*MARC). Records can be mapped between record syntaxes and
89 Protocol facilities: Init, Search, Retrieve, Browse.
92 Piggy-backed presents are honored in the search-request.
95 Named result sets are supported.
98 Easily configured to support different application profiles, with
99 tables for attribute sets, tag sets, and abstract syntaxes.
100 Additional tables control facilities such as element mappings to
101 different schema (eg., GILS-to-USMARC).
104 Complex composition specifications using Espec-1 are partially
105 supported (simple element requests only).
108 Element Set Names are established the Espec-1 capability of the
109 system, and are given in configuration files as simple element
110 requests (and possibly variant requests).
113 Some variant support (not fully implemented yet).
116 Using the YAZ toolkit for the protocol implementation, the
117 server can utilise a plug-in XTI/mOSI implementation (not included) to
118 provide SR services over an OSI stack, as well as Z39.50 over TCP/IP.
127 This is an early alfa-release of the software, to allow you to look at
128 it - try it out, and assess whether it can be of use to you. We expect
129 this version to be followed by a succession of beta-releases until we
130 arrive at a stable first version.
132 These are some of the plans that we have for the software in the near
133 and far future, approximately ordered after their relative importance.
135 asterisk will be implemented before the
141 *Allow the system to handle additional input formats. Specifically
142 MARC records and general, structured ASCII records (such as mail/news
143 files) parameterized by regular expressions.
146 *Complete the support for variants. Finalize support for the WAIS
147 retrieval methodology.
150 *Finalize the data element <it/include/ facility to support multimedia
151 data elements in records.
154 *Port the system to Windows NT.
157 Add robust database updating - tolerant to crashes or hard interrupts
158 during register updating.
161 Add online updating, to permit register updating while users are
162 accessing the system.
165 Add index and data compression to save disk space.
168 Add more sophisticated relevance ranking mechanisms. Add support for soundex
169 and stemming. Add relevance feedback support.
175 Add support for very large records by implementing segmentation and
179 Support the Item Update extended service of the protocol.
182 The Zebra search engine supports approximate string matching in the
183 index. We'd like to find a way to support and control this from RPN.
186 We want to add a management system that allows you to
187 control your databases and configuration tables from a graphical
188 interface. We'll probably use Tcl/Tk to stay platform-independent.
192 Programmers thrive on user feedback. If you are interested in a facility that
193 you don't see mentioned here, or if there's something you think we
194 could do better, please drop us a mail. If you think it's all really
195 neat, you're of course welcome to drop us a line saying that, too.
198 <sect>Compiling the software
201 Zebra uses the YAZ package to implement Z39.50, so you
202 have to compile YAZ before going further. Specifically, Zebra uses
203 the YAZ header files in <tt>yaz/include/..</tt> and its public library
204 <tt>yaz/lib/libyaz.a</tt>.
206 As with YAZ, an ANSI C compiler is required in order to compile the Zebra
207 server system — GNU C works fine.
209 Unpack the Zebra software. You might put Zebra in the same directory level
210 as YAZ, for example if YAZ is placed in ..<tt>/src/yaz-</tt>.., then
211 Zebra is placed in ..<tt>/src/zebra-</tt>.
213 Edit the top-level <tt>Makefile</tt> in the Zebra directory in which
214 you specify the location of YAZ by setting make variables.
215 The <tt>OSILIB</tt> should be empty if YAZ wasn't compiled with
216 MOSI support. Some systems, such as Solaris, have separate socket
217 libraries and for those systems you need to specify the
218 <tt>NETLIB</tt> variable.
220 When you are done editing the <tt>Makefile</tt> type:
225 If successful, two executables have been created in the sub-directory
228 <tag><tt>zebrasrv</tt></tag> The Z39.50 server and search engine.
229 <tag><tt>zebraidx</tt></tag> The administrative tool for the search index.
235 This section will get you started quickly! We will try to index a few sample
236 GILS records that are included with the Zebra distribution. Go to the
237 <tt>test</tt> subdirectory. There you will find a configuration
238 file named <tt>zebra.cfg</tt> with the following contents:
240 # Where are the YAZ tables located.
241 profilePath: /usr/local/yaz
243 # Files that describe the attribute sets supported.
248 Now, edit the file and set <tt>profilePath</tt> to the path of the
249 YAZ profile tables (sub directory <tt>tab</tt> of YAZ).
251 The 48 test records are located in the sub directory <tt>records</tt>.
252 To index these, type:
254 $ ../index/zebraidx -t grs update records
257 In the command above the option <tt>-t</tt> specified the record
258 type — in this case <tt>grs</tt>. The word <tt>update</tt> followed
259 by a directory root updates all files below that directory node.
261 If your indexing command went successful, you are now ready to
262 fire up a server. To start a server on port 2100, type:
264 $ ../index/zebrasrv tcp:@:2100
267 The Zebra index that you've just made has one database called Default. It will
268 return either USMARC, GRS-1, or SUTRS depending on what your client asks
271 To test the server, you can use any Z39.50 client (1992 or later). For
272 instance, you can use the demo client that comes with YAZ: Just cd to
273 the <tt/client/ subdirectory of the YAZ distribution and type:
276 $ client tcp:localhost:2100
279 When the client has connected, you can type:
286 To try other retrieval formats for the same record, try:
295 If you've made it this far, there's a reasonably good chance that
296 you've made it through the compilation OK.
298 <sect>Administrating Zebra
302 Unlike many other retrieval systems, Zebra offers incremental
303 modifications of an existing index. Needless to say, these facilities
304 make the administration of Zebra a bit more complicated than
305 systems that use the &dquot;index-it-all&dquot; approach.
307 Normally, when Zebra modifies the index it reads a number of records
309 Depending on your specifications and on the contents of each record
310 one the following events take place for each record:
312 <tag>Insert</tag> The record is indexed as if it never occurred
313 before. Either the Zebra system doesn't know how to identify the record or
314 Zebra can identify the record but didn't find it to be already indexed.
315 <tag>Modify</tag> The record has already been indexed. In this case
316 either the contents of the record or the location (file) of the record
317 indicates that it has been indexed before.
318 <tag>Delete</tag> The record is deleted from the index. As in the
319 update-case it must be able to identify the record.
322 Please note that in both the modify- and delete- case the Zebra
323 indexer must be able to make a unique key that identifies the record in
326 To administrate the Zebra retrieval system, you run the
327 <tt>zebraidx</tt> program. This program supports a number of options
328 which are preceded by a minus, and a few commands (not preceded by
331 Both the Zebra administrative tool and the Z39.50 server share a
332 set of index files and a global configuration file. The
333 name of the configuration file defaults to <tt>zebra.cfg</tt>.
334 The configuration file includes specifications on how to index
335 various kinds of records and where the other configuration files
336 are located. <tt>zebrasrv</tt> and <tt>zebraidx</tt> <em>must</em>
337 be run in the same directory where the configuration file if you do
338 not indicate the location of the configuration file by option
343 Indexing is a record-per-record process, in which
344 either insert/modify/delete will occur. Before a record is indexed
345 search keys are extracted from whatever might be the layout the
346 original record (sgml,html,text, etc..). The Zebra system
347 currently only supports SGML-like, structured records and unstructured text
349 To specify a particular extraction process, use either the
350 command line option <tt>-t</tt> or specify a
351 <tt>recordType</tt> setting in the configuration file.
353 <sect1>The Zebra Configuration File
355 The Zebra configuration file, read by <tt>zebraidx</tt> and
356 <tt>zebrasrv</tt> defaults to <tt>zebra.cfg</tt> unless specified
357 by <tt>-c</tt> option.
359 You can edit the configuration file with a normal text editor.
360 Setting names and values are seperated by colons in the file. Lines
361 starting with a hash sign (<tt/#/) are treated as comments.
363 A set of records that share common characteristics are called a group.
364 When <tt>zebraidx</tt> is run and you wish to address a given group
365 you specify that group with the <tt>-g</tt> option. In this case
366 settings that have the group name as their prefix will be used
367 by <tt>zebraidx</tt> and not default values. The default values have no prefix.
369 The group is written before the option itself separated by a dot.
370 For instance, to set the record type for group <tt/public/ to <tt/grs/ (structured records)
374 public.recordType: grs
377 To set the default value of the record type to text write:
383 The configuration settings are summarized below. They will be
384 explained further in the following sections.
387 <tag><it>group</it>recordType<it>name</it></tag>
388 Specifies how records with the file extension <it>name</it> should
389 be handled by the indexer. This option may also be specified
390 as a command line option (<tt>-t</tt>).
391 <tag><it>group</it>recordId</tag>
392 Specifies how the record is to be identified when updated.
393 <tag><it>group</it>database</tag>
394 Specifies the Z39.50 database.
395 <tag><it>group</it>storeKeys</tag>
396 Specifies whether key information should be saved for a given
397 group of records. If you plan to update/delete this type of
398 records later this should be specified as 1; otherwise it
399 should be 0 (default).
400 <tag><it>group</it>storeData</tag>
401 Specifies whether the records should be stored internally
402 in the Zebra system tables. If you want to maintain the raw records yourself,
403 this option should be false (0). If you want Zebra to take care of the records
404 for you, it should be true(1).
406 Specifies the location of the various files that Zebra uses to represent
408 <tag>profilePath</tag>
409 Specifies the location of profile specification paths.
411 Specifies the filename(s) of attribute set files for use in searching.
414 <sect1>Locating Records
416 The default behaviour of the Zebra system is to reference the
417 records from their original location, i.e. where they were found when you
420 If your records files are temporary - for example if you retrieve
421 them from the outside, or if they where temporarily mounted on a CD-ROM,
422 you may want Zebra to make a copy of them. To do this,
423 you specify 1 (true) in the <tt>storedata</tt> setting. When
424 the Z39.50 server retrieves records they will be read from the
425 internal file structures of the system.
427 <sect1>Indexing with no Record IDs (Simple Indexing)
430 If you have a set of records that you <em/never/ wish to delete
431 or modify you may find &dquot;indexing without records IDs&dquot; convenient.
432 This indexing method uses less space than the other methods and
435 To use this method, you simply don't provide the <tt>recordId</tt> entry
436 for the group of files that you index. To add a set of records you use
437 <tt>zebraidx</tt> with the <tt>update</tt> command. The
438 <tt>update</tt> command will always add all of the records to the index
439 becuase Zebra doesn't know how to match the new set of records with
442 Consider a system in which you have a group of text files called
443 <tt>simple</tt>. That group of records should belong to a Z39.50 database
444 called <tt>textbase</tt>. The following <tt/zebra.cfg/ file will suffice:
447 profilePath: /usr/local/yaz
450 simple.recordType: text
451 simple.database: textbase
454 <sect1>Indexing with File Record IDs
457 If you have a set of external records that you wish to index you may
458 use the file key feature of the Zebra system. In short, the file key
459 feature mirrors a directory structure and its files efficiently. To
460 perform indexing of a directory with file keys, you specify the top-level
461 directory after the <tt>update</tt> command. The command will recursively
462 traverse the directories and compare each with whatever have been
463 indexed before in the same directory. If a file is new (not in
464 the previous version of the directory) it is inserted;
465 if a file was already indexed and it has been modified
466 since the last insertion the index is also modified; if a file is missing
467 since the last visit it is deleted from the index.
469 The resulting system is easy to administer. To delete a record
470 you simply have to delete the corresponding file (with <tt/rm/).
471 To force update of a given file, you may use the <tt>touch</tt>
472 command. And to add files create new files (or directories with files).
473 For your changes to take effect you must run <tt>zebraidx</tt> with
474 the same directory root again.
476 To use this method, you must specify <tt>file</tt> as the value
477 of <tt>recordId</tt> in the configuration file. In the configuration
478 also set <tt>storeKeys</tt> to <tt>1</tt>, since the Zebra
479 indexer must save additional information per record in order to
480 modify/delete the records at a later time.
482 For example, to update group <tt>esdd</tt> records below
483 <tt>/home/grs</tt> you could type:
485 $ zebraidx -g esdd update /home/grs
488 The corresponding configuration file includes:
495 <em>Important note: You cannot start out with a group of records with simple
496 indexing (no record IDs as in the previous section) and then later
497 enable file record Ids. Zebra must know from the first time that you
499 the files should be indexed with file record IDs.
502 <sect1>Indexing with General Record IDs
504 When using this method you specify an (almost) arbritrary record key
505 based on the contents of the record itself and other system
506 information. If you have a group of records that have an external
507 ID associated with each records, this method is convenient. For
508 example, the record may contain a title or a unique ID-number. In either
509 case you specify the Z39.50 attribute set and use-attribute location
510 in which this information is stored.
512 As before, the record ID is defined by the <tt>recordId</tt> setting
513 in the configuration file. The value of the record ID specification
514 consists of one or more tokens separated by whitespace. The resulting
516 represented in the index by concatenating the tokens and separating them by
519 There are three kinds of tokens:
521 <tag>Internal record info</tag> The token refers to a key that is
522 extracted from the record. The syntax of this token is
523 <tt/(/ <em/set/ <tt/,/ <em/use/ <tt/)/, where <em/set/ is the
524 attribute set ordinal number and <em/use/ is the use value of the attribute.
525 <tag>System variable</tag> The system variables are preceded by
526 <tt>$</tt> and immediately followed by the system variable name, which
529 <tag>group</tag> Group name.
530 <tag>database</tag> Current database specified.
531 <tag>type</tag> Record type.
533 <tag>Constant string</tag> A string used as part of id — surrounded
534 by single- or double quotes.
537 The test GILS records that comes with the Zebra distribution contain a
539 in the Control-Identifier field. This field is mapped to the Bib-1
540 use attribute 1007. To use this field as a record id, specify
541 <tt>(1,1007)</tt> as the value of the <tt>recordId</tt> in the
542 configuration file. If you have other record types that don't
543 contain an ID in the same field, you might add the record type
544 in the record id of the gils records as well, to prevent matches
545 of other types of records. In this case the recordId might be
548 gils.recordId: $type (1,1007)
551 As for the file record id case described in the previous section
552 updating your system is simply a matter of running <tt>zebraidx</tt>
553 with the <tt>update</tt> command. However, the update with general
554 keys is considerably slower than with file record IDs, since all files
555 visited must be (re)read to find their IDs.
557 <sect1>Register location
560 Normally, the index files that form dictionaries, inverted
561 files, record info, etc., are stored in the directory where you run
562 <tt>zebraidx</tt>. If you wish to store these, possibly large, files
563 somewhere else, you must add the <tt>register</tt> entry to the
564 configuration file. Furthermore, the Zebra system allows its file
566 span multiple file systems, which is useful if a very large number of
569 The value <tt>register</tt> of register is a sequence of tokens.
570 Each token takes the form:
572 <em>dir</em><tt>:</tt><em>size</em>.
574 The <em>dir</em> specifies a directory in which index files will be
575 stored and the <em>size</em> specifies the maximum size of all
576 files in that directory. The Zebra indexer system fill each directory
577 in the order specified and use the next specified directories as needed.
578 The <em>size</em> is an integer followed by a qualifier
579 code, <tt>M</tt> for megabytes, <tt>k</tt> for kilobytes.
581 For instance, if you have two spare disks :) and the first disk is mounted
582 on <tt>/d1</tt> and has 200 Mb of free space and the
583 second, mounted on <tt>/d2</tt> has 300 Mb, you could
584 put this entry in your configuration file:
586 register: /d1:200M /d2:300M
589 <sect>The Z39.50 Server
593 <sect1>Running the server
595 The server <tt>zebrasrv</tt> supports the same set of options as the
596 test server <tt>ztest</tt> that comes with YAZ. As for the
597 <tt>zebraidx</tt> the option <tt>-c</tt> specifies the configuration
598 filename. When the Zebra server is executed with its normal log level it
599 prints (not too detailed) information about the incoming queries.
600 This is useful if you don't happen to know what attributes your client sends.
602 Note that the server doesn't support the static mode (-S).
604 <sect1>How the server handles queries
606 What elements of Bib-1 are supported and where are result sets
612 Copyright © 1995, Index Data.
616 Use and redistribution in source or binary form, with or without
617 modification, of any or all of this software and documentation is
618 permitted, provided that the following conditions are met:
620 1. This copyright and permission notice appear with all copies of the
621 software and its documentation. Notices of copyright or attribution
622 which appear at the beginning of any file must remain unchanged.
624 2. The names of Index Data or the individual authors may not be used to
625 endorse or promote products derived from this software without specific
626 prior written permission.
628 3. Source code or binary versions of this software and its documentation
629 may be used in not-for-profit applications. For profit aplications -
630 including marketing a product based in whole or in part on this software,
631 or providing for-pay database services - must obtain a commercial
632 license from Index Data.
634 THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT WARRANTY OF ANY KIND,
635 EXPRESS, IMPLIED, OR OTHERWISE, INCLUDING WITHOUT LIMITATION, ANY
636 WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
637 IN NO EVENT SHALL INDEX DATA BE LIABLE FOR ANY SPECIAL, INCIDENTAL,
638 INDIRECT OR CONSEQUENTIAL DAMAGES OF ANY KIND, OR ANY DAMAGES
639 WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER OR
640 NOT ADVISED OF THE POSSIBILITY OF DAMAGE, AND ON ANY THEORY OF
641 LIABILITY, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE
644 <sect>About Index Data
647 Index Data is a consulting and software-development enterprise that
648 specialises in library and information management systems. Our
649 interests and expertise span a broad range of related fields, and one
650 of our primary, long-term objectives is the development of a powerful
651 information management
652 system with open network interfaces and hypermedia capabilities.
654 We make this software available free of charge for noncommercial
655 purposes, as a service to the networking community, and to further
656 the development of quality software for open network communication.
658 We'll be happy to answer questions about the software, and about ourselves
664 DK-2200 København N&nl
671 Email: info@index.ping.dk
674 The <it>Random House College Dictionary</it>, 1975 edition
675 offers this definition of the
676 word &dquot;Zebra&dquot;:
679 Zebra, n., any of several horselike, African mammals of the genus Equus,
680 having a characteristic pattern of black or dark-brown stripes on
681 a whitish background.