1 <chapter id="administration">
2 <!-- $Id: administration.xml,v 1.5 2002-04-10 14:47:48 heikki Exp $ -->
3 <title>Administrating Zebra</title>
6 Unlike many simpler retrieval systems, Zebra supports safe, incremental
7 updates to an existing index.
11 Normally, when Zebra modifies the index it reads a number of records
13 Depending on your specifications and on the contents of each record
14 one the following events take place for each record:
21 The record is indexed as if it never occurred before.
22 Either the Zebra system doesn't know how to identify the record or
23 Zebra can identify the record but didn't find it to be already indexed.
31 The record has already been indexed.
32 In this case either the contents of the record or the location
33 (file) of the record indicates that it has been indexed before.
41 The record is deleted from the index. As in the
42 update-case it must be able to identify the record.
50 Please note that in both the modify- and delete- case the Zebra
51 indexer must be able to generate a unique key that identifies the record
52 in question (more on this below).
56 To administrate the Zebra retrieval system, you run the
57 <literal>zebraidx</literal> program.
58 This program supports a number of options which are preceded by a dash,
59 and a few commands (not preceded by dash).
63 Both the Zebra administrative tool and the Z39.50 server share a
64 set of index files and a global configuration file.
65 The name of the configuration file defaults to
66 <literal>zebra.cfg</literal>.
67 The configuration file includes specifications on how to index
68 various kinds of records and where the other configuration files
69 are located. <literal>zebrasrv</literal> and <literal>zebraidx</literal>
70 <emphasis>must</emphasis> be run in the directory where the
71 configuration file lives unless you indicate the location of the
72 configuration file by option <literal>-c</literal>.
75 <sect1 id="record-types">
76 <title>Record Types</title>
79 Indexing is a per-record process, in which either insert/modify/delete
80 will occur. Before a record is indexed search keys are extracted from
81 whatever might be the layout the original record (sgml,html,text, etc..).
82 The Zebra system currently supports two fundamental types of records:
83 structured and simple text.
84 To specify a particular extraction process, use either the
85 command line option <literal>-t</literal> or specify a
86 <literal>recordType</literal> setting in the configuration file.
91 <sect1 id="configuration-file">
92 <title>The Zebra Configuration File</title>
95 The Zebra configuration file, read by <literal>zebraidx</literal> and
96 <literal>zebrasrv</literal> defaults to <literal>zebra.cfg</literal>
97 unless specified by <literal>-c</literal> option.
101 You can edit the configuration file with a normal text editor.
102 parameter names and values are separated by colons in the file. Lines
103 starting with a hash sign (<literal>#</literal>) are
108 If you manage different sets of records that share common
109 characteristics, you can organize the configuration settings for each
111 When <literal>zebraidx</literal> is run and you wish to address a
112 given group you specify the group name with the <literal>-g</literal>
114 In this case settings that have the group name as their prefix
115 will be used by <literal>zebraidx</literal>.
116 If no <literal>-g</literal> option is specified, the settings
117 without prefix are used.
121 In the configuration file, the group name is placed before the option
122 name itself, separated by a dot (.). For instance, to set the record type
123 for group <literal>public</literal> to <literal>grs.sgml</literal>
124 (the SGML-like format for structured records) you would write:
129 public.recordType: grs.sgml
134 To set the default value of the record type to <literal>text</literal>
145 The available configuration settings are summarized below. They will be
146 explained further in the following sections.
150 FIXME - Didn't Adam make something to have multiple databases in multiple dirs...
158 <emphasis>group</emphasis>
159 .recordType[<emphasis>.name</emphasis>]:
160 <replaceable>type</replaceable>
164 Specifies how records with the file extension
165 <emphasis>name</emphasis> should be handled by the indexer.
166 This option may also be specified as a command line option
167 (<literal>-t</literal>). Note that if you do not specify a
168 <emphasis>name</emphasis>, the setting applies to all files.
169 In general, the record type specifier consists of the elements (each
170 element separated by dot), <emphasis>fundamental-type</emphasis>,
171 <emphasis>file-read-type</emphasis> and arguments. Currently, two
172 fundamental types exist, <literal>text</literal> and
173 <literal>grs</literal>.
178 <term><emphasis>group</emphasis>.recordId:
179 <replaceable>record-id-spec</replaceable></term>
182 Specifies how the records are to be identified when updated. See
183 <xref linkend="locating-records"/>.
188 <term><emphasis>group</emphasis>.database:
189 <replaceable>database</replaceable></term>
192 Specifies the Z39.50 database name.
193 FIXME - now we can have multiple databases in one server. -H
198 <term><emphasis>group</emphasis>.storeKeys:
199 <replaceable>boolean</replaceable></term>
202 Specifies whether key information should be saved for a given
203 group of records. If you plan to update/delete this type of
204 records later this should be specified as 1; otherwise it
205 should be 0 (default), to save register space.
206 See <xref linkend="file-ids"/>.
211 <term><emphasis>group</emphasis>.storeData:
212 <replaceable>boolean</replaceable></term>
215 Specifies whether the records should be stored internally
216 in the Zebra system files.
217 If you want to maintain the raw records yourself,
218 this option should be false (0).
219 If you want Zebra to take care of the records for you, it
225 <term>register: <replaceable>register-location</replaceable></term>
228 Specifies the location of the various register files that Zebra uses
229 to represent your databases.
230 See <xref linkend="register-location"/>.
235 <term>shadow: <replaceable>register-location</replaceable></term>
238 Enables the <emphasis>safe update</emphasis> facility of Zebra, and
239 tells the system where to place the required, temporary files.
240 See <xref linkend="shadow-registers"/>.
245 <term>lockDir: <replaceable>directory</replaceable></term>
248 Directory in which various lock files are stored.
253 <term>keyTmpDir: <replaceable>directory</replaceable></term>
256 Directory in which temporary files used during zebraidx' update
262 <term>setTmpDir: <replaceable>directory</replaceable></term>
265 Specifies the directory that the server uses for temporary result sets.
266 If not specified <literal>/tmp</literal> will be used.
271 <term>profilePath: <literal>path</literal></term>
274 Specifies a path of profile specification files.
275 The path is composed of one or more directories separated by
276 colon. Similar to PATH for UNIX systems.
281 <term>attset: <replaceable>filename</replaceable></term>
284 Specifies the filename(s) of attribute set files for use in
285 searching. At least the Bib-1 set should be loaded
286 (<literal>bib1.att</literal>).
287 The <literal>profilePath</literal> setting is used to look for
289 See <xref linkend="attset-files"/>
294 <term>memMax: <replaceable>size</replaceable></term>
297 Specifies <replaceable>size</replaceable> of internal memory
298 to use for the zebraidx program.
299 The amount is given in megabytes - default is 4 (4 MB).
305 <term>root: <replaceable>dir</replaceable></term>
308 Specifies a directory base for Zebra. All relative paths
309 given (in profilePath, register, shadow) are based on this
310 directory. This setting is useful if if you Zebra server
311 is running in a different directory from where
312 <literal>zebra.cfg</literal> is located.
322 <sect1 id="locating-records">
323 <title>Locating Records</title>
326 The default behavior of the Zebra system is to reference the
327 records from their original location, i.e. where they were found when you
328 ran <literal>zebraidx</literal>.
329 That is, when a client wishes to retrieve a record
330 following a search operation, the files are accessed from the place
331 where you originally put them - if you remove the files (without
332 running <literal>zebraidx</literal> again, the client
333 will receive a diagnostic message.
337 If your input files are not permanent - for example if you retrieve
338 your records from an outside source, or if they were temporarily
339 mounted on a CD-ROM drive,
340 you may want Zebra to make an internal copy of them. To do this,
341 you specify 1 (true) in the <literal>storeData</literal> setting. When
342 the Z39.50 server retrieves the records they will be read from the
343 internal file structures of the system.
348 <sect1 id="simple-indexing">
349 <title>Indexing with no Record IDs (Simple Indexing)</title>
352 If you have a set of records that are not expected to change over time
353 you may can build your database without record IDs.
354 This indexing method uses less space than the other methods and
359 To use this method, you simply omit the <literal>recordId</literal> entry
360 for the group of files that you index. To add a set of records you use
361 <literal>zebraidx</literal> with the <literal>update</literal> command. The
362 <literal>update</literal> command will always add all of the records that it
363 encounters to the index - whether they have already been indexed or
364 not. If the set of indexed files change, you should delete all of the
365 index files, and build a new index from scratch.
369 Consider a system in which you have a group of text files called
370 <literal>simple</literal>.
371 That group of records should belong to a Z39.50 database called
372 <literal>textbase</literal>.
373 The following <literal>zebra.cfg</literal> file will suffice:
378 profilePath: /usr/local/yaz
380 simple.recordType: text
381 simple.database: textbase
387 Since the existing records in an index can not be addressed by their
388 IDs, it is impossible to delete or modify records when using this method.
393 <sect1 id="file-ids">
394 <title>Indexing with File Record IDs</title>
397 If you have a set of files that regularly change over time: Old files
398 are deleted, new ones are added, or existing files are modified, you
399 can benefit from using the <emphasis>file ID</emphasis>
400 indexing methodology.
401 Examples of this type of database might include an index of WWW
402 resources, or a USENET news spool area.
403 Briefly speaking, the file key methodology uses the directory paths
404 of the individual records as a unique identifier for each record.
405 To perform indexing of a directory with file keys, again, you specify
406 the top-level directory after the <literal>update</literal> command.
407 The command will recursively traverse the directories and compare
408 each one with whatever have been indexed before in that same directory.
409 If a file is new (not in the previous version of the directory) it
410 is inserted into the registers; if a file was already indexed and
411 it has been modified since the last update, the index is also
412 modified; if a file has been removed since the last
413 visit, it is deleted from the index.
417 The resulting system is easy to administrate. To delete a record you
418 simply have to delete the corresponding file (say, with the
419 <literal>rm</literal> command). And to add records you create new
420 files (or directories with files). For your changes to take effect
421 in the register you must run <literal>zebraidx update</literal> with
422 the same directory root again. This mode of operation requires more
423 disk space than simpler indexing methods, but it makes it easier for
424 you to keep the index in sync with a frequently changing set of data.
425 If you combine this system with the <emphasis>safe update</emphasis>
426 facility (see below), you never have to take your server off-line for
427 maintenance or register updating purposes.
431 To enable indexing with pathname IDs, you must specify
432 <literal>file</literal> as the value of <literal>recordId</literal>
433 in the configuration file. In addition, you should set
434 <literal>storeKeys</literal> to <literal>1</literal>, since the Zebra
435 indexer must save additional information about the contents of each record
436 in order to modify the indices correctly at a later time.
440 FIXME - There must be a simpler way to do this with Adams string tags -H
444 For example, to update records of group <literal>esdd</literal>
446 <literal>/data1/records/</literal> you should type:
448 $ zebraidx -g esdd update /data1/records
453 The corresponding configuration file includes:
456 esdd.recordType: grs.sgml
462 <para>You cannot start out with a group of records with simple
463 indexing (no record IDs as in the previous section) and then later
464 enable file record Ids. Zebra must know from the first time that you
466 the files should be indexed with file record IDs.
471 You cannot explicitly delete records when using this method (using the
472 <literal>delete</literal> command to <literal>zebraidx</literal>. Instead
473 you have to delete the files from the file system (or move them to a
475 and then run <literal>zebraidx</literal> with the
476 <literal>update</literal> command.
480 <sect1 id="generic-ids">
481 <title>Indexing with General Record IDs</title>
484 When using this method you construct an (almost) arbitrary, internal
485 record key based on the contents of the record itself and other system
486 information. If you have a group of records that explicitly associates
487 an ID with each record, this method is convenient. For example, the
488 record format may contain a title or a ID-number - unique within the group.
489 In either case you specify the Z39.50 attribute set and use-attribute
490 location in which this information is stored, and the system looks at
491 that field to determine the identity of the record.
495 As before, the record ID is defined by the <literal>recordId</literal>
496 setting in the configuration file. The value of the record ID specification
497 consists of one or more tokens separated by whitespace. The resulting
498 ID is represented in the index by concatenating the tokens and
499 separating them by ASCII value (1).
503 There are three kinds of tokens:
507 <term>Internal record info</term>
510 The token refers to a key that is
511 extracted from the record. The syntax of this token is
512 <literal>(</literal> <emphasis>set</emphasis> <literal>,</literal>
513 <emphasis>use</emphasis> <literal>)</literal>,
514 where <emphasis>set</emphasis> is the
515 attribute set name <emphasis>use</emphasis> is the
516 name or value of the attribute.
521 <term>System variable</term>
524 The system variables are preceded by
529 and immediately followed by the system variable name, which
542 <term>database</term>
545 Current database specified.
562 <term>Constant string</term>
565 A string used as part of the ID — surrounded
566 by single- or double quotes.
574 For instance, the sample GILS records that come with the Zebra
575 distribution contain a unique ID in the data tagged Control-Identifier.
576 The data is mapped to the Bib-1 use attribute Identifier-standard
577 (code 1007). To use this field as a record id, specify
578 <literal>(bib1,Identifier-standard)</literal> as the value of the
579 <literal>recordId</literal> in the configuration file.
580 If you have other record types that uses the same field for a
581 different purpose, you might add the record type
582 (or group or database name) to the record id of the gils
583 records as well, to prevent matches with other types of records.
584 In this case the recordId might be set like this:
587 gils.recordId: $type (bib1,Identifier-standard)
593 (see <xref linkend="data-model"/>
594 for details of how the mapping between elements of your records and
595 searchable attributes is established).
599 As for the file record ID case described in the previous section,
600 updating your system is simply a matter of running
601 <literal>zebraidx</literal>
602 with the <literal>update</literal> command. However, the update with general
603 keys is considerably slower than with file record IDs, since all files
604 visited must be (re)read to discover their IDs.
608 As you might expect, when using the general record IDs
609 method, you can only add or modify existing records with the
610 <literal>update</literal> command.
611 If you wish to delete records, you must use the,
612 <literal>delete</literal> command, with a directory as a parameter.
613 This will remove all records that match the files below that root
619 <sect1 id="register-location">
620 <title>Register Location</title>
623 Normally, the index files that form dictionaries, inverted
624 files, record info, etc., are stored in the directory where you run
625 <literal>zebraidx</literal>. If you wish to store these, possibly large,
626 files somewhere else, you must add the <literal>register</literal>
627 entry to the <literal>zebra.cfg</literal> file.
628 Furthermore, the Zebra system allows its file
629 structures to span multiple file systems, which is useful for
630 managing very large databases.
634 The value of the <literal>register</literal> setting is a sequence
635 of tokens. Each token takes the form:
638 <emphasis>dir</emphasis><literal>:</literal><emphasis>size</emphasis>.
641 The <emphasis>dir</emphasis> specifies a directory in which index files
642 will be stored and the <emphasis>size</emphasis> specifies the maximum
643 size of all files in that directory. The Zebra indexer system fills
644 each directory in the order specified and use the next specified
645 directories as needed.
646 The <emphasis>size</emphasis> is an integer followed by a qualifier
648 <literal>b</literal> for bytes,
649 <literal>k</literal> for kilobytes.
650 <literal>M</literal> for megabytes,
651 <literal>G</literal> for gigabytes.
655 For instance, if you have allocated two disks for your register, and
656 the first disk is mounted
657 on <literal>/d1</literal> and has 2GB of free space and the
658 second, mounted on <literal>/d2</literal> has 3.6 GB, you could
659 put this entry in your configuration file:
662 register: /d1:2G /d2:3600M
668 Note that Zebra does not verify that the amount of space specified is
669 actually available on the directory (file system) specified - it is
670 your responsibility to ensure that enough space is available, and that
671 other applications do not attempt to use the free space. In a large
672 production system, it is recommended that you allocate one or more
673 file system exclusively to the Zebra register files.
678 <sect1 id="shadow-registers">
679 <title>Safe Updating - Using Shadow Registers</title>
682 <title>Description</title>
685 The Zebra server supports <emphasis>updating</emphasis> of the index
686 structures. That is, you can add, modify, or remove records from
687 databases managed by Zebra without rebuilding the entire index.
688 Since this process involves modifying structured files with various
689 references between blocks of data in the files, the update process
690 is inherently sensitive to system crashes, or to process interruptions:
691 Anything but a successfully completed update process will leave the
692 register files in an unknown state, and you will essentially have no
693 recourse but to re-index everything, or to restore the register files
694 from a backup medium.
695 Further, while the update process is active, users cannot be
696 allowed to access the system, as the contents of the register files
697 may change unpredictably.
701 You can solve these problems by enabling the shadow register system in
703 During the updating procedure, <literal>zebraidx</literal> will temporarily
704 write changes to the involved files in a set of "shadow
705 files", without modifying the files that are accessed by the
706 active server processes. If the update procedure is interrupted by a
707 system crash or a signal, you simply repeat the procedure - the
708 register files have not been changed or damaged, and the partially
709 written shadow files are automatically deleted before the new updating
714 At the end of the updating procedure (or in a separate operation, if
715 you so desire), the system enters a "commit mode". First,
716 any active server processes are forced to access those blocks that
717 have been changed from the shadow files rather than from the main
718 register files; the unmodified blocks are still accessed at their
719 normal location (the shadow files are not a complete copy of the
720 register files - they only contain those parts that have actually been
721 modified). If the commit process is interrupted at any point during the
722 commit process, the server processes will continue to access the
723 shadow files until you can repeat the commit procedure and complete
724 the writing of data to the main register files. You can perform
725 multiple update operations to the registers before you commit the
726 changes to the system files, or you can execute the commit operation
727 at the end of each update operation. When the commit phase has
728 completed successfully, any running server processes are instructed to
729 switch their operations to the new, operational register, and the
730 temporary shadow files are deleted.
736 <title>How to Use Shadow Register Files</title>
739 The first step is to allocate space on your system for the shadow
741 You do this by adding a <literal>shadow</literal> entry to the
742 <literal>zebra.cfg</literal> file.
743 The syntax of the <literal>shadow</literal> entry is exactly the
744 same as for the <literal>register</literal> entry
745 (see <xref linkend="register-location"/>).
746 The location of the shadow area should be
747 <emphasis>different</emphasis> from the location of the main register
748 area (if you have specified one - remember that if you provide no
749 <literal>register</literal> setting, the default register area is the
750 working directory of the server and indexing processes).
754 The following excerpt from a <literal>zebra.cfg</literal> file shows
755 one example of a setup that configures both the main register
756 location and the shadow file area.
757 Note that two directories or partitions have been set aside
758 for the shadow file area. You can specify any number of directories
759 for each of the file areas, but remember that there should be no
760 overlaps between the directories used for the main registers and the
761 shadow files, respectively.
768 shadow: /scratch1:100M /scratch2:200M
774 When shadow files are enabled, an extra command is available at the
775 <literal>zebraidx</literal> command line.
776 In order to make changes to the system take effect for the
777 users, you'll have to submit a "commit" command after a
778 (sequence of) update operation(s).
779 You can ask the indexer to commit the changes immediately
780 after the update operation:
786 $ zebraidx update /d1/records update /d2/more-records commit
792 Or you can execute multiple updates before committing the changes:
798 $ zebraidx -g books update /d1/records update /d2/more-records
799 $ zebraidx -g fun update /d3/fun-records
806 If one of the update operations above had been interrupted, the commit
807 operation on the last line would fail: <literal>zebraidx</literal>
808 will not let you commit changes that would destroy the running register.
809 You'll have to rerun all of the update operations since your last
810 commit operation, before you can commit the new changes.
814 Similarly, if the commit operation fails, <literal>zebraidx</literal>
815 will not let you start a new update operation before you have
816 successfully repeated the commit operation.
817 The server processes will keep accessing the shadow files rather
818 than the (possibly damaged) blocks of the main register files
819 until the commit operation has successfully completed.
823 You should be aware that update operations may take slightly longer
824 when the shadow register system is enabled, since more file access
825 operations are involved. Further, while the disk space required for
826 the shadow register data is modest for a small update operation, you
827 may prefer to disable the system if you are adding a very large number
828 of records to an already very large database (we use the terms
829 <emphasis>large</emphasis> and <emphasis>modest</emphasis>
830 very loosely here, since every application will have a
831 different perception of size).
832 To update the system without the use of the the shadow files,
833 simply run <literal>zebraidx</literal> with the <literal>-n</literal>
834 option (note that you do not have to execute the
835 <emphasis>commit</emphasis> command of <literal>zebraidx</literal>
836 when you temporarily disable the use of the shadow registers in
838 Note also that, just as when the shadow registers are not enabled,
839 server processes will be barred from accessing the main register
840 while the update procedure takes place.
848 <!-- Keep this comment at the end of the file
853 sgml-minimize-attributes:nil
854 sgml-always-quote-attributes:t
857 sgml-parent-document: "zebra.xml"
858 sgml-local-catalogs: nil
859 sgml-namecase-general:t