1 <chapter id="administration">
2 <!-- $Id: administration.xml,v 1.18 2005-05-30 13:22:11 adam Exp $ -->
3 <title>Administrating Zebra</title>
4 <!-- ### It's a bit daft that this chapter (which describes half of
5 the configuration-file formats) is separated from
6 "recordmodel.xml" (which describes the other half) by the
7 instructions on running zebraidx and zebrasrv. Some careful
8 re-ordering is required here.
12 Unlike many simpler retrieval systems, Zebra supports safe, incremental
13 updates to an existing index.
17 Normally, when Zebra modifies the index it reads a number of records
19 Depending on your specifications and on the contents of each record
20 one the following events take place for each record:
27 The record is indexed as if it never occurred before.
28 Either the Zebra system doesn't know how to identify the record or
29 Zebra can identify the record but didn't find it to be already indexed.
37 The record has already been indexed.
38 In this case either the contents of the record or the location
39 (file) of the record indicates that it has been indexed before.
47 The record is deleted from the index. As in the
48 update-case it must be able to identify the record.
56 Please note that in both the modify- and delete- case the Zebra
57 indexer must be able to generate a unique key that identifies the record
58 in question (more on this below).
62 To administrate the Zebra retrieval system, you run the
63 <literal>zebraidx</literal> program.
64 This program supports a number of options which are preceded by a dash,
65 and a few commands (not preceded by dash).
69 Both the Zebra administrative tool and the Z39.50 server share a
70 set of index files and a global configuration file.
71 The name of the configuration file defaults to
72 <literal>zebra.cfg</literal>.
73 The configuration file includes specifications on how to index
74 various kinds of records and where the other configuration files
75 are located. <literal>zebrasrv</literal> and <literal>zebraidx</literal>
76 <emphasis>must</emphasis> be run in the directory where the
77 configuration file lives unless you indicate the location of the
78 configuration file by option <literal>-c</literal>.
81 <sect1 id="record-types">
82 <title>Record Types</title>
85 Indexing is a per-record process, in which either insert/modify/delete
86 will occur. Before a record is indexed search keys are extracted from
87 whatever might be the layout the original record (sgml,html,text, etc..).
88 The Zebra system currently supports two fundamental types of records:
89 structured and simple text.
90 To specify a particular extraction process, use either the
91 command line option <literal>-t</literal> or specify a
92 <literal>recordType</literal> setting in the configuration file.
97 <sect1 id="configuration-file">
98 <title>The Zebra Configuration File</title>
101 The Zebra configuration file, read by <literal>zebraidx</literal> and
102 <literal>zebrasrv</literal> defaults to <literal>zebra.cfg</literal>
103 unless specified by <literal>-c</literal> option.
107 You can edit the configuration file with a normal text editor.
108 parameter names and values are separated by colons in the file. Lines
109 starting with a hash sign (<literal>#</literal>) are
114 If you manage different sets of records that share common
115 characteristics, you can organize the configuration settings for each
117 When <literal>zebraidx</literal> is run and you wish to address a
118 given group you specify the group name with the <literal>-g</literal>
120 In this case settings that have the group name as their prefix
121 will be used by <literal>zebraidx</literal>.
122 If no <literal>-g</literal> option is specified, the settings
123 without prefix are used.
127 In the configuration file, the group name is placed before the option
128 name itself, separated by a dot (.). For instance, to set the record type
129 for group <literal>public</literal> to <literal>grs.sgml</literal>
130 (the SGML-like format for structured records) you would write:
135 public.recordType: grs.sgml
140 To set the default value of the record type to <literal>text</literal>
151 The available configuration settings are summarized below. They will be
152 explained further in the following sections.
156 FIXME - Didn't Adam make something to have multiple databases in multiple dirs...
164 <emphasis>group</emphasis>
165 .recordType[<emphasis>.name</emphasis>]:
166 <replaceable>type</replaceable>
170 Specifies how records with the file extension
171 <emphasis>name</emphasis> should be handled by the indexer.
172 This option may also be specified as a command line option
173 (<literal>-t</literal>). Note that if you do not specify a
174 <emphasis>name</emphasis>, the setting applies to all files.
175 In general, the record type specifier consists of the elements (each
176 element separated by dot), <emphasis>fundamental-type</emphasis>,
177 <emphasis>file-read-type</emphasis> and arguments. Currently, two
178 fundamental types exist, <literal>text</literal> and
179 <literal>grs</literal>.
184 <term><emphasis>group</emphasis>.recordId:
185 <replaceable>record-id-spec</replaceable></term>
188 Specifies how the records are to be identified when updated. See
189 <xref linkend="locating-records"/>.
194 <term><emphasis>group</emphasis>.database:
195 <replaceable>database</replaceable></term>
198 Specifies the Z39.50 database name.
199 <!-- FIXME - now we can have multiple databases in one server. -H -->
204 <term><emphasis>group</emphasis>.storeKeys:
205 <replaceable>boolean</replaceable></term>
208 Specifies whether key information should be saved for a given
209 group of records. If you plan to update/delete this type of
210 records later this should be specified as 1; otherwise it
211 should be 0 (default), to save register space.
212 <!-- ### this is the first mention of "register" -->
213 See <xref linkend="file-ids"/>.
218 <term><emphasis>group</emphasis>.storeData:
219 <replaceable>boolean</replaceable></term>
222 Specifies whether the records should be stored internally
223 in the Zebra system files.
224 If you want to maintain the raw records yourself,
225 this option should be false (0).
226 If you want Zebra to take care of the records for you, it
232 <!-- ### probably a better place to define "register" -->
233 <term>register: <replaceable>register-location</replaceable></term>
236 Specifies the location of the various register files that Zebra uses
237 to represent your databases.
238 See <xref linkend="register-location"/>.
243 <term>shadow: <replaceable>register-location</replaceable></term>
246 Enables the <emphasis>safe update</emphasis> facility of Zebra, and
247 tells the system where to place the required, temporary files.
248 See <xref linkend="shadow-registers"/>.
253 <term>lockDir: <replaceable>directory</replaceable></term>
256 Directory in which various lock files are stored.
261 <term>keyTmpDir: <replaceable>directory</replaceable></term>
264 Directory in which temporary files used during zebraidx's update
270 <term>setTmpDir: <replaceable>directory</replaceable></term>
273 Specifies the directory that the server uses for temporary result sets.
274 If not specified <literal>/tmp</literal> will be used.
279 <term>profilePath: <replaceable>path</replaceable></term>
282 Specifies a path of profile specification files.
283 The path is composed of one or more directories separated by
284 colon. Similar to PATH for UNIX systems.
289 <term>attset: <replaceable>filename</replaceable></term>
292 Specifies the filename(s) of attribute set files for use in
293 searching. At least the Bib-1 set should be loaded
294 (<literal>bib1.att</literal>).
295 The <literal>profilePath</literal> setting is used to look for
297 See <xref linkend="attset-files"/>
302 <term>memMax: <replaceable>size</replaceable></term>
305 Specifies <replaceable>size</replaceable> of internal memory
306 to use for the zebraidx program.
307 The amount is given in megabytes - default is 4 (4 MB).
308 The more memory, the faster large updates happen, up to about
309 half the free memory available on the computer.
314 <term>tempfiles: <replaceable>Yes/Auto/No</replaceable></term>
317 Tells zebra if it should use temporary files when indexing. The
318 default is Auto, in which case zebra uses temporary files only
319 if it would need more that <replaceable>memMax</replaceable>
320 megabytes of memory. This should be good for most uses.
326 <term>root: <replaceable>dir</replaceable></term>
329 Specifies a directory base for Zebra. All relative paths
330 given (in profilePath, register, shadow) are based on this
331 directory. This setting is useful if your Zebra server
332 is running in a different directory from where
333 <literal>zebra.cfg</literal> is located.
339 <term>passwd: <replaceable>file</replaceable></term>
342 Specifies a file with description of user accounts for Zebra.
343 The format is similar to that known to Apache's htpasswd files
344 and UNIX' passwd files. Non-empty lines not beginning with
345 # are considered account lines. There is one account per-line.
346 A line consists of fields separate by a single colon character.
347 First field is username, second is password.
353 <term>passwd.c: <replaceable>file</replaceable></term>
356 Specifies a file with description of user accounts for Zebra.
357 File format is similar to that used by the passwd directive except
358 that the password are encrypted. Use Apache's htpasswd or similar
365 <term>perm.<replaceable>user</replaceable>:
366 <replaceable>permstring</replaceable></term>
369 Specifies permissions (priviledge) for a user that are allowed
370 to access Zebra via the passwd system. There are two kinds
371 of permissions currently: read (r) and write(w). By default
372 users not listed in a permission directive are given the read
373 priviledge. To specify permissions for a user with no
374 username, or Z39.50 anonymous style use
375 <literal>anonymous</literal>. The permstring consists of
376 a sequence of characters. Include character <literal>w</literal>
377 for write/update access, <literal>r</literal> for read access.
387 <sect1 id="locating-records">
388 <title>Locating Records</title>
391 The default behavior of the Zebra system is to reference the
392 records from their original location, i.e. where they were found when you
393 ran <literal>zebraidx</literal>.
394 That is, when a client wishes to retrieve a record
395 following a search operation, the files are accessed from the place
396 where you originally put them - if you remove the files (without
397 running <literal>zebraidx</literal> again, the server will return
398 diagnostic number 14 (``System error in presenting records'') to
403 If your input files are not permanent - for example if you retrieve
404 your records from an outside source, or if they were temporarily
405 mounted on a CD-ROM drive,
406 you may want Zebra to make an internal copy of them. To do this,
407 you specify 1 (true) in the <literal>storeData</literal> setting. When
408 the Z39.50 server retrieves the records they will be read from the
409 internal file structures of the system.
414 <sect1 id="simple-indexing">
415 <title>Indexing with no Record IDs (Simple Indexing)</title>
418 If you have a set of records that are not expected to change over time
419 you may can build your database without record IDs.
420 This indexing method uses less space than the other methods and
425 To use this method, you simply omit the <literal>recordId</literal> entry
426 for the group of files that you index. To add a set of records you use
427 <literal>zebraidx</literal> with the <literal>update</literal> command. The
428 <literal>update</literal> command will always add all of the records that it
429 encounters to the index - whether they have already been indexed or
430 not. If the set of indexed files change, you should delete all of the
431 index files, and build a new index from scratch.
435 Consider a system in which you have a group of text files called
436 <literal>simple</literal>.
437 That group of records should belong to a Z39.50 database called
438 <literal>textbase</literal>.
439 The following <literal>zebra.cfg</literal> file will suffice:
444 profilePath: /usr/local/idzebra/tab
446 simple.recordType: text
447 simple.database: textbase
453 Since the existing records in an index can not be addressed by their
454 IDs, it is impossible to delete or modify records when using this method.
459 <sect1 id="file-ids">
460 <title>Indexing with File Record IDs</title>
463 If you have a set of files that regularly change over time: Old files
464 are deleted, new ones are added, or existing files are modified, you
465 can benefit from using the <emphasis>file ID</emphasis>
466 indexing methodology.
467 Examples of this type of database might include an index of WWW
468 resources, or a USENET news spool area.
469 Briefly speaking, the file key methodology uses the directory paths
470 of the individual records as a unique identifier for each record.
471 To perform indexing of a directory with file keys, again, you specify
472 the top-level directory after the <literal>update</literal> command.
473 The command will recursively traverse the directories and compare
474 each one with whatever have been indexed before in that same directory.
475 If a file is new (not in the previous version of the directory) it
476 is inserted into the registers; if a file was already indexed and
477 it has been modified since the last update, the index is also
478 modified; if a file has been removed since the last
479 visit, it is deleted from the index.
483 The resulting system is easy to administrate. To delete a record you
484 simply have to delete the corresponding file (say, with the
485 <literal>rm</literal> command). And to add records you create new
486 files (or directories with files). For your changes to take effect
487 in the register you must run <literal>zebraidx update</literal> with
488 the same directory root again. This mode of operation requires more
489 disk space than simpler indexing methods, but it makes it easier for
490 you to keep the index in sync with a frequently changing set of data.
491 If you combine this system with the <emphasis>safe update</emphasis>
492 facility (see below), you never have to take your server off-line for
493 maintenance or register updating purposes.
497 To enable indexing with pathname IDs, you must specify
498 <literal>file</literal> as the value of <literal>recordId</literal>
499 in the configuration file. In addition, you should set
500 <literal>storeKeys</literal> to <literal>1</literal>, since the Zebra
501 indexer must save additional information about the contents of each record
502 in order to modify the indexes correctly at a later time.
506 FIXME - There must be a simpler way to do this with Adams string tags -H
510 For example, to update records of group <literal>esdd</literal>
512 <literal>/data1/records/</literal> you should type:
514 $ zebraidx -g esdd update /data1/records
519 The corresponding configuration file includes:
522 esdd.recordType: grs.sgml
528 <para>You cannot start out with a group of records with simple
529 indexing (no record IDs as in the previous section) and then later
530 enable file record Ids. Zebra must know from the first time that you
532 the files should be indexed with file record IDs.
537 You cannot explicitly delete records when using this method (using the
538 <literal>delete</literal> command to <literal>zebraidx</literal>. Instead
539 you have to delete the files from the file system (or move them to a
541 and then run <literal>zebraidx</literal> with the
542 <literal>update</literal> command.
544 <!-- ### what happens if a file contains multiple records? -->
547 <sect1 id="generic-ids">
548 <title>Indexing with General Record IDs</title>
551 When using this method you construct an (almost) arbitrary, internal
552 record key based on the contents of the record itself and other system
553 information. If you have a group of records that explicitly associates
554 an ID with each record, this method is convenient. For example, the
555 record format may contain a title or a ID-number - unique within the group.
556 In either case you specify the Z39.50 attribute set and use-attribute
557 location in which this information is stored, and the system looks at
558 that field to determine the identity of the record.
562 As before, the record ID is defined by the <literal>recordId</literal>
563 setting in the configuration file. The value of the record ID specification
564 consists of one or more tokens separated by whitespace. The resulting
565 ID is represented in the index by concatenating the tokens and
566 separating them by ASCII value (1).
570 There are three kinds of tokens:
574 <term>Internal record info</term>
577 The token refers to a key that is
578 extracted from the record. The syntax of this token is
579 <literal>(</literal> <emphasis>set</emphasis> <literal>,</literal>
580 <emphasis>use</emphasis> <literal>)</literal>,
581 where <emphasis>set</emphasis> is the
582 attribute set name <emphasis>use</emphasis> is the
583 name or value of the attribute.
588 <term>System variable</term>
591 The system variables are preceded by
596 and immediately followed by the system variable name, which
609 <term>database</term>
612 Current database specified.
629 <term>Constant string</term>
632 A string used as part of the ID — surrounded
633 by single- or double quotes.
641 For instance, the sample GILS records that come with the Zebra
642 distribution contain a unique ID in the data tagged Control-Identifier.
643 The data is mapped to the Bib-1 use attribute Identifier-standard
644 (code 1007). To use this field as a record id, specify
645 <literal>(bib1,Identifier-standard)</literal> as the value of the
646 <literal>recordId</literal> in the configuration file.
647 If you have other record types that uses the same field for a
648 different purpose, you might add the record type
649 (or group or database name) to the record id of the gils
650 records as well, to prevent matches with other types of records.
651 In this case the recordId might be set like this:
654 gils.recordId: $type (bib1,Identifier-standard)
660 (see <xref linkend="data-model"/>
661 for details of how the mapping between elements of your records and
662 searchable attributes is established).
666 As for the file record ID case described in the previous section,
667 updating your system is simply a matter of running
668 <literal>zebraidx</literal>
669 with the <literal>update</literal> command. However, the update with general
670 keys is considerably slower than with file record IDs, since all files
671 visited must be (re)read to discover their IDs.
675 As you might expect, when using the general record IDs
676 method, you can only add or modify existing records with the
677 <literal>update</literal> command.
678 If you wish to delete records, you must use the,
679 <literal>delete</literal> command, with a directory as a parameter.
680 This will remove all records that match the files below that root
686 <sect1 id="register-location">
687 <title>Register Location</title>
690 Normally, the index files that form dictionaries, inverted
691 files, record info, etc., are stored in the directory where you run
692 <literal>zebraidx</literal>. If you wish to store these, possibly large,
693 files somewhere else, you must add the <literal>register</literal>
694 entry to the <literal>zebra.cfg</literal> file.
695 Furthermore, the Zebra system allows its file
696 structures to span multiple file systems, which is useful for
697 managing very large databases.
701 The value of the <literal>register</literal> setting is a sequence
702 of tokens. Each token takes the form:
705 <emphasis>dir</emphasis><literal>:</literal><emphasis>size</emphasis>.
708 The <emphasis>dir</emphasis> specifies a directory in which index files
709 will be stored and the <emphasis>size</emphasis> specifies the maximum
710 size of all files in that directory. The Zebra indexer system fills
711 each directory in the order specified and use the next specified
712 directories as needed.
713 The <emphasis>size</emphasis> is an integer followed by a qualifier
715 <literal>b</literal> for bytes,
716 <literal>k</literal> for kilobytes.
717 <literal>M</literal> for megabytes,
718 <literal>G</literal> for gigabytes.
722 For instance, if you have allocated two disks for your register, and
723 the first disk is mounted
724 on <literal>/d1</literal> and has 2GB of free space and the
725 second, mounted on <literal>/d2</literal> has 3.6 GB, you could
726 put this entry in your configuration file:
729 register: /d1:2G /d2:3600M
735 Note that Zebra does not verify that the amount of space specified is
736 actually available on the directory (file system) specified - it is
737 your responsibility to ensure that enough space is available, and that
738 other applications do not attempt to use the free space. In a large
739 production system, it is recommended that you allocate one or more
740 file system exclusively to the Zebra register files.
745 <sect1 id="shadow-registers">
746 <title>Safe Updating - Using Shadow Registers</title>
749 <title>Description</title>
752 The Zebra server supports <emphasis>updating</emphasis> of the index
753 structures. That is, you can add, modify, or remove records from
754 databases managed by Zebra without rebuilding the entire index.
755 Since this process involves modifying structured files with various
756 references between blocks of data in the files, the update process
757 is inherently sensitive to system crashes, or to process interruptions:
758 Anything but a successfully completed update process will leave the
759 register files in an unknown state, and you will essentially have no
760 recourse but to re-index everything, or to restore the register files
761 from a backup medium.
762 Further, while the update process is active, users cannot be
763 allowed to access the system, as the contents of the register files
764 may change unpredictably.
768 You can solve these problems by enabling the shadow register system in
770 During the updating procedure, <literal>zebraidx</literal> will temporarily
771 write changes to the involved files in a set of "shadow
772 files", without modifying the files that are accessed by the
773 active server processes. If the update procedure is interrupted by a
774 system crash or a signal, you simply repeat the procedure - the
775 register files have not been changed or damaged, and the partially
776 written shadow files are automatically deleted before the new updating
781 At the end of the updating procedure (or in a separate operation, if
782 you so desire), the system enters a "commit mode". First,
783 any active server processes are forced to access those blocks that
784 have been changed from the shadow files rather than from the main
785 register files; the unmodified blocks are still accessed at their
786 normal location (the shadow files are not a complete copy of the
787 register files - they only contain those parts that have actually been
788 modified). If the commit process is interrupted at any point during the
789 commit process, the server processes will continue to access the
790 shadow files until you can repeat the commit procedure and complete
791 the writing of data to the main register files. You can perform
792 multiple update operations to the registers before you commit the
793 changes to the system files, or you can execute the commit operation
794 at the end of each update operation. When the commit phase has
795 completed successfully, any running server processes are instructed to
796 switch their operations to the new, operational register, and the
797 temporary shadow files are deleted.
803 <title>How to Use Shadow Register Files</title>
806 The first step is to allocate space on your system for the shadow
808 You do this by adding a <literal>shadow</literal> entry to the
809 <literal>zebra.cfg</literal> file.
810 The syntax of the <literal>shadow</literal> entry is exactly the
811 same as for the <literal>register</literal> entry
812 (see <xref linkend="register-location"/>).
813 The location of the shadow area should be
814 <emphasis>different</emphasis> from the location of the main register
815 area (if you have specified one - remember that if you provide no
816 <literal>register</literal> setting, the default register area is the
817 working directory of the server and indexing processes).
821 The following excerpt from a <literal>zebra.cfg</literal> file shows
822 one example of a setup that configures both the main register
823 location and the shadow file area.
824 Note that two directories or partitions have been set aside
825 for the shadow file area. You can specify any number of directories
826 for each of the file areas, but remember that there should be no
827 overlaps between the directories used for the main registers and the
828 shadow files, respectively.
835 shadow: /scratch1:100M /scratch2:200M
841 When shadow files are enabled, an extra command is available at the
842 <literal>zebraidx</literal> command line.
843 In order to make changes to the system take effect for the
844 users, you'll have to submit a "commit" command after a
845 (sequence of) update operation(s).
851 $ zebraidx update /d1/records
858 Or you can execute multiple updates before committing the changes:
864 $ zebraidx -g books update /d1/records /d2/more-records
865 $ zebraidx -g fun update /d3/fun-records
872 If one of the update operations above had been interrupted, the commit
873 operation on the last line would fail: <literal>zebraidx</literal>
874 will not let you commit changes that would destroy the running register.
875 You'll have to rerun all of the update operations since your last
876 commit operation, before you can commit the new changes.
880 Similarly, if the commit operation fails, <literal>zebraidx</literal>
881 will not let you start a new update operation before you have
882 successfully repeated the commit operation.
883 The server processes will keep accessing the shadow files rather
884 than the (possibly damaged) blocks of the main register files
885 until the commit operation has successfully completed.
889 You should be aware that update operations may take slightly longer
890 when the shadow register system is enabled, since more file access
891 operations are involved. Further, while the disk space required for
892 the shadow register data is modest for a small update operation, you
893 may prefer to disable the system if you are adding a very large number
894 of records to an already very large database (we use the terms
895 <emphasis>large</emphasis> and <emphasis>modest</emphasis>
896 very loosely here, since every application will have a
897 different perception of size).
898 To update the system without the use of the the shadow files,
899 simply run <literal>zebraidx</literal> with the <literal>-n</literal>
900 option (note that you do not have to execute the
901 <emphasis>commit</emphasis> command of <literal>zebraidx</literal>
902 when you temporarily disable the use of the shadow registers in
904 Note also that, just as when the shadow registers are not enabled,
905 server processes will be barred from accessing the main register
906 while the update procedure takes place.
914 <!-- Keep this comment at the end of the file
919 sgml-minimize-attributes:nil
920 sgml-always-quote-attributes:t
923 sgml-parent-document: "zebra.xml"
924 sgml-local-catalogs: nil
925 sgml-namecase-general:t