1 <chapter id="administration">
2 <!-- $Id: administration.xml,v 1.45 2006-09-20 15:19:38 mike Exp $ -->
3 <title>Administrating Zebra</title>
4 <!-- ### It's a bit daft that this chapter (which describes half of
5 the configuration-file formats) is separated from
6 "recordmodel-grs.xml" (which describes the other half) by the
7 instructions on running zebraidx and zebrasrv. Some careful
8 re-ordering is required here.
12 Unlike many simpler retrieval systems, Zebra supports safe, incremental
13 updates to an existing index.
17 Normally, when Zebra modifies the index it reads a number of records
19 Depending on your specifications and on the contents of each record
20 one the following events take place for each record:
27 The record is indexed as if it never occurred before.
28 Either the Zebra system doesn't know how to identify the record or
29 Zebra can identify the record but didn't find it to be already indexed.
37 The record has already been indexed.
38 In this case either the contents of the record or the location
39 (file) of the record indicates that it has been indexed before.
47 The record is deleted from the index. As in the
48 update-case it must be able to identify the record.
56 Please note that in both the modify- and delete- case the Zebra
57 indexer must be able to generate a unique key that identifies the record
58 in question (more on this below).
62 To administrate the Zebra retrieval system, you run the
63 <literal>zebraidx</literal> program.
64 This program supports a number of options which are preceded by a dash,
65 and a few commands (not preceded by dash).
69 Both the Zebra administrative tool and the Z39.50 server share a
70 set of index files and a global configuration file.
71 The name of the configuration file defaults to
72 <literal>zebra.cfg</literal>.
73 The configuration file includes specifications on how to index
74 various kinds of records and where the other configuration files
75 are located. <literal>zebrasrv</literal> and <literal>zebraidx</literal>
76 <emphasis>must</emphasis> be run in the directory where the
77 configuration file lives unless you indicate the location of the
78 configuration file by option <literal>-c</literal>.
81 <sect1 id="record-types">
82 <title>Record Types</title>
85 Indexing is a per-record process, in which either insert/modify/delete
86 will occur. Before a record is indexed search keys are extracted from
87 whatever might be the layout the original record (sgml,html,text, etc..).
88 The Zebra system currently supports two fundamental types of records:
89 structured and simple text.
90 To specify a particular extraction process, use either the
91 command line option <literal>-t</literal> or specify a
92 <literal>recordType</literal> setting in the configuration file.
97 <sect1 id="zebra-cfg">
98 <title>The Zebra Configuration File</title>
101 The Zebra configuration file, read by <literal>zebraidx</literal> and
102 <literal>zebrasrv</literal> defaults to <literal>zebra.cfg</literal>
103 unless specified by <literal>-c</literal> option.
107 You can edit the configuration file with a normal text editor.
108 parameter names and values are separated by colons in the file. Lines
109 starting with a hash sign (<literal>#</literal>) are
114 If you manage different sets of records that share common
115 characteristics, you can organize the configuration settings for each
117 When <literal>zebraidx</literal> is run and you wish to address a
118 given group you specify the group name with the <literal>-g</literal>
120 In this case settings that have the group name as their prefix
121 will be used by <literal>zebraidx</literal>.
122 If no <literal>-g</literal> option is specified, the settings
123 without prefix are used.
127 In the configuration file, the group name is placed before the option
128 name itself, separated by a dot (.). For instance, to set the record type
129 for group <literal>public</literal> to <literal>grs.sgml</literal>
130 (the SGML-like format for structured records) you would write:
135 public.recordType: grs.sgml
140 To set the default value of the record type to <literal>text</literal>
151 The available configuration settings are summarized below. They will be
152 explained further in the following sections.
156 FIXME - Didn't Adam make something to have multiple databases in multiple dirs...
164 <emphasis>group</emphasis>
165 .recordType[<emphasis>.name</emphasis>]:
166 <replaceable>type</replaceable>
170 Specifies how records with the file extension
171 <emphasis>name</emphasis> should be handled by the indexer.
172 This option may also be specified as a command line option
173 (<literal>-t</literal>). Note that if you do not specify a
174 <emphasis>name</emphasis>, the setting applies to all files.
175 In general, the record type specifier consists of the elements (each
176 element separated by dot), <emphasis>fundamental-type</emphasis>,
177 <emphasis>file-read-type</emphasis> and arguments. Currently, two
178 fundamental types exist, <literal>text</literal> and
179 <literal>grs</literal>.
184 <term><emphasis>group</emphasis>.recordId:
185 <replaceable>record-id-spec</replaceable></term>
188 Specifies how the records are to be identified when updated. See
189 <xref linkend="locating-records"/>.
194 <term><emphasis>group</emphasis>.database:
195 <replaceable>database</replaceable></term>
198 Specifies the Z39.50 database name.
199 <!-- FIXME - now we can have multiple databases in one server. -H -->
204 <term><emphasis>group</emphasis>.storeKeys:
205 <replaceable>boolean</replaceable></term>
208 Specifies whether key information should be saved for a given
209 group of records. If you plan to update/delete this type of
210 records later this should be specified as 1; otherwise it
211 should be 0 (default), to save register space.
212 <!-- ### this is the first mention of "register" -->
213 See <xref linkend="file-ids"/>.
218 <term><emphasis>group</emphasis>.storeData:
219 <replaceable>boolean</replaceable></term>
222 Specifies whether the records should be stored internally
223 in the Zebra system files.
224 If you want to maintain the raw records yourself,
225 this option should be false (0).
226 If you want Zebra to take care of the records for you, it
232 <!-- ### probably a better place to define "register" -->
233 <term>register: <replaceable>register-location</replaceable></term>
236 Specifies the location of the various register files that Zebra uses
237 to represent your databases.
238 See <xref linkend="register-location"/>.
243 <term>shadow: <replaceable>register-location</replaceable></term>
246 Enables the <emphasis>safe update</emphasis> facility of Zebra, and
247 tells the system where to place the required, temporary files.
248 See <xref linkend="shadow-registers"/>.
253 <term>lockDir: <replaceable>directory</replaceable></term>
256 Directory in which various lock files are stored.
261 <term>keyTmpDir: <replaceable>directory</replaceable></term>
264 Directory in which temporary files used during zebraidx's update
270 <term>setTmpDir: <replaceable>directory</replaceable></term>
273 Specifies the directory that the server uses for temporary result sets.
274 If not specified <literal>/tmp</literal> will be used.
279 <term>profilePath: <replaceable>path</replaceable></term>
282 Specifies a path of profile specification files.
283 The path is composed of one or more directories separated by
284 colon. Similar to PATH for UNIX systems.
289 <term>attset: <replaceable>filename</replaceable></term>
292 Specifies the filename(s) of attribute set files for use in
293 searching. At least the Bib-1 set should be loaded
294 (<literal>bib1.att</literal>).
295 The <literal>profilePath</literal> setting is used to look for
297 See <xref linkend="attset-files"/>
302 <term>memMax: <replaceable>size</replaceable></term>
305 Specifies <replaceable>size</replaceable> of internal memory
306 to use for the zebraidx program.
307 The amount is given in megabytes - default is 4 (4 MB).
308 The more memory, the faster large updates happen, up to about
309 half the free memory available on the computer.
314 <term>tempfiles: <replaceable>Yes/Auto/No</replaceable></term>
317 Tells zebra if it should use temporary files when indexing. The
318 default is Auto, in which case zebra uses temporary files only
319 if it would need more that <replaceable>memMax</replaceable>
320 megabytes of memory. This should be good for most uses.
326 <term>root: <replaceable>dir</replaceable></term>
329 Specifies a directory base for Zebra. All relative paths
330 given (in profilePath, register, shadow) are based on this
331 directory. This setting is useful if your Zebra server
332 is running in a different directory from where
333 <literal>zebra.cfg</literal> is located.
339 <term>passwd: <replaceable>file</replaceable></term>
342 Specifies a file with description of user accounts for Zebra.
343 The format is similar to that known to Apache's htpasswd files
344 and UNIX' passwd files. Non-empty lines not beginning with
345 # are considered account lines. There is one account per-line.
346 A line consists of fields separate by a single colon character.
347 First field is username, second is password.
353 <term>passwd.c: <replaceable>file</replaceable></term>
356 Specifies a file with description of user accounts for Zebra.
357 File format is similar to that used by the passwd directive except
358 that the password are encrypted. Use Apache's htpasswd or similar
365 <term>perm.<replaceable>user</replaceable>:
366 <replaceable>permstring</replaceable></term>
369 Specifies permissions (priviledge) for a user that are allowed
370 to access Zebra via the passwd system. There are two kinds
371 of permissions currently: read (r) and write(w). By default
372 users not listed in a permission directive are given the read
373 privilege. To specify permissions for a user with no
374 username, or Z39.50 anonymous style use
375 <literal>anonymous</literal>. The permstring consists of
376 a sequence of characters. Include character <literal>w</literal>
377 for write/update access, <literal>r</literal> for read access and
378 <literal>a</literal> to allow anonymous access through this account.
384 <term>dbaccess <replaceable>accessfile</replaceable></term>
387 Names a file which lists database subscriptions for individual users.
388 The access file should consists of lines of the form <literal>username:
389 dbnames</literal>, where dbnames is a list of database names, seprated by
390 '+'. No whitespace is allowed in the database list.
400 <sect1 id="locating-records">
401 <title>Locating Records</title>
404 The default behavior of the Zebra system is to reference the
405 records from their original location, i.e. where they were found when you
406 run <literal>zebraidx</literal>.
407 That is, when a client wishes to retrieve a record
408 following a search operation, the files are accessed from the place
409 where you originally put them - if you remove the files (without
410 running <literal>zebraidx</literal> again, the server will return
411 diagnostic number 14 (``System error in presenting records'') to
416 If your input files are not permanent - for example if you retrieve
417 your records from an outside source, or if they were temporarily
418 mounted on a CD-ROM drive,
419 you may want Zebra to make an internal copy of them. To do this,
420 you specify 1 (true) in the <literal>storeData</literal> setting. When
421 the Z39.50 server retrieves the records they will be read from the
422 internal file structures of the system.
427 <sect1 id="simple-indexing">
428 <title>Indexing with no Record IDs (Simple Indexing)</title>
431 If you have a set of records that are not expected to change over time
432 you may can build your database without record IDs.
433 This indexing method uses less space than the other methods and
438 To use this method, you simply omit the <literal>recordId</literal> entry
439 for the group of files that you index. To add a set of records you use
440 <literal>zebraidx</literal> with the <literal>update</literal> command. The
441 <literal>update</literal> command will always add all of the records that it
442 encounters to the index - whether they have already been indexed or
443 not. If the set of indexed files change, you should delete all of the
444 index files, and build a new index from scratch.
448 Consider a system in which you have a group of text files called
449 <literal>simple</literal>.
450 That group of records should belong to a Z39.50 database called
451 <literal>textbase</literal>.
452 The following <literal>zebra.cfg</literal> file will suffice:
457 profilePath: /usr/local/idzebra/tab
459 simple.recordType: text
460 simple.database: textbase
466 Since the existing records in an index can not be addressed by their
467 IDs, it is impossible to delete or modify records when using this method.
472 <sect1 id="file-ids">
473 <title>Indexing with File Record IDs</title>
476 If you have a set of files that regularly change over time: Old files
477 are deleted, new ones are added, or existing files are modified, you
478 can benefit from using the <emphasis>file ID</emphasis>
479 indexing methodology.
480 Examples of this type of database might include an index of WWW
481 resources, or a USENET news spool area.
482 Briefly speaking, the file key methodology uses the directory paths
483 of the individual records as a unique identifier for each record.
484 To perform indexing of a directory with file keys, again, you specify
485 the top-level directory after the <literal>update</literal> command.
486 The command will recursively traverse the directories and compare
487 each one with whatever have been indexed before in that same directory.
488 If a file is new (not in the previous version of the directory) it
489 is inserted into the registers; if a file was already indexed and
490 it has been modified since the last update, the index is also
491 modified; if a file has been removed since the last
492 visit, it is deleted from the index.
496 The resulting system is easy to administrate. To delete a record you
497 simply have to delete the corresponding file (say, with the
498 <literal>rm</literal> command). And to add records you create new
499 files (or directories with files). For your changes to take effect
500 in the register you must run <literal>zebraidx update</literal> with
501 the same directory root again. This mode of operation requires more
502 disk space than simpler indexing methods, but it makes it easier for
503 you to keep the index in sync with a frequently changing set of data.
504 If you combine this system with the <emphasis>safe update</emphasis>
505 facility (see below), you never have to take your server off-line for
506 maintenance or register updating purposes.
510 To enable indexing with pathname IDs, you must specify
511 <literal>file</literal> as the value of <literal>recordId</literal>
512 in the configuration file. In addition, you should set
513 <literal>storeKeys</literal> to <literal>1</literal>, since the Zebra
514 indexer must save additional information about the contents of each record
515 in order to modify the indexes correctly at a later time.
519 FIXME - There must be a simpler way to do this with Adams string tags -H
523 For example, to update records of group <literal>esdd</literal>
525 <literal>/data1/records/</literal> you should type:
527 $ zebraidx -g esdd update /data1/records
532 The corresponding configuration file includes:
535 esdd.recordType: grs.sgml
541 <para>You cannot start out with a group of records with simple
542 indexing (no record IDs as in the previous section) and then later
543 enable file record Ids. Zebra must know from the first time that you
545 the files should be indexed with file record IDs.
550 You cannot explicitly delete records when using this method (using the
551 <literal>delete</literal> command to <literal>zebraidx</literal>. Instead
552 you have to delete the files from the file system (or move them to a
554 and then run <literal>zebraidx</literal> with the
555 <literal>update</literal> command.
557 <!-- ### what happens if a file contains multiple records? -->
560 <sect1 id="generic-ids">
561 <title>Indexing with General Record IDs</title>
564 When using this method you construct an (almost) arbitrary, internal
565 record key based on the contents of the record itself and other system
566 information. If you have a group of records that explicitly associates
567 an ID with each record, this method is convenient. For example, the
568 record format may contain a title or a ID-number - unique within the group.
569 In either case you specify the Z39.50 attribute set and use-attribute
570 location in which this information is stored, and the system looks at
571 that field to determine the identity of the record.
575 As before, the record ID is defined by the <literal>recordId</literal>
576 setting in the configuration file. The value of the record ID specification
577 consists of one or more tokens separated by whitespace. The resulting
578 ID is represented in the index by concatenating the tokens and
579 separating them by ASCII value (1).
583 There are three kinds of tokens:
587 <term>Internal record info</term>
590 The token refers to a key that is
591 extracted from the record. The syntax of this token is
592 <literal>(</literal> <emphasis>set</emphasis> <literal>,</literal>
593 <emphasis>use</emphasis> <literal>)</literal>,
594 where <emphasis>set</emphasis> is the
595 attribute set name <emphasis>use</emphasis> is the
596 name or value of the attribute.
601 <term>System variable</term>
604 The system variables are preceded by
609 and immediately followed by the system variable name, which
622 <term>database</term>
625 Current database specified.
642 <term>Constant string</term>
645 A string used as part of the ID — surrounded
646 by single- or double quotes.
654 For instance, the sample GILS records that come with the Zebra
655 distribution contain a unique ID in the data tagged Control-Identifier.
656 The data is mapped to the Bib-1 use attribute Identifier-standard
657 (code 1007). To use this field as a record id, specify
658 <literal>(bib1,Identifier-standard)</literal> as the value of the
659 <literal>recordId</literal> in the configuration file.
660 If you have other record types that uses the same field for a
661 different purpose, you might add the record type
662 (or group or database name) to the record id of the gils
663 records as well, to prevent matches with other types of records.
664 In this case the recordId might be set like this:
667 gils.recordId: $type (bib1,Identifier-standard)
673 (see <xref linkend="grs"/>
674 for details of how the mapping between elements of your records and
675 searchable attributes is established).
679 As for the file record ID case described in the previous section,
680 updating your system is simply a matter of running
681 <literal>zebraidx</literal>
682 with the <literal>update</literal> command. However, the update with general
683 keys is considerably slower than with file record IDs, since all files
684 visited must be (re)read to discover their IDs.
688 As you might expect, when using the general record IDs
689 method, you can only add or modify existing records with the
690 <literal>update</literal> command.
691 If you wish to delete records, you must use the,
692 <literal>delete</literal> command, with a directory as a parameter.
693 This will remove all records that match the files below that root
699 <sect1 id="register-location">
700 <title>Register Location</title>
703 Normally, the index files that form dictionaries, inverted
704 files, record info, etc., are stored in the directory where you run
705 <literal>zebraidx</literal>. If you wish to store these, possibly large,
706 files somewhere else, you must add the <literal>register</literal>
707 entry to the <literal>zebra.cfg</literal> file.
708 Furthermore, the Zebra system allows its file
709 structures to span multiple file systems, which is useful for
710 managing very large databases.
714 The value of the <literal>register</literal> setting is a sequence
715 of tokens. Each token takes the form:
718 <emphasis>dir</emphasis><literal>:</literal><emphasis>size</emphasis>.
721 The <emphasis>dir</emphasis> specifies a directory in which index files
722 will be stored and the <emphasis>size</emphasis> specifies the maximum
723 size of all files in that directory. The Zebra indexer system fills
724 each directory in the order specified and use the next specified
725 directories as needed.
726 The <emphasis>size</emphasis> is an integer followed by a qualifier
728 <literal>b</literal> for bytes,
729 <literal>k</literal> for kilobytes.
730 <literal>M</literal> for megabytes,
731 <literal>G</literal> for gigabytes.
735 For instance, if you have allocated two disks for your register, and
736 the first disk is mounted
737 on <literal>/d1</literal> and has 2GB of free space and the
738 second, mounted on <literal>/d2</literal> has 3.6 GB, you could
739 put this entry in your configuration file:
742 register: /d1:2G /d2:3600M
748 Note that Zebra does not verify that the amount of space specified is
749 actually available on the directory (file system) specified - it is
750 your responsibility to ensure that enough space is available, and that
751 other applications do not attempt to use the free space. In a large
752 production system, it is recommended that you allocate one or more
753 file system exclusively to the Zebra register files.
758 <sect1 id="shadow-registers">
759 <title>Safe Updating - Using Shadow Registers</title>
761 <sect2 id="shadow-registers-description">
762 <title>Description</title>
765 The Zebra server supports <emphasis>updating</emphasis> of the index
766 structures. That is, you can add, modify, or remove records from
767 databases managed by Zebra without rebuilding the entire index.
768 Since this process involves modifying structured files with various
769 references between blocks of data in the files, the update process
770 is inherently sensitive to system crashes, or to process interruptions:
771 Anything but a successfully completed update process will leave the
772 register files in an unknown state, and you will essentially have no
773 recourse but to re-index everything, or to restore the register files
774 from a backup medium.
775 Further, while the update process is active, users cannot be
776 allowed to access the system, as the contents of the register files
777 may change unpredictably.
781 You can solve these problems by enabling the shadow register system in
783 During the updating procedure, <literal>zebraidx</literal> will temporarily
784 write changes to the involved files in a set of "shadow
785 files", without modifying the files that are accessed by the
786 active server processes. If the update procedure is interrupted by a
787 system crash or a signal, you simply repeat the procedure - the
788 register files have not been changed or damaged, and the partially
789 written shadow files are automatically deleted before the new updating
794 At the end of the updating procedure (or in a separate operation, if
795 you so desire), the system enters a "commit mode". First,
796 any active server processes are forced to access those blocks that
797 have been changed from the shadow files rather than from the main
798 register files; the unmodified blocks are still accessed at their
799 normal location (the shadow files are not a complete copy of the
800 register files - they only contain those parts that have actually been
801 modified). If the commit process is interrupted at any point during the
802 commit process, the server processes will continue to access the
803 shadow files until you can repeat the commit procedure and complete
804 the writing of data to the main register files. You can perform
805 multiple update operations to the registers before you commit the
806 changes to the system files, or you can execute the commit operation
807 at the end of each update operation. When the commit phase has
808 completed successfully, any running server processes are instructed to
809 switch their operations to the new, operational register, and the
810 temporary shadow files are deleted.
815 <sect2 id="shadow-registers-how-to-use">
816 <title>How to Use Shadow Register Files</title>
819 The first step is to allocate space on your system for the shadow
821 You do this by adding a <literal>shadow</literal> entry to the
822 <literal>zebra.cfg</literal> file.
823 The syntax of the <literal>shadow</literal> entry is exactly the
824 same as for the <literal>register</literal> entry
825 (see <xref linkend="register-location"/>).
826 The location of the shadow area should be
827 <emphasis>different</emphasis> from the location of the main register
828 area (if you have specified one - remember that if you provide no
829 <literal>register</literal> setting, the default register area is the
830 working directory of the server and indexing processes).
834 The following excerpt from a <literal>zebra.cfg</literal> file shows
835 one example of a setup that configures both the main register
836 location and the shadow file area.
837 Note that two directories or partitions have been set aside
838 for the shadow file area. You can specify any number of directories
839 for each of the file areas, but remember that there should be no
840 overlaps between the directories used for the main registers and the
841 shadow files, respectively.
847 shadow: /scratch1:100M /scratch2:200M
853 When shadow files are enabled, an extra command is available at the
854 <literal>zebraidx</literal> command line.
855 In order to make changes to the system take effect for the
856 users, you'll have to submit a "commit" command after a
857 (sequence of) update operation(s).
863 $ zebraidx update /d1/records
870 Or you can execute multiple updates before committing the changes:
876 $ zebraidx -g books update /d1/records /d2/more-records
877 $ zebraidx -g fun update /d3/fun-records
884 If one of the update operations above had been interrupted, the commit
885 operation on the last line would fail: <literal>zebraidx</literal>
886 will not let you commit changes that would destroy the running register.
887 You'll have to rerun all of the update operations since your last
888 commit operation, before you can commit the new changes.
892 Similarly, if the commit operation fails, <literal>zebraidx</literal>
893 will not let you start a new update operation before you have
894 successfully repeated the commit operation.
895 The server processes will keep accessing the shadow files rather
896 than the (possibly damaged) blocks of the main register files
897 until the commit operation has successfully completed.
901 You should be aware that update operations may take slightly longer
902 when the shadow register system is enabled, since more file access
903 operations are involved. Further, while the disk space required for
904 the shadow register data is modest for a small update operation, you
905 may prefer to disable the system if you are adding a very large number
906 of records to an already very large database (we use the terms
907 <emphasis>large</emphasis> and <emphasis>modest</emphasis>
908 very loosely here, since every application will have a
909 different perception of size).
910 To update the system without the use of the the shadow files,
911 simply run <literal>zebraidx</literal> with the <literal>-n</literal>
912 option (note that you do not have to execute the
913 <emphasis>commit</emphasis> command of <literal>zebraidx</literal>
914 when you temporarily disable the use of the shadow registers in
916 Note also that, just as when the shadow registers are not enabled,
917 server processes will be barred from accessing the main register
918 while the update procedure takes place.
926 <sect1 id="administration-ranking">
927 <title>Relevance Ranking and Sorting of Result Sets</title>
929 <sect2 id="administration-overview">
930 <title>Overview</title>
932 The default ordering of a result set is left up to the server,
933 which inside Zebra means sorting in ascending document ID order.
934 This is not always the order humans want to browse the sometimes
935 quite large hit sets. Ranking and sorting comes to the rescue.
939 In cases where a good presentation ordering can be computed at
940 indexing time, we can use a fixed <literal>static ranking</literal>
941 scheme, which is provided for the <literal>alvis</literal>
942 indexing filter. This defines a fixed ordering of hit lists,
943 independently of the query issued.
947 There are cases, however, where relevance of hit set documents is
948 highly dependent on the query processed.
949 Simply put, <literal>dynamic relevance ranking</literal>
950 sorts a set of retrieved records such that those most likely to be
951 relevant to your request are retrieved first.
952 Internally, Zebra retrieves all documents that satisfy your
953 query, and re-orders the hit list to arrange them based on
954 a measurement of similarity between your query and the content of
959 Finally, there are situations where hit sets of documents should be
960 <literal>sorted</literal> during query time according to the
961 lexicographical ordering of certain sort indexes created at
967 <sect2 id="administration-ranking-static">
968 <title>Static Ranking</title>
971 Zebra uses internally inverted indexes to look up term occurencies
972 in documents. Multiple queries from different indexes can be
973 combined by the binary boolean operations <literal>AND</literal>,
974 <literal>OR</literal> and/or <literal>NOT</literal> (which
975 is in fact a binary <literal>AND NOT</literal> operation).
976 To ensure fast query execution
977 speed, all indexes have to be sorted in the same order.
980 The indexes are normally sorted according to document
981 <literal>ID</literal> in
982 ascending order, and any query which does not invoke a special
983 re-ranking function will therefore retrieve the result set in
985 <literal>ID</literal>
993 directive in the main core Zebra configuration file, the internal document
994 keys used for ordering are augmented by a preceding integer, which
995 contains the static rank of a given document, and the index lists
997 first by ascending static rank,
998 then by ascending document <literal>ID</literal>.
1000 is the ``best'' rank, as it occurs at the
1001 beginning of the list; higher numbers represent worse scores.
1004 The experimental <literal>alvis</literal> filter provides a
1005 directive to fetch static rank information out of the indexed XML
1006 records, thus making <emphasis>all</emphasis> hit sets ordered
1007 after <emphasis>ascending</emphasis> static
1008 rank, and for those doc's which have the same static rank, ordered
1009 after <emphasis>ascending</emphasis> doc <literal>ID</literal>.
1010 See <xref linkend="record-model-alvisxslt"/> for the gory details.
1015 <sect2 id="administration-ranking-dynamic">
1016 <title>Dynamic Ranking</title>
1018 In order to fiddle with the static rank order, it is necessary to
1019 invoke additional re-ranking/re-ordering using dynamic
1020 ranking or score functions. These functions return positive
1021 integer scores, where <emphasis>highest</emphasis> score is
1023 hit sets are sorted according to <emphasis>descending</emphasis>
1025 to the index lists which are sorted according to
1026 ascending rank number and document ID).
1029 Dynamic ranking is enabled by a directive like one of the
1030 following in the zebra configuration file (use only one of these a time!):
1032 rank: rank-1 # default TDF-IDF like
1033 rank: rank-static # dummy do-nothing
1038 Dynamic ranking is done at query time rather than
1039 indexing time (this is why we
1040 call it ``dynamic ranking'' in the first place ...)
1041 It is invoked by adding
1042 the Bib-1 relation attribute with
1043 value ``relevance'' to the PQF query (that is,
1044 <literal>@attr 2=102</literal>, see also
1045 <ulink url="&url.z39.50;bib1.html">
1046 The BIB-1 Attribute Set Semantics</ulink>, also in
1047 <ulink url="&url.z39.50.attset.bib1;">HTML</ulink>).
1048 To find all articles with the word <literal>Eoraptor</literal> in
1049 the title, and present them relevance ranked, issue the PQF query:
1051 @attr 2=102 @attr 1=4 Eoraptor
1055 <sect3 id="administration-ranking-dynamic-rank1">
1056 <title>Dynamically ranking using PQF queries with the 'rank-1'
1060 The default <literal>rank-1</literal> ranking module implements a
1061 TF/IDF (Term Frequecy over Inverse Document Frequency) like
1062 algorithm. In contrast to the usual defintion of TF/IDF
1063 algorithms, which only considers searching in one full-text
1064 index, this one works on multiple indexes at the same time.
1066 Zebra does boolean queries and searches in specific addressed
1067 indexes (there are inverted indexes pointing from terms in the
1068 dictionary to documents and term positions inside documents).
1072 <term>Query Components</term>
1075 First, the boolean query is dismantled into it's principal components,
1076 i.e. atomic queries where one term is looked up in one index.
1077 For example, the query
1079 @attr 2=102 @and @attr 1=1010 Utah @attr 1=1018 Springer
1081 is a boolean AND between the atomic parts
1083 @attr 2=102 @attr 1=1010 Utah
1087 @attr 2=102 @attr 1=1018 Springer
1089 which gets processed each for itself.
1095 <term>Atomic hit lists</term>
1098 Second, for each atomic query, the hit list of documents is
1102 In this example, two hit lists for each index
1103 <literal>@attr 1=1010</literal> and
1104 <literal>@attr 1=1018</literal> are computed.
1110 <term>Atomic scores</term>
1113 Third, each document in the hit list is assigned a score (_if_ ranking
1114 is enabled and requested in the query) using a TF/IDF scheme.
1117 In this example, both atomic parts of the query assign the magic
1118 <literal>@attr 2=102</literal> relevance attribute, and are
1119 to be used in the relevance ranking functions.
1122 It is possible to apply dynamic ranking on only parts of the
1125 @and @attr 2=102 @attr 1=1010 Utah @attr 1=1018 Springer
1127 searches for all documents which have the term 'Utah' on the
1128 body of text, and which have the term 'Springer' in the publisher
1129 field, and sort them in the order of the relevance ranking made on
1130 the body-of-text index only.
1136 <term>Hit list merging</term>
1139 Fourth, the atomic hit lists are merged according to the boolean
1140 conditions to a final hit list of documents to be returned.
1143 This step is always performed, independently of the fact that
1144 dynamic ranking is enabled or not.
1150 <term>Document score computation</term>
1153 Fifth, the total score of a document is computed as a linear
1154 combination of the atomic scores of the atomic hit lists
1157 Ranking weights may be used to pass a value to a ranking
1158 algorithm, using the non-standard BIB-1 attribute type 9.
1159 This allows one branch of a query to use one value while
1160 another branch uses a different one. For example, we can search
1161 for <literal>utah</literal> in the
1162 <literal>@attr 1=4</literal> index with weight 30, as
1163 well as in the <literal>@attr 1=1010</literal> index with weight 20:
1165 @attr 2=102 @or @attr 9=30 @attr 1=4 utah @attr 9=20 @attr 1=1010 city
1169 The default weight is
1170 sqrt(1000) ~ 34 , as the Z39.50 standard prescribes that the top score
1171 is 1000 and the bottom score is 0, encoded in integers.
1175 The ranking-weight feature is experimental. It may change in future
1183 <term>Re-sorting of hit list</term>
1186 Finally, the final hit list is re-ordered according to scores.
1194 Still need to describe the exact TF/IDF formula. Here's the info, need -->
1195 <!--to extract it in human readable form .. MC
1197 static int calc (void *set_handle, zint sysno, zint staticrank,
1200 int i, lo, divisor, score = 0;
1201 struct rank_set_info *si = (struct rank_set_info *) set_handle;
1203 if (!si->no_rank_entries)
1204 return -1; /* ranking not enabled for any terms */
1206 for (i = 0; i < si->no_entries; i++)
1208 yaz_log(log_level, "calc: i=%d rank_flag=%d lo=%d",
1209 i, si->entries[i].rank_flag, si->entries[i].local_occur);
1210 if (si->entries[i].rank_flag && (lo = si->entries[i].local_occur))
1211 score += (8+log2_int (lo)) * si->entries[i].global_inv *
1212 si->entries[i].rank_weight;
1214 divisor = si->no_rank_entries * (8+log2_int (si->last_pos/si->no_entries));
1215 score = score / divisor;
1216 yaz_log(log_level, "calc sysno=" ZINT_FORMAT " score=%d", sysno, score);
1219 /* reset the counts for the next term */
1220 for (i = 0; i < si->no_entries; i++)
1221 si->entries[i].local_occur = 0;
1226 where lo = si->entries[i].local_occur is the local documents term-within-index frequency, si->entries[i].global_inv represents the IDF part (computed in static void *begin()), and
1227 si->entries[i].rank_weight is the weight assigner per index (default 34, or set in the @attr 9=xyz magic)
1229 Finally, the IDF part is computed as:
1231 static void *begin (struct zebra_register *reg,
1232 void *class_handle, RSET rset, NMEM nmem,
1233 TERMID *terms, int numterms)
1235 struct rank_set_info *si =
1236 (struct rank_set_info *) nmem_malloc (nmem,sizeof(*si));
1239 yaz_log(log_level, "rank-1 begin");
1240 si->no_entries = numterms;
1241 si->no_rank_entries = 0;
1243 si->entries = (struct rank_term_info *)
1244 nmem_malloc (si->nmem, sizeof(*si->entries)*numterms);
1245 for (i = 0; i < numterms; i++)
1247 zint g = rset_count(terms[i]->rset);
1248 yaz_log(log_level, "i=%d flags=%s '%s'", i,
1249 terms[i]->flags, terms[i]->name );
1250 if (!strncmp (terms[i]->flags, "rank,", 5))
1252 const char *cp = strstr(terms[i]->flags+4, ",w=");
1253 si->entries[i].rank_flag = 1;
1255 si->entries[i].rank_weight = atoi (cp+3);
1257 si->entries[i].rank_weight = 34; /* sqrroot of 1000 */
1258 yaz_log(log_level, " i=%d weight=%d g="ZINT_FORMAT, i,
1259 si->entries[i].rank_weight, g);
1260 (si->no_rank_entries)++;
1263 si->entries[i].rank_flag = 0;
1264 si->entries[i].local_occur = 0; /* FIXME */
1265 si->entries[i].global_occur = g;
1266 si->entries[i].global_inv = 32 - log2_int (g);
1267 yaz_log(log_level, " global_inv = %d g = " ZINT_FORMAT,
1268 (int) (32-log2_int (g)), g);
1269 si->entries[i].term = terms[i];
1270 si->entries[i].term_index=i;
1271 terms[i]->rankpriv = &(si->entries[i]);
1277 where g = rset_count(terms[i]->rset) is the count of all documents in this specific index hit list, and the IDF part then is
1279 si->entries[i].global_inv = 32 - log2_int (g);
1286 The <literal>rank-1</literal> algorithm
1287 does not use the static rank
1288 information in the list keys, and will produce the same ordering
1289 with or without static ranking enabled.
1294 <sect3 id="administration-ranking-dynamic-rank1">
1295 <title>Dynamically ranking PQF queries with the 'rank-static'
1298 The dummy <literal>rank-static</literal> reranking/scoring
1299 function returns just
1300 <literal>score = max int - staticrank</literal>
1301 in order to preserve the static ordering of hit sets that would
1302 have been produced had it not been invoked.
1303 Obviously, to combine static and dynamic ranking usefully,
1305 to make a new ranking
1306 function; this is left
1307 as an exercise for the reader.
1314 <literal>Dynamic ranking</literal> is not compatible
1315 with <literal>estimated hit sizes</literal>, as all documents in
1316 a hit set must be accessed to compute the correct placing in a
1317 ranking sorted list. Therefore the use attribute setting
1318 <literal>@attr 2=102</literal> clashes with
1319 <literal>@attr 9=integer</literal>.
1324 we might want to add ranking like this:
1326 Simple BM25 Extension to Multiple Weighted Fields
1327 Stephen Robertson, Hugo Zaragoza and Michael Taylor
1331 mitaylor2microsoft.com
1336 <sect3 id="administration-ranking-dynamic-cql">
1337 <title>Dynamically ranking CQL queries</title>
1339 Dynamic ranking can be enabled during sever side CQL
1340 query expansion by adding <literal>@attr 2=102</literal>
1341 chunks to the CQL config file. For example
1343 relationModifier.relevant = 2=102
1345 invokes dynamic ranking each time a CQL query of the form
1348 Z> f alvis.text =/relevant house
1350 is issued. Dynamic ranking can also be automatically used on
1351 specific CQL indexes by (for example) setting
1353 index.alvis.text = 1=text 2=102
1355 which then invokes dynamic ranking each time a CQL query of the form
1358 Z> f alvis.text = house
1368 <sect2 id="administration-ranking-sorting">
1369 <title>Sorting</title>
1371 Zebra sorts efficiently using special sorting indexes
1372 (type=<literal>s</literal>; so each sortable index must be known
1373 at indexing time, specified in the configuration of record
1374 indexing. For example, to enable sorting according to the BIB-1
1375 <literal>Date/time-added-to-db</literal> field, one could add the line
1377 xelm /*/@created Date/time-added-to-db:s
1379 to any <literal>.abs</literal> record-indexing configuration file.
1380 Similarly, one could add an indexing element of the form
1382 <z:index name="date-modified" type="s">
1383 <xsl:value-of select="some/xpath"/>
1386 to any <literal>alvis</literal>-filter indexing stylesheet.
1389 Indexing can be specified at searching time using a query term
1390 carrying the non-standard
1391 BIB-1 attribute-type <literal>7</literal>. This removes the
1392 need to send a Z39.50 <literal>Sort Request</literal>
1393 separately, and can dramatically improve latency when the client
1394 and server are on separate networks.
1395 The sorting part of the query is separate from the rest of the
1396 query - the actual search specification - and must be combined
1400 A sorting subquery needs two attributes: an index (such as a
1401 BIB-1 type-1 attribute) specifying which index to sort on, and a
1402 type-7 attribute whose value is be <literal>1</literal> for
1403 ascending sorting, or <literal>2</literal> for descending. The
1404 term associated with the sorting attribute is the priority of
1405 the sort key, where <literal>0</literal> specifies the primary
1406 sort key, <literal>1</literal> the secondary sort key, and so
1409 <para>For example, a search for water, sort by title (ascending),
1410 is expressed by the PQF query
1412 @or @attr 1=1016 water @attr 7=1 @attr 1=4 0
1414 whereas a search for water, sort by title ascending,
1415 then date descending would be
1417 @or @or @attr 1=1016 water @attr 7=1 @attr 1=4 0 @attr 7=2 @attr 1=30 1
1421 Notice the fundamental differences between <literal>dynamic
1422 ranking</literal> and <literal>sorting</literal>: there can be
1423 only one ranking function defined and configured; but multiple
1424 sorting indexes can be specified dynamically at search
1425 time. Ranking does not need to use specific indexes, so
1426 dynamic ranking can be enabled and disabled without
1427 re-indexing; whereas, sorting indexes need to be
1428 defined before indexing.
1436 <sect1 id="administration-extended-services">
1437 <title>Extended Services: Remote Insert, Update and Delete</title>
1441 Extended services are only supported when accessing the Zebra
1442 server using the <ulink url="&url.z39.50;">Z39.50</ulink>
1443 protocol. The <ulink url="&url.sru;">SRU</ulink> protocol does
1444 not support extended services.
1449 The extended services are not enabled by default in zebra - due to the
1450 fact that they modify the system. Zebra can be configured
1452 search, and to allow only updates for a particular admin user
1453 in the main zebra configuration file <filename>zebra.cfg</filename>.
1454 For user <literal>admin</literal>, you could use:
1458 passwd: passwordfile
1460 And in the password file
1461 <filename>passwordfile</filename>, you have to specify users and
1462 encrypted passwords as colon separated strings.
1463 Use a tool like <filename>htpasswd</filename>
1464 to maintain the encrypted passwords.
1468 It is essential to configure Zebra to store records internally,
1470 modifications and deletion of records:
1475 The general record type should be set to any record filter which
1476 is able to parse XML records, you may use any of the two
1477 declarations (but not both simultaneously!)
1480 # recordType: alvis.filter_alvis_config.xml
1482 To enable transaction safe shadow indexing,
1483 which is extra important for this kind of operation, set
1485 shadow: directoryname: size (e.g. 1000M)
1490 It is not possible to carry information about record types or
1491 similar to Zebra when using extended services, due to
1492 limitations of the <ulink url="&url.z39.50;">Z39.50</ulink>
1493 protocol. Therefore, indexing filters can not be chosen on a
1494 per-record basis. One and only one general XML indexing filter
1496 <!-- but because it is represented as an OID, we would need some
1497 form of proprietary mapping scheme between record type strings and
1500 However, as a minimum, it would be extremely useful to enable
1501 people to use MARC21, assuming grs.marcxml.marc21 as a record
1508 <sect2 id="administration-extended-services-z3950">
1509 <title>Extended services in the Z39.50 protocol</title>
1512 The <ulink url="&url.z39.50;">Z39.50</ulink> standard allows
1513 servers to accept special binary <emphasis>extended services</emphasis>
1514 protocol packages, which may be used to insert, update and delete
1515 records into servers. These carry control and update
1516 information to the servers, which are encoded in seven package fields:
1519 <table id="administration-extended-services-z3950-table" frame="top">
1520 <title>Extended services Z39.50 Package Fields</title>
1524 <entry>Parameter</entry>
1525 <entry>Value</entry>
1526 <entry>Notes</entry>
1531 <entry><literal>type</literal></entry>
1532 <entry><literal>'update'</literal></entry>
1533 <entry>Must be set to trigger extended services</entry>
1536 <entry><literal>action</literal></entry>
1537 <entry><literal>string</literal></entry>
1539 Extended service action type with
1540 one of four possible values: <literal>recordInsert</literal>,
1541 <literal>recordReplace</literal>,
1542 <literal>recordDelete</literal>,
1543 and <literal>specialUpdate</literal>
1547 <entry><literal>record</literal></entry>
1548 <entry><literal>XML string</literal></entry>
1549 <entry>An XML formatted string containing the record</entry>
1552 <entry><literal>syntax</literal></entry>
1553 <entry><literal>'xml'</literal></entry>
1554 <entry>Only XML record syntax is supported</entry>
1557 <entry><literal>recordIdOpaque</literal></entry>
1558 <entry><literal>string</literal></entry>
1560 Optional client-supplied, opaque record
1561 identifier used under insert operations.
1565 <entry><literal>recordIdNumber </literal></entry>
1566 <entry><literal>positive number</literal></entry>
1567 <entry>Zebra's internal system number, only for update
1572 <entry><literal>databaseName</literal></entry>
1573 <entry><literal>database identifier</literal></entry>
1575 The name of the database to which the extended services should be
1585 The <literal>action</literal> parameter can be any of
1586 <literal>recordInsert</literal> (will fail if the record already exists),
1587 <literal>recordReplace</literal> (will fail if the record does not exist),
1588 <literal>recordDelete</literal> (will fail if the record does not
1590 <literal>specialUpdate</literal> (will insert or update the record
1595 During a <literal>recordInsert</literal> action, the
1596 usual rules for internal record ID generation apply, unless an
1597 optional <literal>recordIdNumber</literal> Zebra internal ID or a
1598 <literal>recordIdOpaque</literal> string identifier is assigned.
1599 The default ID generation is
1600 configured using the <literal>recordId:</literal> from
1601 <filename>zebra.cfg</filename>.
1605 The actions <literal>recordReplace</literal> or
1606 <literal>recordDelete</literal> need specification of the additional
1607 <literal>recordIdNumber</literal> parameter, which must be an
1608 existing Zebra internal system ID number, or the optional
1609 <literal>recordIdOpaque</literal> string parameter.
1613 When retrieving existing
1614 records indexed with GRS indexing filters, the Zebra internal
1615 ID number is returned in the field
1616 <literal>/*/id:idzebra/localnumber</literal> in the namespace
1617 <literal>xmlns:id="http://www.indexdata.dk/zebra/"</literal>,
1618 where it can be picked up for later record updates or deletes.
1621 Records indexed with the <literal>alvis</literal> filter
1622 have similar means to discover the internal Zebra ID.
1626 The <literal>recordIdOpaque</literal> string parameter
1627 is an client-supplied, opaque record
1628 identifier, which may be used under
1629 insert, update and delete operations. The
1630 client software is responsible for assigning these to
1631 records. This identifier will
1632 replace zebra's own automagic identifier generation with a unique
1633 mapping from <literal>recordIdOpaque</literal> to the
1634 Zebra internal <literal>recordIdNumber</literal>.
1635 <emphasis>The opaque <literal>recordIdOpaque</literal> string
1637 are not visible in retrieval records, nor are
1638 searchable, so the value of this parameter is
1639 questionable. It serves mostly as a convenient mapping from
1640 application domain string identifiers to Zebra internal ID's.
1646 <sect2 id="administration-extended-services-yaz-client">
1647 <title>Extended services from yaz-client</title>
1650 We can now start a yaz-client admin session and create a database:
1653 $ yaz-client localhost:9999 -u admin/secret
1657 Now the <literal>Default</literal> database was created,
1658 we can insert an XML file (esdd0006.grs
1659 from example/gils/records) and index it:
1662 Z> update insert id1234 esdd0006.grs
1665 The 3rd parameter - <literal>id1234</literal> here -
1666 is the <literal>recordIdOpaque</literal> package field.
1669 Actually, we should have a way to specify "no opaque record id" for
1670 yaz-client's update command.. We'll fix that.
1673 The newly inserted record can be searched as usual:
1678 Received SearchResponse.
1679 Search was a success.
1680 Number of hits: 1, setno 1
1681 SearchResult-1: term=utah cnt=1
1688 Let's delete the beast, using the same
1689 <literal>recordIdOpaque</literal> string parameter:
1692 Z> update delete id1234
1693 No last record (update ignored)
1694 Z> update delete 1 esdd0006.grs
1695 Got extended services response
1700 Received SearchResponse.
1701 Search was a success.
1702 Number of hits: 0, setno 2
1703 SearchResult-1: term=utah cnt=0
1710 If shadow register is enabled in your
1711 <filename>zebra.cfg</filename>,
1712 you must run the adm-commit command
1718 after each update session in order write your changes from the
1719 shadow to the life register space.
1724 <sect2 id="administration-extended-services-yaz-php">
1725 <title>Extended services from yaz-php</title>
1728 Extended services are also available from the YAZ PHP client layer. An
1729 example of an YAZ-PHP extended service transaction is given here:
1732 $record = '<record><title>A fine specimen of a record</title></record>';
1734 $options = array('action' => 'recordInsert',
1736 'record' => $record,
1737 'databaseName' => 'mydatabase'
1740 yaz_es($yaz, 'update', $options);
1741 yaz_es($yaz, 'commit', array());
1744 if ($error = yaz_error($yaz))
1754 <!-- Keep this comment at the end of the file
1759 sgml-minimize-attributes:nil
1760 sgml-always-quote-attributes:t
1763 sgml-parent-document: "zebra.xml"
1764 sgml-local-catalogs: nil
1765 sgml-namecase-general:t