1 <chapter id="querymodel">
2 <!-- $Id: querymodel.xml,v 1.22 2006-07-03 11:45:01 marc Exp $ -->
3 <title>Query Model</title>
5 <sect1 id="querymodel-overview">
6 <title>Query Model Overview</title>
8 <sect2 id="querymodel-query-languages">
9 <title>Query Languages</title>
12 Zebra is born as a networking Information Retrieval engine adhering
13 to the international standards
14 <ulink url="&url.z39.50;">Z39.50</ulink> and
15 <ulink url="&url.sru;">SRU</ulink>,
17 <literal>type-1 Reverse Polish Notation (RPN)</literal> query
19 Unfortunately, this model has only defined a binary
20 encoded representation, which is used as transport packaging in
21 the Z39.50 protocol layer. This representation is not human
22 readable, nor defines any convenient way to specify queries.
25 Since the <literal>type-1 (RPN)</literal>
26 query structure has no direct, useful string
27 representation, every client application needs to provide some
28 form of mapping from a local query notation or representation to it.
32 <sect3 id="querymodel-query-languages-pqf">
33 <title>Prefix Query Format (PQF)</title>
35 Index Data has defined a textual representation in the
36 <ulink url="&url.yaz.pqf;">Prefix Query Format</ulink>, short
37 <emphasis>PQF</emphasis>, which maps
38 one-to-one to binary encoded
39 <emphasis>type-1 RPN</emphasis> queries.
40 PQF has been adopted by other
41 parties developing Z39.50 software, and is often referred to as
42 <literal>Prefix Query Notation</literal>, or in short
43 <literal>PQN</literal>. See
44 <xref linkend="querymodel-pqf"/> for further explanations and
45 descriptions of Zebra's capabilities.
49 <sect3 id="querymodel-query-languages-cql">
50 <title>Common Query Language (CQL)</title>
52 The query model of the type-1 RPN,
53 expressed in PQF/PQN is natively supported.
54 On the other hand, the default SRU
55 web services <emphasis>Common Query Language</emphasis>
56 <ulink url="&url.cql;">CQL</ulink> is not natively supported.
59 Zebra can be configured to understand and map CQL to PQF. See
60 <xref linkend="querymodel-cql-to-pqf"/>.
66 <sect2 id="querymodel-operation-types">
67 <title>Operation types</title>
69 Zebra supports all of the three different
70 <literal>Z39.50/SRU</literal> operations defined in the
71 standards: <literal>explain</literal>, <literal>search</literal>,
72 and <literal>scan</literal>. A short description of the
73 functionality and purpose of each is quite in order here.
76 <sect3 id="querymodel-operation-type-explain">
77 <title>Explain Operation</title>
79 The <emphasis>syntax</emphasis> of Z39.50/SRU queries is
80 well known to any client, but the specific
81 <emphasis>semantics</emphasis> - taking into account a
82 particular servers functionalities and abilities - must be
83 discovered from case to case. Enters the
84 <literal>explain</literal> operation, which provides the means
86 <emphasis>fields</emphasis> (also called
87 <emphasis>indexes</emphasis> or <emphasis>access points</emphasis>)
88 are provided, which default parameter the server uses, which
89 retrieve document formats are defined, and which specific parts
90 of the general query model are supported.
93 The Z39.50 embeds the <literal>explain</literal> operation
95 <literal>search</literal> in the magic
96 <literal>IR-Explain-1</literal> database;
97 see <xref linkend="querymodel-exp1"/>.
100 In SRU, <literal>explain</literal> is an entirely separate
101 operation, which returns an <literal>ZeeRex
102 XML</literal> record according to the
103 structure defined by the protocol.
106 In both cases, the information gathered through
107 <literal>explain</literal> operations can be used to
108 auto-configure a client user interface to the servers
113 <sect3 id="querymodel-operation-type-search">
114 <title>Search Operation</title>
116 Search and retrieve interactions are the raison d'ĂȘtre.
117 They are used to query the remote database and
118 return search result documents. Search queries span from
119 simple free text searches to nested complex boolean queries,
120 targeting specific indexes, and possibly enhanced with many
121 query semantic specifications. Search interactions are the heart
122 and soul of Z39.50/SRU servers.
126 <sect3 id="querymodel-operation-type-scan">
127 <title>Scan Operation</title>
129 The <literal>scan</literal> operation is a helper functionality,
130 which operates on one index or access point a time.
134 the means to investigate the content of specific indexes.
135 Scanning an index returns a handful of terms actually found in
136 the indexes, and in addition the <literal>scan</literal>
137 operation returns the number of documents indexed by each term.
138 A search client can use this information to propose proper
139 spelling of search terms, to auto-fill search boxes, or to
140 display controlled vocabularies.
149 <sect1 id="querymodel-pqf">
150 <title>Prefix Query Format syntax and semantics</title>
152 The <ulink url="&url.yaz.pqf;">PQF grammar</ulink>
153 is documented in the YAZ manual, and shall not be
154 repeated here. This textual PQF representation
155 is not transmistted to Zebra during search, but it is in the
156 client mapped to the equivalent Z39.50 binary
160 <sect2 id="querymodel-pqf-tree">
161 <title>PQF tree structure</title>
163 The PQF parse tree - or the equivalent textual representation -
164 may start with one specification of the
165 <emphasis>attribute set</emphasis> used. Following is a query
167 consists of <emphasis>atomic query parts (APT)</emphasis> or
168 <emphasis>named result sets</emphasis>, eventually
169 paired by <emphasis>boolean binary operators</emphasis>, and
170 finally <emphasis>recursively combined </emphasis> into
174 <sect3 id="querymodel-attribute-sets">
175 <title>Attribute sets</title>
177 Attribute sets define the exact meaning and semantics of queries
178 issued. Zebra comes with some predefined attribute set
179 definitions, others can easily be defined and added to the
184 <table id="querymodel-attribute-sets-table"
185 frame="all" rowsep="1" colsep="1" align="center">
187 <caption>Attribute sets predefined in Zebra</caption>
191 <td>Attribute set</td>
200 <td><literal>Explain</literal></td>
201 <td><literal>exp-1</literal></td>
202 <td>Special attribute set used on the special automagic
203 <literal>IR-Explain-1</literal> database to gain information on
204 server capabilities, database names, and database
209 <td><literal>Bib1</literal></td>
210 <td><literal>bib-1</literal></td>
211 <td>Standard PQF query language attribute set which defines the
212 semantics of Z39.50 searching. In addition, all of the
213 non-use attributes (type 2-9) define the hard-wired
219 <td><literal>GILS</literal></td>
220 <td><literal>gils</literal></td>
221 <td>Extension to the <literal>Bib1</literal> attribute set.</td>
226 <td><literal>IDXPATH</literal></td>
227 <td><literal>idxpath</literal></td>
228 <td>Hardwired XPATH like attribute set, only available for
229 indexing with the GRS record model</td>
238 The <literal>use attributes (type 1)</literal> mappings the
239 predefined attribute sets are found in the
240 attribute set configuration files <filename>tab/*.att</filename>.
244 The Zebra internal query processing is modeled after
245 the <literal>Bib1</literal> attribute set, and the non-use
246 attributes type 2-6 are hard-wired in. It is therefore essential
247 to be familiar with <xref linkend="querymodel-bib1-nonuse"/>.
251 <sect3 id="querymodel-boolean-operators">
252 <title>Boolean operators</title>
254 A pair of sub query trees, or of atomic queries, is combined
255 using the standard boolean operators into new query trees.
256 Thus, boolean operators are always internal nodes in the query tree.
259 <table id="querymodel-boolean-operators-table"
260 frame="all" rowsep="1" colsep="1" align="center">
262 <caption>Boolean operators</caption>
271 <tr><td><literal>@and</literal></td>
272 <td>binary <literal>AND</literal> operator</td>
273 <td>Set intersection of two atomic queries hit sets</td>
275 <tr><td><literal>@or</literal></td>
276 <td>binary <literal>OR</literal> operator</td>
277 <td>Set union of two atomic queries hit sets</td>
279 <tr><td><literal>@not</literal></td>
280 <td>binary <literal>AND NOT</literal> operator</td>
281 <td>Set complement of two atomic queries hit sets</td>
283 <tr><td><literal>@prox</literal></td>
284 <td>binary <literal>PROXIMITY</literal> operator</td>
285 <td>Set intersection of two atomic queries hit sets. In
286 addition, the intersection set is purged for all
287 documents which do not satisfy the requested query
288 term proximity. Usually a proper subset of the AND
295 For example, we can combine the terms
296 <emphasis>information</emphasis> and <emphasis>retrieval</emphasis>
297 into different searches in the default index of the default
298 attribute set as follows.
299 Querying for the union of all documents containing the
300 terms <emphasis>information</emphasis> OR
301 <emphasis>retrieval</emphasis>:
303 Z> find @or information retrieval
307 Querying for the intersection of all documents containing the
308 terms <emphasis>information</emphasis> AND
309 <emphasis>retrieval</emphasis>:
310 The hit set is a subset of the corresponding
313 Z> find @and information retrieval
317 Querying for the intersection of all documents containing the
318 terms <emphasis>information</emphasis> AND
319 <emphasis>retrieval</emphasis>, taking proximity into account:
320 The hit set is a subset of the corresponding
322 (see the <ulink url="&url.yaz.pqf;">PQF grammar</ulink> for
323 details on the proximity operator):
325 Z> find @prox 0 3 0 2 k 2 information retrieval
329 Querying for the intersection of all documents containing the
330 terms <emphasis>information</emphasis> AND
331 <emphasis>retrieval</emphasis>, in the same order and near each
332 other as described in the term list.
333 The hit set is a subset of the corresponding
336 Z> find "information retrieval"
342 <sect3 id="querymodel-atomic-queries">
343 <title>Atomic queries (APT)</title>
345 Atomic queries are the query parts which work on one access point
346 only. These consist of <literal>an attribute list</literal>
347 followed by a <literal>single term</literal> or a
348 <literal>quoted term list</literal>, and are often called
349 <emphasis>Attributes-Plus-Terms (APT)</emphasis> queries.
352 Atomic (APT) queries are always leaf nodes in the PQF query tree.
353 UN-supplied non-use attributes type 2-9 are either inherited from
354 higher nodes in the query tree, or are set to Zebra's default values.
355 See <xref linkend="querymodel-bib1"/> for details.
358 <table id="querymodel-atomic-queries-table"
359 frame="all" rowsep="1" colsep="1" align="center">
361 <caption>Atomic queries (APT)</caption>
371 <td><emphasis>attribute list</emphasis></td>
372 <td>List of <literal>orthogonal</literal> attributes</td>
373 <td>Any of the orthogonal attribute types may be omitted,
374 these are inherited from higher query tree nodes, or if not
375 inherited, are set to the default Zebra configuration values.
379 <td><emphasis>term</emphasis></td>
380 <td>single <literal>term</literal>
381 or <literal>quoted term list</literal> </td>
382 <td>Here the search terms or list of search terms is added
388 Querying for the term <emphasis>information</emphasis> in the
389 default index using the default attribute set, the server choice
390 of access point/index, and the default non-use attributes.
396 Equivalent query fully specified including all default values:
398 Z> find @attrset bib-1 @attr 1=1017 @attr 2=3 @attr 3=3 @attr 4=1 @attr 5=100 @attr 6=1 information
403 Finding all documents which have the term
404 <emphasis>debussy</emphasis> in the title field.
406 Z> find @attr 1=4 debussy
411 The <literal>scan</literal> operation is only supported with
412 atomic APT queries, as it is bound to one access point at a
413 time. Boolean query trees are not allowed during
414 <literal>scan</literal>.
418 For example, we might want to scan the title index, starting with
420 <emphasis>debussy</emphasis>, and displaying this and the
421 following terms in lexicographic order:
423 Z> scan @attr 1=4 debussy
429 <sect3 id="querymodel-resultset">
430 <title>Named Result Sets</title>
432 Named result sets are supported in Zebra, and result sets can be
433 used as operands without limitations. It follows that named
434 result sets are leaf nodes in the PQF query tree, exactly as
435 atomic APT queries are.
438 After the execution of a search, the result set is available at
439 the server, such that the client can use it for subsequent
440 searches or retrieval requests. The Z30.50 standard actually
441 stresses the fact that result sets are volatile. It may cease
442 to exist at any time point after search, and the server will
443 send a diagnostic to the effect that the requested
444 result set does not exist any more.
448 Defining a named result set and re-using it in the next query,
449 using <literal>yaz-client</literal>. Notice that the client, not
450 the server, assigns the string <literal>'1'</literal> to the
453 Z> f @attr 1=4 mozart
455 Number of hits: 43, setno 1
457 Z> f @and @set 1 @attr 1=4 amadeus
459 Number of hits: 14, setno 2
464 Named result sets are only supported by the Z39.50 protocol.
465 The SRU web service is stateless, and therefore the notion of
466 named result sets does not exist when accessing a Zebra server by
472 <sect3 id="querymodel-use-string">
473 <title>Zebra's special access point of type 'string'</title>
475 The numeric <literal>use (type 1)</literal> attribute is usually
476 referred to from a given
477 attribute set. In addition, Zebra let you use
478 <emphasis>any internal index
479 name defined in your configuration</emphasis>
480 as use attribute value. This is a great feature for
481 debugging, and when you do
482 not need the complexity of defined use attribute values. It is
483 the preferred way of accessing Zebra indexes directly.
486 Finding all documents which have the term list "information
487 retrieval" in an Zebra index, using it's internal full string
488 name. Scanning the same index.
490 Z> find @attr 1=sometext "information retrieval"
491 Z> scan @attr 1=sometext aterm
495 Searching or scanning
496 the bib-1 use attribute 54 using it's string name:
498 Z> find @attr 1=Code-language eng
499 Z> scan @attr 1=Code-language ""
503 It is possible to search
504 in any silly string index - if it's defined in your
505 indexation rules and can be parsed by the PQF parser.
506 This is definitely not the recommended use of
507 this facility, as it might confuse your users with some very
510 Z> find @attr 1=silly/xpath/alike[@index]/name "information retrieval"
514 See also <xref linkend="querymodel-pqf-apt-mapping"/> for details, and
515 <xref linkend="server-sru"/>
516 for the SRU PQF query extension using string names as a fast
521 <sect3 id="querymodel-use-xpath">
522 <title>Zebra's special access point of type 'XPath'
523 for GRS filters</title>
525 As we have seen above, it is possible (albeit seldom a great
527 <ulink url="http://www.w3.org/TR/xpath">XPath 1.0</ulink> based
528 search by defining <literal>use (type 1)</literal>
529 <emphasis>string</emphasis> attributes which in appearance
530 <emphasis>resemble XPath queries</emphasis>. There are two
531 problems with this approach: first, the XPath-look-alike has to
532 be defined at indexation time, no new undefined
533 XPath queries can entered at search time, and second, it might
534 confuse users very much that an XPath-alike index name in fact
535 gets populated from a possible entirely different XML element
536 than it pretends to access.
539 When using the <literal>GRS Record Model</literal>
540 (see <xref linkend="record-model-grs"/>), we have the
541 possibility to embed <emphasis>life</emphasis>
543 in the PQF queries, which are here called
544 <literal>use (type 1)</literal> <emphasis>xpath</emphasis>
545 attributes. You must enable the
546 <literal>xpath enable</literal> directive in your
547 <literal>.abs</literal> configuration files.
550 Only a <emphasis>very</emphasis> restricted subset of the
551 <ulink url="http://www.w3.org/TR/xpath">XPath 1.0</ulink>
552 standard is supported as the GRS record model is simpler than
553 a full XML DOM structure. See the following examples for
557 Finding all documents which have the term "content"
558 inside a text node found in a specific XML DOM
559 <emphasis>subtree</emphasis>, whose starting element is
562 Z> find @attr 1=/root content
563 Z> find @attr 1=/root/first content
565 <emphasis>Notice that the
566 XPath must be absolute, i.e., must start with '/', and that the
567 XPath <literal>descendant-or-self</literal> axis followed by a
568 text node selection <literal>text()</literal> is implicitly
569 appended to the stated XPath.
571 It follows that the above searches are interpreted as:
573 Z> find @attr 1=/root//text() content
574 Z> find @attr 1=/root/first//text() content
579 Searching inside attribute strings is possible:
581 Z> find @attr 1=/link/@creator morten
586 Filter the addressing XPath by a predicate working on exact
588 attributes (in the XML sense) can be done: return all those docs which
589 have the term "english" contained in one of all text sub nodes of
590 the subtree defined by the XPath
591 <literal>/record/title[@lang='en']</literal>. And similar
594 Z> find @attr 1=/record/title[@lang='en'] english
595 Z> find @attr 1=/link[@creator='sisse'] sibelius
596 Z> find @attr 1=/link[@creator='sisse']/description[@xml:lang='da'] sibelius
601 Combining numeric indexes, boolean expressions,
602 and xpath based searches is possible:
604 Z> find @attr 1=/record/title @and foo bar
605 Z> find @and @attr 1=/record/title foo @attr 1=4 bar
609 Escaping PQF keywords and other non-parseable XPath constructs
610 with <literal>'{ }'</literal> to prevent client-side PQF parsing
613 Z> find @attr {1=/root/first[@attr='danish']} content
614 Z> find @attr {1=/record/@set} oai
618 It is worth mentioning that these dynamic performed XPath
619 queries are a performance bottleneck, as no optimized
620 specialized indexes can be used. Therefore, avoid the use of
621 this facility when speed is essential, and the database content
622 size is medium to large.
629 <sect2 id="querymodel-exp1">
630 <title>Explain Attribute Set</title>
632 The Z39.50 standard defines the
633 <ulink url="&url.z39.50.explain;">Explain</ulink> attribute set
634 <literal>Exp-1</literal>, which is used to discover information
635 about a server's search semantics and functional capabilities
636 Zebra exposes a "classic"
637 Explain database by base name <literal>IR-Explain-1</literal>, which
638 is populated with system internal information.
641 The attribute-set <literal>exp-1</literal> consists of a single
642 <literal>use attribute (type 1)</literal>.
645 In addition, the non-Use
646 <literal>bib-1</literal> attributes, that is, the types
647 <literal>Relation</literal>, <literal>Position</literal>,
648 <literal>Structure</literal>, <literal>Truncation</literal>,
649 and <literal>Completeness</literal> are imported from
650 the <literal>bib-1</literal> attribute set, and may be used
651 within any explain query.
654 <sect3 id="querymodel-exp1-use">
655 <title>Use Attributes (type = 1)</title>
657 The following Explain search attributes are supported:
658 <literal>ExplainCategory</literal> (@attr 1=1),
659 <literal>DatabaseName</literal> (@attr 1=3),
660 <literal>DateAdded</literal> (@attr 1=9),
661 <literal>DateChanged</literal>(@attr 1=10).
664 A search in the use attribute <literal>ExplainCategory</literal>
665 supports only these predefined values:
666 <literal>CategoryList</literal>, <literal>TargetInfo</literal>,
667 <literal>DatabaseInfo</literal>, <literal>AttributeDetails</literal>.
670 See <filename>tab/explain.att</filename> and the
671 <ulink url="&url.z39.50;">Z39.50</ulink> standard
672 for more information.
677 <title>Explain searches with yaz-client</title>
679 Classic Explain only defines retrieval of Explain information
680 via ASN.1. Practically no Z39.50 clients supports this. Fortunately
681 they don't have to - Zebra allows retrieval of this information
683 <literal>SUTRS</literal>, <literal>XML</literal>,
684 <literal>GRS-1</literal> and <literal>ASN.1</literal> Explain.
688 List supported categories to find out which explain commands are
692 Z> find @attr exp1 1=1 categorylist
699 Get target info, that is, investigate which databases exist at
700 this server endpoint:
703 Z> find @attr exp1 1=1 targetinfo
714 List all supported databases, the number of hits
715 is the number of databases found, which most commonly are the
717 the <literal>Default</literal> and the
718 <literal>IR-Explain-1</literal> databases.
721 Z> find @attr exp1 1=1 databaseinfo
728 Get database info record for database <literal>Default</literal>.
731 Z> find @and @attr exp1 1=1 databaseinfo @attr exp1 1=3 Default
733 Identical query with explicitly specified attribute set:
736 Z> find @attrset exp1 @and @attr 1=1 databaseinfo @attr 1=3 Default
741 Get attribute details record for database
742 <literal>Default</literal>.
743 This query is very useful to study the internal Zebra indexes.
744 If records have been indexed using the <literal>alvis</literal>
745 XSLT filter, the string representation names of the known indexes can be
749 Z> find @and @attr exp1 1=1 attributedetails @attr exp1 1=3 Default
751 Identical query with explicitly specified attribute set:
754 Z> find @attrset exp1 @and @attr 1=1 attributedetails @attr 1=3 Default
761 <sect2 id="querymodel-bib1">
762 <title>Bib1 Attribute Set</title>
764 Most of the information contained in this section is an excerpt of
765 the <literal>ATTRIBUTE SET BIB-1 (Z39.50-1995)
767 found at <ulink url="&url.z39.50.attset.bib1.1995;">. The BIB-1
768 Attribute Set Semantics</ulink> from 1995, also in an updated
769 <ulink url="&url.z39.50.attset.bib1;">Bib-1
770 Attribute Set</ulink>
771 version from 2003. Index Data is not the copyright holder of this
772 information, except for the configuration details, the listing of
773 Zebra's capabilities, and the example queries.
777 <sect3 id="querymodel-bib1-use">
778 <title>Use Attributes (type 1)</title>
781 A use attribute specifies an access point for any atomic query.
782 These access points are highly dependent on the attribute set used
783 in the query, and are user configurable using the following
784 default configuration files:
785 <filename>tab/bib1.att</filename>,
786 <filename>tab/dan1.att</filename>,
787 <filename>tab/explain.att</filename>, and
788 <filename>tab/gils.att</filename>.
791 For example, some few <literal>Bib-1</literal> use
792 attributes from the <filename>tab/bib1.att</filename> are:
796 att 3 Conference-name
799 att 1009 Subject-name-personal
800 att 1010 Body-of-text
801 att 1011 Date/time-added-to-db
804 att 1017 Server-choice
808 att 1036 Author-Title-Subject
812 New attribute sets can be added by adding new
813 <filename>tab/*.att</filename> configuration files, which need to
814 be sourced in the main configuration <filename>zebra.cfg</filename>.
817 In addition, Zebra allows the access of
818 <emphasis>internal index names</emphasis> and <emphasis>dynamic
819 XPath</emphasis> as use attributes; see
820 <xref linkend="querymodel-use-string"/> and
821 <xref linkend="querymodel-use-xpath"/>.
825 Phrase search for <emphasis>information retrieval</emphasis> in
826 the title-register, scanning the same register afterwards:
828 Z> find @attr 1=4 "information retrieval"
829 Z> scan @attr 1=4 information
837 <sect2 id="querymodel-bib1-nonuse">
838 <title>Zebra general Bib1 Non-Use Attributes (type 2-6)</title>
840 <sect3 id="querymodel-bib1-relation">
841 <title>Relation Attributes (type 2)</title>
844 Relation attributes describe the relationship of the access
846 of the relation) to the search term as qualified by the attributes (right
847 side of the relation), e.g., Date-publication <= 1975.
850 <table id="querymodel-bib1-relation-table"
851 frame="all" rowsep="1" colsep="1" align="center">
853 <caption>Relation Attributes (type 2)</caption>
868 <td>Less than or equal</td>
878 <td>Greater or equal</td>
883 <td>Greater than</td>
908 <td>AlwaysMatches</td>
916 The relation attributes
917 <literal>1-5</literal> are supported and work exactly as
919 All ordering operations are based on a lexicographical ordering,
920 <emphasis>expect</emphasis> when the
921 <literal>structure attribute numeric (109)</literal> is used. In
922 this case, ordering is numerical. See
923 <xref linkend="querymodel-bib1-structure"/>.
925 Z> find @attr 1=Title @attr 2=1 music
927 Number of hits: 11745, setno 1
929 Z> find @attr 1=Title @attr 2=2 music
931 Number of hits: 11771, setno 2
933 Z> find @attr 1=Title @attr 2=3 music
935 Number of hits: 532, setno 3
937 Z> find @attr 1=Title @attr 2=4 music
939 Number of hits: 11463, setno 4
941 Z> find @attr 1=Title @attr 2=5 music
943 Number of hits: 11419, setno 5
948 The relation attribute
949 <literal>Relevance (102)</literal> is supported, see
950 <xref linkend="administration-ranking"/> for full information.
954 Ranked search for <emphasis>information retrieval</emphasis> in
957 Z> find @attr 1=4 @attr 2=102 "information retrieval"
962 The relation attribute
963 <literal>AlwaysMatches (103)</literal> is in the default
965 supported in conjecture with structure attribute
966 <literal>Phrase (1)</literal> (which may be omitted by
968 It can be configured to work with other structure attributes,
969 see the configuration file
970 <filename>tab/default.idx</filename> and
971 <xref linkend="querymodel-pqf-apt-mapping"/>.
974 <literal>AlwaysMatches (103)</literal> is a
975 great way to discover how many documents have been indexed in a
976 given field. The search term is ignored, but needed for correct
977 PQF syntax. An empty search term may be supplied.
979 Z> find @attr 1=Title @attr 2=103 ""
980 Z> find @attr 1=Title @attr 2=103 @attr 4=1 ""
987 <sect3 id="querymodel-bib1-position">
988 <title>Position Attributes (type 3)</title>
991 The position attribute specifies the location of the search term
992 within the field or subfield in which it appears.
995 <table id="querymodel-bib1-position-table"
996 frame="all" rowsep="1" colsep="1" align="center">
998 <caption>Position Attributes (type 3)</caption>
1008 <td>First in field </td>
1010 <td>unsupported</td>
1013 <td>First in subfield</td>
1015 <td>unsupported</td>
1018 <td>Any position in field</td>
1026 The position attribute values <literal>first in field (1)</literal>,
1027 and <literal>first in subfield(2)</literal> are unsupported.
1028 Using them silently maps to
1029 <literal>any position in field (3)</literal>. A proper diagnostic
1030 should have been issued.
1034 <sect3 id="querymodel-bib1-structure">
1035 <title>Structure Attributes (type 4)</title>
1038 The structure attribute specifies the type of search
1039 term. This causes the search to be mapped on
1040 different Zebra internal indexes, which must have been defined
1045 The possible values of the
1046 <literal>structure attribute (type 4)</literal> can be defined
1047 using the configuration file <filename>
1048 tab/default.idx</filename>.
1049 The default configuration is summarized in this table.
1052 <table id="querymodel-bib1-structure-table"
1053 frame="all" rowsep="1" colsep="1" align="center">
1055 <caption>Structure Attributes (type 4)</caption>
1085 <td>Date (normalized)</td>
1095 <td>Date (un-normalized)</td>
1097 <td>unsupported</td>
1100 <td>Name (normalized) </td>
1102 <td>unsupported</td>
1105 <td>Name (un-normalized) </td>
1107 <td>unsupported</td>
1112 <td>unsupported</td>
1120 <td>Free-form-text</td>
1125 <td>Document-text</td>
1130 <td>Local-number</td>
1137 <td>unsupported</td>
1140 <td>Numeric string</td>
1149 The structure attribute values
1150 <literal>Word list (6)</literal>
1151 is supported, and maps to the boolean <literal>AND</literal>
1152 combination of words supplied. The word list is useful when
1153 google-like bag-of-word queries need to be translated from a GUI
1154 query language to PQF. For example, the following queries
1157 Z> find @attr 1=Title @attr 4=6 "mozart amadeus"
1158 Z> find @attr 1=Title @and mozart amadeus
1163 The structure attribute value
1164 <literal>Free-form-text (105)</literal> and
1165 <literal>Document-text (106)</literal>
1166 are supported, and map both to the boolean <literal>OR</literal>
1167 combination of words supplied. The following queries
1170 Z> find @attr 1=Body-of-text @attr 4=105 "bach salieri teleman"
1171 Z> find @attr 1=Body-of-text @attr 4=106 "bach salieri teleman"
1172 Z> find @attr 1=Body-of-text @or bach @or salieri teleman
1174 This <literal>OR</literal> list of terms is very useful in
1175 combination with relevance ranking:
1177 Z> find @attr 1=Body-of-text @attr 2=102 @attr 4=105 "bach salieri teleman"
1182 The structure attribute value
1183 <literal>Local number (107)</literal>
1184 is supported, and maps always to the Zebra internal document ID,
1185 irrespectively which use attribute is specified. The following queries
1186 have exactly the same unique record in the hit set:
1188 Z> find @attr 4=107 10
1189 Z> find @attr 1=4 @attr 4=107 10
1190 Z> find @attr 1=1010 @attr 4=107 10
1196 the GILS schema (<literal>gils.abs</literal>), the
1197 west-bounding-coordinate is indexed as type <literal>n</literal>,
1198 and is therefore searched by specifying
1199 <emphasis>structure</emphasis>=<emphasis>Numeric String</emphasis>.
1200 To match all those records with west-bounding-coordinate greater
1201 than -114 we use the following query:
1203 Z> find @attr 4=109 @attr 2=5 @attr gils 1=2038 -114
1207 The exact mapping between PQF queries and Zebra internal indexes
1208 and index types is explained in
1209 <xref linkend="querymodel-pqf-apt-mapping"/>.
1214 <sect3 id="querymodel-bib1-truncation">
1215 <title>Truncation Attributes (type = 5)</title>
1218 The truncation attribute specifies whether variations of one or
1219 more characters are allowed between search term and hit terms, or
1220 not. Using non-default truncation attributes will broaden the
1221 document hit set of a search query.
1224 <table id="querymodel-bib1-truncation-table"
1225 frame="all" rowsep="1" colsep="1" align="center">
1227 <caption>Truncation Attributes (type 5)</caption>
1237 <td>Right truncation </td>
1242 <td>Left truncation</td>
1247 <td>Left and right truncation</td>
1252 <td>Do not truncate</td>
1257 <td>Process # in search term</td>
1275 The truncation attribute values 1-3 perform the obvious way:
1277 Z> scan @attr 1=Body-of-text schnittke
1283 Z> find @attr 1=Body-of-text @attr 5=1 schnittke
1285 Number of hits: 95, setno 7
1287 Z> find @attr 1=Body-of-text @attr 5=2 schnittke
1289 Number of hits: 81, setno 6
1291 Z> find @attr 1=Body-of-text @attr 5=3 schnittke
1293 Number of hits: 95, setno 8
1298 The truncation attribute value
1299 <literal>Process # in search term (101)</literal> is a
1300 poor-man's regular expression search. It maps
1301 each <literal>#</literal> to <literal>.*</literal>, and
1302 performs then a <literal>Regexp-1 (102)</literal> regular
1303 expression search. The following two queries are equivalent:
1305 Z> find @attr 1=Body-of-text @attr 5=101 schnit#ke
1306 Z> find @attr 1=Body-of-text @attr 5=102 schnit.*ke
1308 Number of hits: 89, setno 10
1313 The truncation attribute value
1314 <literal>Regexp-1 (102)</literal> is a normal regular search,
1315 see <xref linkend="querymodel-regular"/> for details.
1317 Z> find @attr 1=Body-of-text @attr 5=102 schnit+ke
1318 Z> find @attr 1=Body-of-text @attr 5=102 schni[a-t]+ke
1323 The truncation attribute value
1324 <literal>Regexp-2 (103) </literal> is a Zebra specific extension
1325 which allows <emphasis>fuzzy</emphasis> matches. One single
1326 error in spelling of search terms is allowed, i.e., a document
1327 is hit if it includes a term which can be mapped to the used
1328 search term by one character substitution, addition, deletion or
1331 Z> find @attr 1=Body-of-text @attr 5=100 schnittke
1333 Number of hits: 81, setno 14
1335 Z> find @attr 1=Body-of-text @attr 5=103 schnittke
1337 Number of hits: 103, setno 15
1343 <sect3 id="querymodel-bib1-completeness">
1344 <title>Completeness Attributes (type = 6)</title>
1348 The <literal>Completeness Attributes (type = 6)</literal>
1349 is used to specify that a given search term or term list is either
1350 part of the terms of a given index/field
1351 (<literal>Incomplete subfield (1)</literal>), or is
1352 what literally is found in the entire field's index
1353 (<literal>Complete field (3)</literal>).
1356 <table id="querymodel-bib1-completeness-table"
1357 frame="all" rowsep="1" colsep="1" align="center">
1358 <caption>Completeness Attributes (type = 6)</caption>
1361 <td>Completeness</td>
1368 <td>Incomplete subfield</td>
1373 <td>Complete subfield</td>
1378 <td>Complete field</td>
1386 The <literal>Completeness Attributes (type = 6)</literal>
1387 is only partially and conditionally
1388 supported in the sense that it is ignored if the hit index is
1389 not of structure <literal>type="w"</literal> or
1390 <literal>type="p"</literal>.
1393 <literal>Incomplete subfield (1)</literal> is the default, and
1395 register <literal>type="w"</literal>, whereas
1396 <literal>Complete field (3)</literal> triggers
1397 search and scan in index <literal>type="p"</literal>.
1400 The <literal>Complete subfield (2)</literal> is a reminiscens
1401 from the happy <literal>MARC</literal>
1402 binary format days. Zebra does not support it, but maps silently
1403 to <literal>Complete field (3)</literal>.
1407 The exact mapping between PQF queries and Zebra internal indexes
1408 and index types is explained in
1409 <xref linkend="querymodel-pqf-apt-mapping"/>.
1417 <sect1 id="querymodel-zebra">
1418 <title>Advanced Zebra PQF Features</title>
1420 The Zebra internal query engine has been extended to specific needs
1421 not covered by the <literal>bib-1</literal> attribute set query
1422 model. These extensions are <emphasis>non-standard</emphasis>
1423 and <emphasis>non-portable</emphasis>: most functional extensions
1424 are modeled over the <literal>bib-1</literal> attribute set,
1425 defining type 7-9 attributes.
1426 There are also the special
1427 <literal>string</literal> type index names for the
1428 <literal>idxpath</literal> attribute set.
1431 <sect2 id="querymodel-zebra-attr-allrecords">
1432 <title>Zebra specific retrieval of all records</title>
1434 Zebra defines a hardwired <literal>string</literal> index name
1435 called <literal>_ALLRECORDS</literal>. It matches any record
1436 contained in the database, if used in conjunction with
1437 the relation attribute
1438 <literal>AlwaysMatches (103)</literal>.
1441 The <literal>_ALLRECORDS</literal> index name is used for total database
1442 export. The search term is ignored, it may be empty.
1444 Z> find @attr 1=_ALLRECORDS @attr 2=103 ""
1448 Combination with other index types can be made. For example, to
1449 find all records which are <emphasis>not</emphasis> indexed in
1450 the <literal>Title</literal> register, issue one of the two
1453 Z> find @not @attr 1=_ALLRECORDS @attr 2=103 "" @attr 1=Title @attr 2=103 ""
1454 Z> find @not @attr 1=_ALLRECORDS @attr 2=103 "" @attr 1=4 @attr 2=103 ""
1458 The special string index <literal>_ALLRECORDS</literal> is
1459 experimental, and the provided functionality and syntax may very
1460 well change in future releases of Zebra.
1465 <sect2 id="querymodel-zebra-attr-search">
1466 <title>Zebra specific Search Extensions to all Attribute Sets</title>
1468 Zebra extends the Bib1 attribute types, and these extensions are
1469 recognized regardless of attribute
1470 set used in a <literal>search</literal> operation query.
1473 <table id="querymodel-zebra-attr-search-table"
1474 frame="all" rowsep="1" colsep="1" align="center">
1476 <caption>Zebra Search Attribute Extensions</caption>
1482 <td>Zebra version</td>
1487 <td>Embedded Sort</td>
1499 <td>Rank Weight</td>
1505 <td>Approx Limit</td>
1511 <td>Term Reference</td>
1519 <sect3 id="querymodel-zebra-attr-sorting">
1520 <title>Zebra Extension Embedded Sort Attribute (type 7)</title>
1523 The embedded sort is a way to specify sort within a query - thus
1524 removing the need to send a Sort Request separately. It is both
1525 faster and does not require clients to deal with the Sort
1530 All ordering operations are based on a lexicographical ordering,
1531 <emphasis>expect</emphasis> when the
1532 <literal>structure attribute numeric (109)</literal> is used. In
1533 this case, ordering is numerical. See
1534 <xref linkend="querymodel-bib1-structure"/>.
1538 The possible values after attribute <literal>type 7</literal> are
1539 <literal>1</literal> ascending and
1540 <literal>2</literal> descending.
1541 The attributes+term (APT) node is separate from the
1542 rest and must be <literal>@or</literal>'ed.
1543 The term associated with APT is the sorting level in integers,
1544 where <literal>0</literal> means primary sort,
1545 <literal>1</literal> means secondary sort, and so forth.
1546 See also <xref linkend="administration-ranking"/>.
1549 For example, searching for water, sort by title (ascending)
1551 Z> find @or @attr 1=1016 water @attr 7=1 @attr 1=4 0
1555 Or, searching for water, sort by title ascending, then date descending
1557 Z> find @or @or @attr 1=1016 water @attr 7=1 @attr 1=4 0 @attr 7=2 @attr 1=30 1
1563 Zebra Extension Term Set Attribute
1564 From the manual text, I can not see what is the point with this feature.
1565 I think it makes more sense when there are multiple terms in a query, or
1568 We decided 2006-06-03 to disable this feature, as it is covered by
1569 scan within a resultset. Better use ressources to upgrade this
1570 feature for good performance.
1574 <sect3 id="querymodel-zebra-attr-estimation">
1575 <title>Zebra Extension Term Set Attribute (type 8)</title>
1577 The Term Set feature is a facility that allows a search to store
1578 hitting terms in a "pseudo" resultset; thus a search (as usual) +
1579 a scan-like facility. Requires a client that can do named result
1580 sets since the search generates two result sets. The value for
1581 attribute 8 is the name of a result set (string). The terms in
1582 the named term set are returned as SUTRS records.
1585 For example, searching for u in title, right truncated, and
1586 storing the result in term set named 'aset'
1588 Z> find @attr 5=1 @attr 1=4 @attr 8=aset u
1592 The model has one serious flaw: we don't know the size of term
1593 set. Experimental. Do not use in production code.
1599 <sect3 id="querymodel-zebra-attr-weight">
1600 <title>Zebra Extension Rank Weight Attribute (type 9)</title>
1603 Rank weight is a way to pass a value to a ranking algorithm - so
1604 that one APT has one value - while another as a different one.
1605 See also <xref linkend="administration-ranking"/>.
1608 For example, searching for utah in title with weight 30 as well
1609 as any with weight 20:
1611 Z> find @attr 2=102 @or @attr 9=30 @attr 1=4 utah @attr 9=20 utah
1615 <sect3 id="querymodel-zebra-attr-limit">
1616 <title>Zebra Extension Approximative Limit Attribute (type 9)</title>
1619 Zebra computes - unless otherwise configured -
1620 the exact hit count for every APT
1621 (leaf) in the query tree. These hit counts are returned as part of
1622 the searchResult-1 facility in the binary encoded Z39.50 search
1626 By setting an estimation limit size of the resultset of the APT
1627 leaves, Zebra stoppes processing the result set when the limit
1629 Hit counts under this limit are still precise, but hit counts over it
1630 are estimated using the statistics gathered from the chopped
1634 Specifying a limit of <literal>0</literal> resuts in exact hit counts.
1637 For example, we might be interested in exact hit count for a, but
1638 for b we allow hit count estimates for 1000 and higher.
1640 Z> find @and a @attr 9=1000 b
1644 The estimated hit count facility makes searches faster, as one
1645 only needs to process large hit lists partially.
1646 It is mostly used in huge databases, where you you want trade
1647 exactness of hit counts against speed of execution.
1650 Do not use approximative hit count limits
1651 in conjunction with relevance ranking, as re-sorting of the
1652 result set obviosly only works when the entire result set has
1656 This facility clashes with rank weight, because there all
1657 documents in the hit lists need to be examined for scoring and
1659 It is an experimental
1660 extension. Do not use in production code.
1663 <sect3 id="querymodel-zebra-attr-termref">
1664 <title>Zebra Extension Term Reference Attribute (type 10)</title>
1667 Zebra supports the <literal>searchResult-1</literal> facility.
1668 If the <literal>Term Reference Attribute (type 10)</literal> is
1669 given, that specifies a subqueryId value returned as part of the
1670 search result. It is a way for a client to name an APT part of a
1680 Experimental. Do not use in production code.
1687 <sect2 id="querymodel-zebra-attr-scan">
1688 <title>Zebra specific Scan Extensions to all Attribute Sets</title>
1690 Zebra extends the Bib1 attribute types, and these extensions are
1691 recognized regardless of attribute
1692 set used in a <literal>scan</literal> operation query.
1694 <table id="querymodel-zebra-attr-scan-table"
1695 frame="all" rowsep="1" colsep="1" align="center">
1697 <caption>Zebra Scan Attribute Extensions</caption>
1703 <td>Zebra version</td>
1708 <td>Result Set Narrow</td>
1714 <td>Approximative Limit</td>
1722 <sect3 id="querymodel-zebra-attr-narrow">
1723 <title>Zebra Extension Result Set Narrow (type 8)</title>
1726 If attribute <literal>Result Set Narrow (type 8)</literal>
1727 is given for <literal>scan</literal>, the value is the name of a
1728 result set. Each hit count in <literal>scan</literal> is
1729 <literal>@and</literal>'ed with the result set given.
1732 Consider for example
1733 the case of scanning all title fields around the
1734 scanterm <emphasis>mozart</emphasis>, then refining the scan by
1735 issuing a filtering query for <emphasis>amadeus</emphasis> to
1736 restrict the scan to the result set of the query:
1738 Z> scan @attr 1=4 mozart
1741 mozartforskningen (1)
1745 Z> f @attr 1=4 amadeus
1747 Number of hits: 15, setno 2
1749 Z> scan @attr 1=4 @attr 8=2 mozart
1752 mozartforskningen (0)
1760 Experimental. Do not use in production code.
1763 <sect3 id="querymodel-zebra-attr-approx">
1764 <title>Zebra Extension Approximative Limit (type 9)</title>
1767 The <literal>Zebra Extension Approximative Limit (type
1768 9)</literal> is a way to enable approximate
1769 hit counts for <literal>scan</literal> hit counts, in the same
1770 way as for <literal>search</literal> hit counts.
1779 Experimental and buggy. Definitely not to be used in production code.
1786 <sect2 id="querymodel-idxpath">
1787 <title>Zebra special IDXPATH Attribute Set for GRS indexing</title>
1789 The attribute-set <literal>idxpath</literal> consists of a single
1790 <literal>Use (type 1)</literal> attribute. All non-use attributes
1794 This feature is enabled when defining the
1795 <literal>xpath enable</literal> option in the GRS filter
1796 <filename>*.abs</filename> configuration files. If one wants to use
1797 the special <literal>idxpath</literal> numeric attribute set, the
1798 main Zebra configuration file <filename>zebra.cfg</filename>
1799 directive <literal>attset: idxpath.att</literal> must be enabled.
1801 <warning>The <literal>idxpath</literal> is deprecated, may not be
1802 supported in future Zebra versions, and should definitely
1803 not be used in production code.
1806 <sect3 id="querymodel-idxpath-use">
1807 <title>IDXPATH Use Attributes (type = 1)</title>
1809 This attribute set allows one to search GRS filter indexed
1810 records by XPATH like structured index names.
1813 <warning>The <literal>idxpath</literal> option defines hard-coded
1814 index names, which might clash with your own index names.
1817 <table id="querymodel-idxpath-use-table"
1818 frame="all" rowsep="1" colsep="1" align="center">
1820 <caption>Zebra specific IDXPATH Use Attributes (type 1)</caption>
1825 <td>String Index</td>
1831 <td>XPATH Begin</td>
1833 <td>_XPATH_BEGIN</td>
1843 <td>XPATH CData</td>
1845 <td>_XPATH_CDATA</td>
1849 <td>XPATH Attribute Name</td>
1851 <td>_XPATH_ATTR_NAME</td>
1855 <td>XPATH Attribute CData</td>
1857 <td>_XPATH_ATTR_CDATA</td>
1865 See <filename>tab/idxpath.att</filename> for more information.
1868 Search for all documents starting with root element
1869 <literal>/root</literal> (either using the numeric or the string
1872 Z> find @attrset idxpath @attr 1=1 @attr 4=3 root/
1873 Z> find @attr idxpath 1=1 @attr 4=3 root/
1874 Z> find @attr 1=_XPATH_BEGIN @attr 4=3 root/
1878 Search for all documents where specific nested XPATH
1879 <literal>/c1/c2/../cn</literal> exists. Notice the very
1880 counter-intuitive <emphasis>reverse</emphasis> notation!
1882 Z> find @attrset idxpath @attr 1=1 @attr 4=3 cn/cn-1/../c1/
1883 Z> find @attr 1=_XPATH_BEGIN @attr 4=3 cn/cn-1/../c1/
1887 Search for CDATA string <emphasis>text</emphasis> in any element
1889 Z> find @attrset idxpath @attr 1=1016 text
1890 Z> find @attr 1=_XPATH_CDATA text
1894 Search for CDATA string <emphasis>anothertext</emphasis> in any
1897 Z> find @attrset idxpath @attr 1=1015 anothertext
1898 Z> find @attr 1=_XPATH_ATTR_CDATA anothertext
1902 Search for all documents with have an XML element node
1903 including an XML attribute named <emphasis>creator</emphasis>
1905 Z> find @attrset idxpath @attr 1=3 @attr 4=3 creator
1906 Z> find @attr 1=_XPATH_ATTR_NAME @attr 4=3 creator
1910 Combining usual <literal>bib-1</literal> attribute set searches
1911 with <literal>idxpath</literal> attribute set searches:
1913 Z> find @and @attr idxpath 1=1 @attr 4=3 link/ @attr 1=4 mozart
1914 Z> find @and @attr 1=_XPATH_BEGIN @attr 4=3 link/ @attr 1=_XPATH_CDATA mozart
1918 Scanning is supported on all <literal>idxpath</literal>
1919 indexes, both specified as numeric use attributes, or as string
1922 Z> scan @attrset idxpath @attr 1=1016 text
1923 Z> scan @attr 1=_XPATH_ATTR_CDATA anothertext
1924 Z> scan @attrset idxpath @attr 1=3 @attr 4=3 ''
1932 <sect2 id="querymodel-pqf-apt-mapping">
1933 <title>Mapping from PQF atomic APT queries to Zebra internal
1934 register indexes</title>
1936 The rules for PQF APT mapping are rather tricky to grasp in the
1937 first place. We deal first with the rules for deciding which
1938 internal register or string index to use, according to the use
1939 attribute or access point specified in the query. Thereafter we
1940 deal with the rules for determining the correct structure type of
1944 <sect3 id="querymodel-pqf-apt-mapping-accesspoint">
1945 <title>Mapping of PQF APT access points</title>
1947 Zebra understands four fundamental different types of access
1948 points, of which only the
1949 <emphasis>numeric use attribute</emphasis> type access points
1950 are defined by the <ulink url="&url.z39.50;">Z39.50</ulink>
1952 All other access point types are Zebra specific, and non-portable.
1955 <table id="querymodel-zebra-mapping-accesspoint-types"
1956 frame="all" rowsep="1" colsep="1" align="center">
1958 <caption>Access point name mapping</caption>
1961 <td>Access Point</td>
1969 <td>Use attribute</td>
1971 <td>[1-9][1-9]*</td>
1972 <td>directly mapped to string index name</td>
1975 <td>String index name</td>
1977 <td>[a-zA-Z](\-?[a-zA-Z0-9])*</td>
1978 <td>normalized name is used as internal string index name</td>
1981 <td>Zebra internal index name</td>
1983 <td>_[a-zA-Z](_?[a-zA-Z0-9])*</td>
1984 <td>hardwired internal string index name</td>
1987 <td>XPATH special index</td>
1990 <td>special xpath search for GRS indexed records</td>
1996 <literal>Attribute set names</literal> and
1997 <literal>string index names</literal> are normalizes
1998 according to the following rules: all <emphasis>single</emphasis>
1999 hyphens <literal>'-'</literal> are stripped, and all upper case
2000 letters are folded to lower case.
2004 <emphasis>Numeric use attributes</emphasis> are mapped
2005 to the Zebra internal
2006 string index according to the attribute set definition in use.
2007 The default attribute set is <literal>Bib-1</literal>, and may be
2008 omitted in the PQF query.
2012 According to normalization and numeric
2013 use attribute mapping, it follows that the following
2014 PQF queries are considered equivalent (assuming the default
2015 configuration has not been altered):
2017 Z> find @attr 1=Body-of-text serenade
2018 Z> find @attr 1=bodyoftext serenade
2019 Z> find @attr 1=BodyOfText serenade
2020 Z> find @attr 1=bO-d-Y-of-tE-x-t serenade
2021 Z> find @attr 1=1010 serenade
2022 Z> find @attrset Bib-1 @attr 1=1010 serenade
2023 Z> find @attrset bib1 @attr 1=1010 serenade
2024 Z> find @attrset Bib1 @attr 1=1010 serenade
2025 Z> find @attrset b-I-b-1 @attr 1=1010 serenade
2030 The <emphasis>numerical</emphasis>
2031 <literal>use attributes (type 1)</literal>
2032 are interpreted according to the
2033 attribute sets which have been loaded in the
2034 <literal>zebra.cfg</literal> file, and are matched against specific
2035 fields as specified in the <literal>.abs</literal> file which
2036 describes the profile of the records which have been loaded.
2037 If no use attribute is provided, a default of
2038 <literal>Bib-1 Use Any (1016)</literal> is
2040 The predefined <literal>use attribute sets</literal>
2041 can be reconfigured by tweaking the configuration files
2042 <filename>tab/*.att</filename>, and
2043 new attribute sets can be defined by adding similar files in the
2044 configuration path <literal>profilePath</literal> of the server.
2048 <literal>String indexes</literal> can be accessed directly,
2049 independently which attribute set is in use. These are just
2050 ignored. The above mentioned name normalization applies.
2051 <literal>String index names</literal> are defined in the
2052 used indexing filter configuration files, for example in the
2053 <literal>GRS</literal>
2054 <filename>*.abs</filename> configuration files, or in the
2055 <literal>alvis</literal> filter XSLT indexing stylesheets.
2059 <literal>Zebra internal indexes</literal> can be accessed directly,
2060 according to the same rules as the user defined
2061 <literal>string indexes</literal>. The only difference is that
2062 <literal>Zebra internal index names</literal> are hardwired,
2064 must start with the character <literal>'_'</literal>.
2068 Finally, <literal>XPATH</literal> access points are only
2069 available using the <literal>GRS</literal> filter for indexing.
2070 These access point names must start with the character
2071 <literal>'/'</literal>, they are <emphasis>not
2072 normalized</emphasis>, but passed unaltered to the Zebra internal
2073 XPATH engine. See <xref linkend="querymodel-use-xpath"/>.
2081 <sect3 id="querymodel-pqf-apt-mapping-structuretype">
2082 <title>Mapping of PQF APT structure and completeness to
2083 register type</title>
2085 Internally Zebra has in it's default configuration several
2086 different types of registers or indexes, whose tokenization and
2087 character normalization rules differ. This reflects the fact that
2088 searching fundamental different tokens like dates, numbers,
2089 bitfields and string based text needs different rule sets.
2092 <table id="querymodel-zebra-mapping-structure-types"
2093 frame="all" rowsep="1" colsep="1" align="center">
2095 <caption>Structure and completeness mapping to register types</caption>
2099 <td>Completeness</td>
2100 <td>Register type</td>
2107 phrase (@attr 4=1), word (@attr 4=2),
2108 word-list (@attr 4=6),
2109 free-form-text (@attr 4=105), or document-text (@attr 4=106)
2111 <td>Incomplete field (@attr 6=1)</td>
2113 <td>Traditional tokenized and character normalized word index</td>
2117 phrase (@attr 4=1), word (@attr 4=2),
2118 word-list (@attr 4=6),
2119 free-form-text (@attr 4=105), or document-text (@attr 4=106)
2121 <td>complete field' (@attr 6=3)</td>
2122 <td>Phrase ('p')</td>
2123 <td>Character normalized, but not tokenized index for phrase
2128 <td>urx (@attr 4=104)</td>
2130 <td>URX/URL ('u')</td>
2131 <td>Special index for URL web addresses</td>
2134 <td>numeric (@attr 4=109)</td>
2136 <td>Numeric ('u')</td>
2137 <td>Special index for digital numbers</td>
2140 <td>key (@attr 4=3)</td>
2142 <td>Null bitmap ('0')</td>
2143 <td>Used for non-tokenizated and non-normalized bit sequences</td>
2146 <td>year (@attr 4=4)</td>
2149 <td>Non-tokenizated and non-normalized 4 digit numbers</td>
2152 <td>date (@attr 4=5)</td>
2155 <td>Non-tokenizated and non-normalized ISO date strings</td>
2161 <td>Used with special sort attribute set (@attr 7=1, @attr 7=2)</td>
2167 <td>Internal record ID register, used whenever
2168 Relation Always Matches (@attr 2=103) is specified</td>
2173 <!-- see in util/zebramap.c -->
2176 If a <emphasis>Structure</emphasis> attribute of
2177 <emphasis>Phrase</emphasis> is used in conjunction with a
2178 <emphasis>Completeness</emphasis> attribute of
2179 <emphasis>Complete (Sub)field</emphasis>, the term is matched
2180 against the contents of the phrase (long word) register, if one
2181 exists for the given <emphasis>Use</emphasis> attribute.
2182 A phrase register is created for those fields in the
2183 GRS <filename>*.abs</filename> file that contains a
2184 <literal>p</literal>-specifier.
2186 Z> scan @attr 1=Title @attr 4=1 @attr 6=3 beethoven
2188 bayreuther festspiele (1)
2189 * beethoven bibliography database (1)
2192 Z> find @attr 1=Title @attr 4=1 @attr 6=3 "beethoven bibliography"
2194 Number of hits: 0, setno 5
2196 Z> find @attr 1=Title @attr 4=1 @attr 6=3 "beethoven bibliography database"
2198 Number of hits: 1, setno 6
2203 If <emphasis>Structure</emphasis>=<emphasis>Phrase</emphasis> is
2204 used in conjunction with <emphasis>Incomplete Field</emphasis> - the
2205 default value for <emphasis>Completeness</emphasis>, the
2206 search is directed against the normal word registers, but if the term
2207 contains multiple words, the term will only match if all of the words
2208 are found immediately adjacent, and in the given order.
2209 The word search is performed on those fields that are indexed as
2210 type <literal>w</literal> in the GRS <filename>*.abs</filename> file.
2212 Z> scan @attr 1=Title @attr 4=1 @attr 6=1 beethoven
2218 Z> find @attr 1=Title @attr 4=1 @attr 6=1 beethoven
2220 Number of hits: 18, setno 1
2222 Z> find @attr 1=Title @attr 4=1 @attr 6=1 "beethoven bibliography"
2224 Number of hits: 2, setno 2
2230 If the <emphasis>Structure</emphasis> attribute is
2231 <emphasis>Word List</emphasis>,
2232 <emphasis>Free-form Text</emphasis>, or
2233 <emphasis>Document Text</emphasis>, the term is treated as a
2234 natural-language, relevance-ranked query.
2235 This search type uses the word register, i.e. those fields
2236 that are indexed as type <literal>w</literal> in the
2237 GRS <filename>*.abs</filename> file.
2241 If the <emphasis>Structure</emphasis> attribute is
2242 <emphasis>Numeric String</emphasis> the term is treated as an integer.
2243 The search is performed on those fields that are indexed
2244 as type <literal>n</literal> in the GRS
2245 <filename>*.abs</filename> file.
2249 If the <emphasis>Structure</emphasis> attribute is
2250 <emphasis>URX</emphasis> the term is treated as a URX (URL) entity.
2251 The search is performed on those fields that are indexed as type
2252 <literal>u</literal> in the <filename>*.abs</filename> file.
2256 If the <emphasis>Structure</emphasis> attribute is
2257 <emphasis>Local Number</emphasis> the term is treated as
2258 native Zebra Record Identifier.
2262 If the <emphasis>Relation</emphasis> attribute is
2263 <emphasis>Equals</emphasis> (default), the term is matched
2264 in a normal fashion (modulo truncation and processing of
2265 individual words, if required).
2266 If <emphasis>Relation</emphasis> is <emphasis>Less Than</emphasis>,
2267 <emphasis>Less Than or Equal</emphasis>,
2268 <emphasis>Greater than</emphasis>, or <emphasis>Greater than or
2269 Equal</emphasis>, the term is assumed to be numerical, and a
2270 standard regular expression is constructed to match the given
2272 If <emphasis>Relation</emphasis> is <emphasis>Relevance</emphasis>,
2273 the standard natural-language query processor is invoked.
2277 For the <emphasis>Truncation</emphasis> attribute,
2278 <emphasis>No Truncation</emphasis> is the default.
2279 <emphasis>Left Truncation</emphasis> is not supported.
2280 <emphasis>Process # in search term</emphasis> is supported, as is
2281 <emphasis>Regxp-1</emphasis>.
2282 <emphasis>Regxp-2</emphasis> enables the fault-tolerant (fuzzy)
2283 search. As a default, a single error (deletion, insertion,
2284 replacement) is accepted when terms are matched against the register
2291 <sect2 id="querymodel-regular">
2292 <title>Zebra Regular Expressions in Truncation Attribute (type = 5)</title>
2295 Each term in a query is interpreted as a regular expression if
2296 the truncation value is either <emphasis>Regxp-1 (@attr 5=102)</emphasis>
2297 or <emphasis>Regxp-2 (@attr 5=103)</emphasis>.
2298 Both query types follow the same syntax with the operands:
2301 <table id="querymodel-regular-operands-table"
2302 frame="all" rowsep="1" colsep="1" align="center">
2304 <caption>Regular Expression Operands</caption>
2307 <tr><td>one</td><td>two</td></tr>
2312 <td><literal>x</literal></td>
2313 <td>Matches the character <literal>x</literal>.</td>
2316 <td><literal>.</literal></td>
2317 <td>Matches any character.</td>
2320 <td><literal>[ .. ]</literal></td>
2321 <td>Matches the set of characters specified;
2322 such as <literal>[abc]</literal> or <literal>[a-c]</literal>.</td>
2328 The above operands can be combined with the following operators:
2331 <table id="querymodel-regular-operators-table"
2332 frame="all" rowsep="1" colsep="1" align="center">
2333 <caption>Regular Expression Operators</caption>
2336 <tr><td>one</td><td>two</td></tr>
2341 <td><literal>x*</literal></td>
2342 <td>Matches <literal>x</literal> zero or more times.
2343 Priority: high.</td>
2346 <td><literal>x+</literal></td>
2347 <td>Matches <literal>x</literal> one or more times.
2348 Priority: high.</td>
2351 <td><literal>x?</literal></td>
2352 <td> Matches <literal>x</literal> zero or once.
2353 Priority: high.</td>
2356 <td><literal>xy</literal></td>
2357 <td> Matches <literal>x</literal>, then <literal>y</literal>.
2358 Priority: medium.</td>
2361 <td><literal>x|y</literal></td>
2362 <td> Matches either <literal>x</literal> or <literal>y</literal>.
2366 <td><literal>( )</literal></td>
2367 <td>The order of evaluation may be changed by using parentheses.</td>
2373 If the first character of the <literal>Regxp-2</literal> query
2374 is a plus character (<literal>+</literal>) it marks the
2375 beginning of a section with non-standard specifiers.
2376 The next plus character marks the end of the section.
2377 Currently Zebra only supports one specifier, the error tolerance,
2378 which consists one digit.
2379 <!-- TODO Nice thing, but what does
2380 that error tolerance digit *mean*? Maybe an example would be nice? -->
2384 Since the plus operator is normally a suffix operator the addition to
2385 the query syntax doesn't violate the syntax for standard regular
2390 For example, a phrase search with regular expressions in
2391 the title-register is performed like this:
2393 Z> find @attr 1=4 @attr 5=102 "informat.* retrieval"
2398 Combinations with other attributes are possible. For example, a
2399 ranked search with a regular expression:
2401 Z> find @attr 1=4 @attr 5=102 @attr 2=102 "informat.* retrieval"
2409 The RecordType parameter in the <literal>zebra.cfg</literal> file, or
2410 the <literal>-t</literal> option to the indexer tells Zebra how to
2411 process input records.
2412 Two basic types of processing are available - raw text and structured
2413 data. Raw text is just that, and it is selected by providing the
2414 argument <literal>text</literal> to Zebra. Structured records are
2415 all handled internally using the basic mechanisms described in the
2416 subsequent sections.
2417 Zebra can read structured records in many different formats.
2423 <sect1 id="querymodel-cql-to-pqf">
2424 <title>Server Side CQL to PQF Query Translation</title>
2427 <literal><cql2rpn>l2rpn.txt</cql2rpn></literal>
2428 YAZ Frontend Virtual
2429 Hosts option, one can configure
2430 the YAZ Frontend CQL-to-PQF
2431 converter, specifying the interpretation of various
2432 <ulink url="&url.cql;">CQL</ulink>
2433 indexes, relations, etc. in terms of Type-1 query attributes.
2434 <!-- The yaz-client config file -->
2437 For example, using server-side CQL-to-PQF conversion, one might
2438 query a zebra server like this:
2441 yaz-client localhost:9999
2443 Z> find text=(plant and soil)
2446 and - if properly configured - even static relevance ranking can
2447 be performed using CQL query syntax:
2450 Z> find text = /relevant (plant and soil)
2456 By the way, the same configuration can be used to
2457 search using client-side CQL-to-PQF conversion:
2458 (the only difference is <literal>querytype cql2rpn</literal>
2460 <literal>querytype cql</literal>, and the call specifying a local
2464 yaz-client -q local/cql2pqf.txt localhost:9999
2465 Z> querytype cql2rpn
2466 Z> find text=(plant and soil)
2472 Exhaustive information can be found in the
2473 Section "Specification of CQL to RPN mappings" in the YAZ manual.
2474 <ulink url="http://www.indexdata.dk/yaz/doc/tools.tkl#tools.cql.map">
2475 http://www.indexdata.dk/yaz/doc/tools.tkl#tools.cql.map</ulink>,
2476 and shall therefore not be repeated here.
2481 <ulink url="http://www.loc.gov/z3950/agency/zing/cql/dc-indexes.html">
2482 http://www.loc.gov/z3950/agency/zing/cql/dc-indexes.html</ulink>
2483 for the Maintenance Agency's work-in-progress mapping of Dublin Core
2484 indexes to Attribute Architecture (util, XD and BIB-2)
2494 <!-- Keep this comment at the end of the file
2499 sgml-minimize-attributes:nil
2500 sgml-always-quote-attributes:t
2503 sgml-parent-document: "zebra.xml"
2504 sgml-local-catalogs: nil
2505 sgml-namecase-general:t