1 <chapter id="querymodel">
2 <title>Query Model</title>
4 <section id="querymodel-overview">
5 <title>Query Model Overview</title>
7 <section id="querymodel-query-languages">
8 <title>Query Languages</title>
11 &zebra; is born as a networking Information Retrieval engine adhering
12 to the international standards
13 <ulink url="&url.z39.50;">&acro.z3950;</ulink> and
14 <ulink url="&url.sru;">&acro.sru;</ulink>,
16 type-1 Reverse Polish Notation (&acro.rpn;) query
18 Unfortunately, this model has only defined a binary
19 encoded representation, which is used as transport packaging in
20 the &acro.z3950; protocol layer. This representation is not human
21 readable, nor defines any convenient way to specify queries.
24 Since the type-1 (&acro.rpn;)
25 query structure has no direct, useful string
26 representation, every client application needs to provide some
27 form of mapping from a local query notation or representation to it.
31 <section id="querymodel-query-languages-pqf">
32 <title>Prefix Query Format (&acro.pqf;)</title>
34 Index Data has defined a textual representation in the
35 <ulink url="&url.yaz.pqf;">Prefix Query Format</ulink>, short
36 <emphasis>&acro.pqf;</emphasis>, which maps
37 one-to-one to binary encoded
38 <emphasis>type-1 &acro.rpn;</emphasis> queries.
39 &acro.pqf; has been adopted by other
40 parties developing &acro.z3950; software, and is often referred to as
41 <emphasis>Prefix Query Notation</emphasis>, or in short
43 <xref linkend="querymodel-rpn"/> for further explanations and
44 descriptions of &zebra;'s capabilities.
48 <section id="querymodel-query-languages-cql">
49 <title>Common Query Language (&acro.cql;)</title>
51 The query model of the type-1 &acro.rpn;,
52 expressed in &acro.pqf;/&acro.pqn; is natively supported.
53 On the other hand, the default &acro.sru;
54 web services <emphasis>Common Query Language</emphasis>
55 <ulink url="&url.cql;">&acro.cql;</ulink> is not natively supported.
58 &zebra; can be configured to understand and map &acro.cql; to &acro.pqf;. See
59 <xref linkend="querymodel-cql-to-pqf"/>.
65 <section id="querymodel-operation-types">
66 <title>Operation types</title>
68 &zebra; supports all of the three different
69 &acro.z3950;/&acro.sru; operations defined in the
70 standards: explain, search,
71 and scan. A short description of the
72 functionality and purpose of each is quite in order here.
75 <section id="querymodel-operation-type-explain">
76 <title>Explain Operation</title>
78 The <emphasis>syntax</emphasis> of &acro.z3950;/&acro.sru; queries is
79 well known to any client, but the specific
80 <emphasis>semantics</emphasis> - taking into account a
81 particular servers functionalities and abilities - must be
82 discovered from case to case. Enters the
83 explain operation, which provides the means for learning which
84 <emphasis>fields</emphasis> (also called
85 <emphasis>indexes</emphasis> or <emphasis>access points</emphasis>)
86 are provided, which default parameter the server uses, which
87 retrieve document formats are defined, and which specific parts
88 of the general query model are supported.
91 The &acro.z3950; embeds the explain operation
94 <literal>IR-Explain-1</literal> database;
95 see <xref linkend="querymodel-exp1"/>.
98 In &acro.sru;, explain is an entirely separate
99 operation, which returns an ZeeRex &acro.xml; record according to the
100 structure defined by the protocol.
103 In both cases, the information gathered through
104 explain operations can be used to
105 auto-configure a client user interface to the servers
110 <section id="querymodel-operation-type-search">
111 <title>Search Operation</title>
113 Search and retrieve interactions are the raison d'ĂȘtre.
114 They are used to query the remote database and
115 return search result documents. Search queries span from
116 simple free text searches to nested complex boolean queries,
117 targeting specific indexes, and possibly enhanced with many
118 query semantic specifications. Search interactions are the heart
119 and soul of &acro.z3950;/&acro.sru; servers.
123 <section id="querymodel-operation-type-scan">
124 <title>Scan Operation</title>
126 The scan operation is a helper functionality,
127 which operates on one index or access point a time.
131 the means to investigate the content of specific indexes.
132 Scanning an index returns a handful of terms actually found in
133 the indexes, and in addition the scan
134 operation returns the number of documents indexed by each term.
135 A search client can use this information to propose proper
136 spelling of search terms, to auto-fill search boxes, or to
137 display controlled vocabularies.
145 <section id="querymodel-rpn">
146 <title>&acro.rpn; queries and semantics</title>
148 The <ulink url="&url.yaz.pqf;">&acro.pqf; grammar</ulink>
149 is documented in the &yaz; manual, and shall not be
150 repeated here. This textual &acro.pqf; representation
151 is not transmitted to &zebra; during search, but it is in the
152 client mapped to the equivalent &acro.z3950; binary
156 <section id="querymodel-rpn-tree">
157 <title>&acro.rpn; tree structure</title>
159 The &acro.rpn; parse tree - or the equivalent textual representation in &acro.pqf; -
160 may start with one specification of the
161 <emphasis>attribute set</emphasis> used. Following is a query
163 consists of <emphasis>atomic query parts (&acro.apt;)</emphasis> or
164 <emphasis>named result sets</emphasis>, eventually
165 paired by <emphasis>boolean binary operators</emphasis>, and
166 finally <emphasis>recursively combined </emphasis> into
170 <section id="querymodel-attribute-sets">
171 <title>Attribute sets</title>
173 Attribute sets define the exact meaning and semantics of queries
174 issued. &zebra; comes with some predefined attribute set
175 definitions, others can easily be defined and added to the
179 <table id="querymodel-attribute-sets-table" frame="top">
180 <title>Attribute sets predefined in &zebra;</title>
184 <entry>Attribute set</entry>
185 <entry>&acro.pqf; notation (Short hand)</entry>
186 <entry>Status</entry>
193 <entry>Explain</entry>
194 <entry><literal>exp-1</literal></entry>
195 <entry>Special attribute set used on the special automagic
196 <literal>IR-Explain-1</literal> database to gain information on
197 server capabilities, database names, and database
198 and semantics.</entry>
199 <entry>predefined</entry>
202 <entry>&acro.bib1;</entry>
203 <entry><literal>bib-1</literal></entry>
204 <entry>Standard &acro.pqf; query language attribute set which defines the
205 semantics of &acro.z3950; searching. In addition, all of the
206 non-use attributes (types 2-14) define the hard-wired
207 &zebra; internal query
209 <entry>default</entry>
213 <entry><literal>gils</literal></entry>
214 <entry>Extension to the &acro.bib1; attribute set.</entry>
215 <entry>predefined</entry>
222 The use attributes (type 1) mappings the
223 predefined attribute sets are found in the
224 attribute set configuration files <filename>tab/*.att</filename>.
229 The &zebra; internal query processing is modeled after
230 the &acro.bib1; attribute set, and the non-use
231 attributes type 2-6 are hard-wired in. It is therefore essential
232 to be familiar with <xref linkend="querymodel-bib1-nonuse"/>.
238 <section id="querymodel-boolean-operators">
239 <title>Boolean operators</title>
241 A pair of sub query trees, or of atomic queries, is combined
242 using the standard boolean operators into new query trees.
243 Thus, boolean operators are always internal nodes in the query tree.
246 <table id="querymodel-boolean-operators-table" frame="top">
247 <title>Boolean operators</title>
251 <entry>Keyword</entry>
252 <entry>Operator</entry>
253 <entry>Description</entry>
257 <row><entry><literal>@and</literal></entry>
258 <entry>binary AND operator</entry>
259 <entry>Set intersection of two atomic queries hit sets</entry>
261 <row><entry><literal>@or</literal></entry>
262 <entry>binary OR operator</entry>
263 <entry>Set union of two atomic queries hit sets</entry>
265 <row><entry><literal>@not</literal></entry>
266 <entry>binary AND NOT operator</entry>
267 <entry>Set complement of two atomic queries hit sets</entry>
269 <row><entry><literal>@prox</literal></entry>
270 <entry>binary PROXIMITY operator</entry>
271 <entry>Set intersection of two atomic queries hit sets. In
272 addition, the intersection set is purged for all
273 documents which do not satisfy the requested query
274 term proximity. Usually a proper subset of the AND
282 For example, we can combine the terms
283 <emphasis>information</emphasis> and <emphasis>retrieval</emphasis>
284 into different searches in the default index of the default
285 attribute set as follows.
286 Querying for the union of all documents containing the
287 terms <emphasis>information</emphasis> OR
288 <emphasis>retrieval</emphasis>:
290 Z> find @or information retrieval
294 Querying for the intersection of all documents containing the
295 terms <emphasis>information</emphasis> AND
296 <emphasis>retrieval</emphasis>:
297 The hit set is a subset of the corresponding
300 Z> find @and information retrieval
304 Querying for the intersection of all documents containing the
305 terms <emphasis>information</emphasis> AND
306 <emphasis>retrieval</emphasis>, taking proximity into account:
307 The hit set is a subset of the corresponding
309 (see the <ulink url="&url.yaz.pqf;">&acro.pqf; grammar</ulink> for
310 details on the proximity operator):
312 Z> find @prox 0 3 0 2 k 2 information retrieval
316 Querying for the intersection of all documents containing the
317 terms <emphasis>information</emphasis> AND
318 <emphasis>retrieval</emphasis>, in the same order and near each
319 other as described in the term list.
320 The hit set is a subset of the corresponding
323 Z> find "information retrieval"
329 <section id="querymodel-atomic-queries">
330 <title>Atomic queries (&acro.apt;)</title>
332 Atomic queries are the query parts which work on one access point
333 only. These consist of <emphasis>an attribute list</emphasis>
334 followed by a <emphasis>single term</emphasis> or a
335 <emphasis>quoted term list</emphasis>, and are often called
336 <emphasis>Attributes-Plus-Terms (&acro.apt;)</emphasis> queries.
339 Atomic (&acro.apt;) queries are always leaf nodes in the &acro.pqf; query tree.
340 UN-supplied non-use attributes types 2-12 are either inherited from
341 higher nodes in the query tree, or are set to &zebra;'s default values.
342 See <xref linkend="querymodel-bib1"/> for details.
345 <table id="querymodel-atomic-queries-table" frame="top">
346 <title>Atomic queries (&acro.apt;)</title>
357 <entry><emphasis>attribute list</emphasis></entry>
358 <entry>List of <emphasis>orthogonal</emphasis> attributes</entry>
359 <entry>Any of the orthogonal attribute types may be omitted,
360 these are inherited from higher query tree nodes, or if not
361 inherited, are set to the default &zebra; configuration values.
365 <entry><emphasis>term</emphasis></entry>
366 <entry>single <emphasis>term</emphasis>
367 or <emphasis>quoted term list</emphasis> </entry>
368 <entry>Here the search terms or list of search terms is added
375 Querying for the term <emphasis>information</emphasis> in the
376 default index using the default attribute set, the server choice
377 of access point/index, and the default non-use attributes.
383 Equivalent query fully specified including all default values:
385 Z> find @attrset bib-1 @attr 1=1017 @attr 2=3 @attr 3=3 @attr 4=1 @attr 5=100 @attr 6=1 information
390 Finding all documents which have the term
391 <emphasis>debussy</emphasis> in the title field.
393 Z> find @attr 1=4 debussy
398 The <emphasis>scan</emphasis> operation is only supported with
399 atomic &acro.apt; queries, as it is bound to one access point at a
400 time. Boolean query trees are not allowed during
401 <emphasis>scan</emphasis>.
405 For example, we might want to scan the title index, starting with
407 <emphasis>debussy</emphasis>, and displaying this and the
408 following terms in lexicographic order:
410 Z> scan @attr 1=4 debussy
416 <section id="querymodel-resultset">
417 <title>Named Result Sets</title>
419 Named result sets are supported in &zebra;, and result sets can be
420 used as operands without limitations. It follows that named
421 result sets are leaf nodes in the &acro.pqf; query tree, exactly as
422 atomic &acro.apt; queries are.
425 After the execution of a search, the result set is available at
426 the server, such that the client can use it for subsequent
427 searches or retrieval requests. The Z30.50 standard actually
428 stresses the fact that result sets are volatile. It may cease
429 to exist at any time point after search, and the server will
430 send a diagnostic to the effect that the requested
431 result set does not exist any more.
435 Defining a named result set and re-using it in the next query,
436 using <application>yaz-client</application>. Notice that the client, not
437 the server, assigns the string '1' to the
440 Z> f @attr 1=4 mozart
442 Number of hits: 43, setno 1
444 Z> f @and @set 1 @attr 1=4 amadeus
446 Number of hits: 14, setno 2
452 Named result sets are only supported by the &acro.z3950; protocol.
453 The &acro.sru; web service is stateless, and therefore the notion of
454 named result sets does not exist when accessing a &zebra; server by
455 the &acro.sru; protocol.
460 <section id="querymodel-use-string">
461 <title>&zebra;'s special access point of type 'string'</title>
463 The numeric <emphasis>use (type 1)</emphasis> attribute is usually
464 referred to from a given
465 attribute set. In addition, &zebra; let you use
466 <emphasis>any internal index
467 name defined in your configuration</emphasis>
468 as use attribute value. This is a great feature for
469 debugging, and when you do
470 not need the complexity of defined use attribute values. It is
471 the preferred way of accessing &zebra; indexes directly.
474 Finding all documents which have the term list "information
475 retrieval" in an &zebra; index, using its internal full string
476 name. Scanning the same index.
478 Z> find @attr 1=sometext "information retrieval"
479 Z> scan @attr 1=sometext aterm
483 Searching or scanning
484 the bib-1 use attribute 54 using its string name:
486 Z> find @attr 1=Code-language eng
487 Z> scan @attr 1=Code-language ""
491 It is possible to search
492 in any silly string index - if it's defined in your
493 indexing rules and can be parsed by the &acro.pqf; parser.
494 This is definitely not the recommended use of
495 this facility, as it might confuse your users with some very
498 Z> find @attr 1=silly/xpath/alike[@index]/name "information retrieval"
502 See also <xref linkend="querymodel-pqf-apt-mapping"/> for details, and
503 <xref linkend="zebrasrv-sru"/>
504 for the &acro.sru; &acro.pqf; query extension using string names as a fast
509 <section id="querymodel-use-xpath">
510 <title>&zebra;'s special access point of type 'XPath'
511 for &acro.grs1; filters</title>
513 As we have seen above, it is possible (albeit seldom a great
515 <ulink url="http://www.w3.org/TR/xpath">XPath 1.0</ulink> based
516 search by defining <emphasis>use (type 1)</emphasis>
517 <emphasis>string</emphasis> attributes which in appearance
518 <emphasis>resemble XPath queries</emphasis>. There are two
519 problems with this approach: first, the XPath-look-alike has to
520 be defined at indexing time, no new undefined
521 XPath queries can entered at search time, and second, it might
522 confuse users very much that an XPath-alike index name in fact
523 gets populated from a possible entirely different &acro.xml; element
524 than it pretends to access.
527 When using the &acro.grs1; Record Model
528 (see <xref linkend="grs"/>), we have the
529 possibility to embed <emphasis>life</emphasis>
531 in the &acro.pqf; queries, which are here called
532 <emphasis>use (type 1)</emphasis> <emphasis>xpath</emphasis>
533 attributes. You must enable the
534 <literal>xpath enable</literal> directive in your
535 <literal>.abs</literal> configuration files.
539 Only a <emphasis>very</emphasis> restricted subset of the
540 <ulink url="http://www.w3.org/TR/xpath">XPath 1.0</ulink>
541 standard is supported as the &acro.grs1; record model is simpler than
542 a full &acro.xml; &acro.dom; structure. See the following examples for
547 Finding all documents which have the term "content"
548 inside a text node found in a specific &acro.xml; &acro.dom;
549 <emphasis>subtree</emphasis>, whose starting element is
552 Z> find @attr 1=/root content
553 Z> find @attr 1=/root/first content
555 <emphasis>Notice that the
556 XPath must be absolute, i.e., must start with '/', and that the
557 XPath <literal>descendant-or-self</literal> axis followed by a
558 text node selection <literal>text()</literal> is implicitly
559 appended to the stated XPath.
561 It follows that the above searches are interpreted as:
563 Z> find @attr 1=/root//text() content
564 Z> find @attr 1=/root/first//text() content
569 Searching inside attribute strings is possible:
571 Z> find @attr 1=/link/@creator morten
576 Filter the addressing XPath by a predicate working on exact
578 attributes (in the &acro.xml; sense) can be done: return all those docs which
579 have the term "english" contained in one of all text sub nodes of
580 the subtree defined by the XPath
581 <literal>/record/title[@lang='en']</literal>. And similar
584 Z> find @attr 1=/record/title[@lang='en'] english
585 Z> find @attr 1=/link[@creator='sisse'] sibelius
586 Z> find @attr 1=/link[@creator='sisse']/description[@xml:lang='da'] sibelius
591 Combining numeric indexes, boolean expressions,
592 and xpath based searches is possible:
594 Z> find @attr 1=/record/title @and foo bar
595 Z> find @and @attr 1=/record/title foo @attr 1=4 bar
599 Escaping &acro.pqf; keywords and other non-parseable XPath constructs
600 with <literal>'{ }'</literal> to prevent client-side &acro.pqf; parsing
603 Z> find @attr {1=/root/first[@attr='danish']} content
604 Z> find @attr {1=/record/@set} oai
609 It is worth mentioning that these dynamic performed XPath
610 queries are a performance bottleneck, as no optimized
611 specialized indexes can be used. Therefore, avoid the use of
612 this facility when speed is essential, and the database content
613 size is medium to large.
619 <section id="querymodel-exp1">
620 <title>Explain Attribute Set</title>
622 The &acro.z3950; standard defines the
623 <ulink url="&url.z39.50.explain;">Explain</ulink> attribute set
624 Exp-1, which is used to discover information
625 about a server's search semantics and functional capabilities
626 &zebra; exposes a "classic"
627 Explain database by base name <literal>IR-Explain-1</literal>, which
628 is populated with system internal information.
631 The attribute-set <literal>exp-1</literal> consists of a single
632 use attribute (type 1).
635 In addition, the non-Use
636 &acro.bib1; attributes, that is, the types
637 <emphasis>Relation</emphasis>, <emphasis>Position</emphasis>,
638 <emphasis>Structure</emphasis>, <emphasis>Truncation</emphasis>,
639 and <emphasis>Completeness</emphasis> are imported from
640 the &acro.bib1; attribute set, and may be used
641 within any explain query.
644 <section id="querymodel-exp1-use">
645 <title>Use Attributes (type = 1)</title>
647 The following Explain search attributes are supported:
648 <literal>ExplainCategory</literal> (@attr 1=1),
649 <literal>DatabaseName</literal> (@attr 1=3),
650 <literal>DateAdded</literal> (@attr 1=9),
651 <literal>DateChanged</literal>(@attr 1=10).
654 A search in the use attribute <literal>ExplainCategory</literal>
655 supports only these predefined values:
656 <literal>CategoryList</literal>, <literal>TargetInfo</literal>,
657 <literal>DatabaseInfo</literal>, <literal>AttributeDetails</literal>.
660 See <filename>tab/explain.att</filename> and the
661 <ulink url="&url.z39.50;">&acro.z3950;</ulink> standard
662 for more information.
666 <section id="querymodel-examples">
667 <title>Explain searches with yaz-client</title>
669 Classic Explain only defines retrieval of Explain information
670 via ASN.1. Practically no &acro.z3950; clients supports this. Fortunately
671 they don't have to - &zebra; allows retrieval of this information
673 &acro.sutrs;, &acro.xml;,
674 &acro.grs1; and <literal>ASN.1</literal> Explain.
678 List supported categories to find out which explain commands are
682 Z> find @attr exp1 1=1 categorylist
689 Get target info, that is, investigate which databases exist at
690 this server endpoint:
693 Z> find @attr exp1 1=1 targetinfo
704 List all supported databases, the number of hits
705 is the number of databases found, which most commonly are the
707 the <literal>Default</literal> and the
708 <literal>IR-Explain-1</literal> databases.
711 Z> find @attr exp1 1=1 databaseinfo
718 Get database info record for database <literal>Default</literal>.
721 Z> find @and @attr exp1 1=1 databaseinfo @attr exp1 1=3 Default
723 Identical query with explicitly specified attribute set:
726 Z> find @attrset exp1 @and @attr 1=1 databaseinfo @attr 1=3 Default
731 Get attribute details record for database
732 <literal>Default</literal>.
733 This query is very useful to study the internal &zebra; indexes.
734 If records have been indexed using the <literal>alvis</literal>
735 &acro.xslt; filter, the string representation names of the known indexes can be
739 Z> find @and @attr exp1 1=1 attributedetails @attr exp1 1=3 Default
741 Identical query with explicitly specified attribute set:
744 Z> find @attrset exp1 @and @attr 1=1 attributedetails @attr 1=3 Default
751 <section id="querymodel-bib1">
752 <title>&acro.bib1; Attribute Set</title>
754 Most of the information contained in this section is an excerpt of
755 the ATTRIBUTE SET &acro.bib1; (&acro.z3950;-1995) SEMANTICS
756 found at <ulink url="&url.z39.50.attset.bib1.1995;">. The &acro.bib1;
757 Attribute Set Semantics</ulink> from 1995, also in an updated
758 <ulink url="&url.z39.50.attset.bib1;">&acro.bib1;
759 Attribute Set</ulink>
760 version from 2003. Index Data is not the copyright holder of this
761 information, except for the configuration details, the listing of
762 &zebra;'s capabilities, and the example queries.
766 <section id="querymodel-bib1-use">
767 <title>Use Attributes (type 1)</title>
770 A use attribute specifies an access point for any atomic query.
771 These access points are highly dependent on the attribute set used
772 in the query, and are user configurable using the following
773 default configuration files:
774 <filename>tab/bib1.att</filename>,
775 <filename>tab/dan1.att</filename>,
776 <filename>tab/explain.att</filename>, and
777 <filename>tab/gils.att</filename>.
780 For example, some few &acro.bib1; use
781 attributes from the <filename>tab/bib1.att</filename> are:
785 att 3 Conference-name
788 att 1009 Subject-name-personal
789 att 1010 Body-of-text
790 att 1011 Date/time-added-to-db
793 att 1017 Server-choice
797 att 1036 Author-Title-Subject
801 New attribute sets can be added by adding new
802 <filename>tab/*.att</filename> configuration files, which need to
803 be sourced in the main configuration <filename>zebra.cfg</filename>.
806 In addition, &zebra; allows the access of
807 <emphasis>internal index names</emphasis> and <emphasis>dynamic
808 XPath</emphasis> as use attributes; see
809 <xref linkend="querymodel-use-string"/> and
810 <xref linkend="querymodel-use-xpath"/>.
814 Phrase search for <emphasis>information retrieval</emphasis> in
815 the title-register, scanning the same register afterwards:
817 Z> find @attr 1=4 "information retrieval"
818 Z> scan @attr 1=4 information
826 <section id="querymodel-bib1-nonuse">
827 <title>&zebra; general Bib1 Non-Use Attributes (type 2-6)</title>
829 <section id="querymodel-bib1-relation">
830 <title>Relation Attributes (type 2)</title>
833 Relation attributes describe the relationship of the access
835 of the relation) to the search term as qualified by the attributes (right
836 side of the relation), e.g., Date-publication <= 1975.
839 <table id="querymodel-bib1-relation-table" frame="top">
840 <title>Relation Attributes (type 2)</title>
844 <entry>Relation</entry>
851 <entry>Less than</entry>
853 <entry>supported</entry>
856 <entry>Less than or equal</entry>
858 <entry>supported</entry>
863 <entry>default</entry>
866 <entry>Greater or equal</entry>
868 <entry>supported</entry>
871 <entry>Greater than</entry>
873 <entry>supported</entry>
876 <entry>Not equal</entry>
878 <entry>unsupported</entry>
881 <entry>Phonetic</entry>
883 <entry>unsupported</entry>
888 <entry>unsupported</entry>
891 <entry>Relevance</entry>
893 <entry>supported</entry>
896 <entry>AlwaysMatches</entry>
898 <entry>supported *</entry>
905 AlwaysMatches searches are only supported if alwaysmatches indexing
906 has been enabled. See <xref linkend="default-idx-file"/>
911 The relation attributes 1-5 are supported and work exactly as
913 All ordering operations are based on a lexicographical ordering,
914 <emphasis>except</emphasis> when the
915 structure attribute numeric (109) is used. In
916 this case, ordering is numerical. See
917 <xref linkend="querymodel-bib1-structure"/>.
919 Z> find @attr 1=Title @attr 2=1 music
921 Number of hits: 11745, setno 1
923 Z> find @attr 1=Title @attr 2=2 music
925 Number of hits: 11771, setno 2
927 Z> find @attr 1=Title @attr 2=3 music
929 Number of hits: 532, setno 3
931 Z> find @attr 1=Title @attr 2=4 music
933 Number of hits: 11463, setno 4
935 Z> find @attr 1=Title @attr 2=5 music
937 Number of hits: 11419, setno 5
942 The relation attribute
943 <emphasis>Relevance (102)</emphasis> is supported, see
944 <xref linkend="administration-ranking"/> for full information.
948 Ranked search for <emphasis>information retrieval</emphasis> in
951 Z> find @attr 1=4 @attr 2=102 "information retrieval"
956 The relation attribute
957 <emphasis>AlwaysMatches (103)</emphasis> is in the default
959 supported in conjecture with structure attribute
960 <emphasis>Phrase (1)</emphasis> (which may be omitted by
962 It can be configured to work with other structure attributes,
963 see the configuration file
964 <filename>tab/default.idx</filename> and
965 <xref linkend="querymodel-pqf-apt-mapping"/>.
968 <emphasis>AlwaysMatches (103)</emphasis> is a
969 great way to discover how many documents have been indexed in a
970 given field. The search term is ignored, but needed for correct
971 &acro.pqf; syntax. An empty search term may be supplied.
973 Z> find @attr 1=Title @attr 2=103 ""
974 Z> find @attr 1=Title @attr 2=103 @attr 4=1 ""
981 <section id="querymodel-bib1-position">
982 <title>Position Attributes (type 3)</title>
985 The position attribute specifies the location of the search term
986 within the field or subfield in which it appears.
989 <table id="querymodel-bib1-position-table" frame="top">
990 <title>Position Attributes (type 3)</title>
994 <entry>Position</entry>
1001 <entry>First in field </entry>
1003 <entry>supported *</entry>
1006 <entry>First in subfield</entry>
1008 <entry>supported *</entry>
1011 <entry>Any position in field</entry>
1013 <entry>default</entry>
1021 &zebra; only supports first-in-field seaches if the
1022 <literal>firstinfield</literal> is enabled for the index
1023 Refer to <xref linkend="default-idx-file"/>.
1024 &zebra; does not distinguish between first in field and
1025 first in subfield. They result in the same hit count.
1026 Searching for first position in (sub)field in only supported in &zebra;
1032 <section id="querymodel-bib1-structure">
1033 <title>Structure Attributes (type 4)</title>
1036 The structure attribute specifies the type of search
1037 term. This causes the search to be mapped on
1038 different &zebra; internal indexes, which must have been defined
1043 The possible values of the
1044 <literal>structure attribute (type 4)</literal> can be defined
1045 using the configuration file <filename>tab/default.idx</filename>.
1046 The default configuration is summarized in this table.
1049 <table id="querymodel-bib1-structure-table" frame="top">
1050 <title>Structure Attributes (type 4)</title>
1054 <entry>Structure</entry>
1055 <entry>Value</entry>
1056 <entry>Notes</entry>
1061 <entry>Phrase </entry>
1063 <entry>default</entry>
1068 <entry>supported</entry>
1073 <entry>supported</entry>
1078 <entry>supported</entry>
1081 <entry>Date (normalized)</entry>
1083 <entry>supported</entry>
1086 <entry>Word list</entry>
1088 <entry>supported</entry>
1091 <entry>Date (un-normalized)</entry>
1093 <entry>unsupported</entry>
1096 <entry>Name (normalized) </entry>
1098 <entry>unsupported</entry>
1101 <entry>Name (un-normalized) </entry>
1103 <entry>unsupported</entry>
1106 <entry>Structure</entry>
1108 <entry>unsupported</entry>
1113 <entry>supported</entry>
1116 <entry>Free-form-text</entry>
1118 <entry>supported</entry>
1121 <entry>Document-text</entry>
1123 <entry>supported</entry>
1126 <entry>Local-number</entry>
1128 <entry>supported</entry>
1131 <entry>String</entry>
1133 <entry>unsupported</entry>
1136 <entry>Numeric string</entry>
1138 <entry>supported</entry>
1144 The structure attribute values
1145 <literal>Word list (6)</literal>
1146 is supported, and maps to the boolean <literal>AND</literal>
1147 combination of words supplied. The word list is useful when
1148 Google-like bag-of-word queries need to be translated from a GUI
1149 query language to &acro.pqf;. For example, the following queries
1152 Z> find @attr 1=Title @attr 4=6 "mozart amadeus"
1153 Z> find @attr 1=Title @and mozart amadeus
1158 The structure attribute value
1159 <literal>Free-form-text (105)</literal> and
1160 <literal>Document-text (106)</literal>
1161 are supported, and map both to the boolean <literal>OR</literal>
1162 combination of words supplied. The following queries
1165 Z> find @attr 1=Body-of-text @attr 4=105 "bach salieri teleman"
1166 Z> find @attr 1=Body-of-text @attr 4=106 "bach salieri teleman"
1167 Z> find @attr 1=Body-of-text @or bach @or salieri teleman
1169 This <literal>OR</literal> list of terms is very useful in
1170 combination with relevance ranking:
1172 Z> find @attr 1=Body-of-text @attr 2=102 @attr 4=105 "bach salieri teleman"
1177 The structure attribute value
1178 <literal>Local number (107)</literal>
1179 is supported, and maps always to the &zebra; internal document ID,
1180 irrespectively which use attribute is specified. The following queries
1181 have exactly the same unique record in the hit set:
1183 Z> find @attr 4=107 10
1184 Z> find @attr 1=4 @attr 4=107 10
1185 Z> find @attr 1=1010 @attr 4=107 10
1191 the GILS schema (<literal>gils.abs</literal>), the
1192 west-bounding-coordinate is indexed as type <literal>n</literal>,
1193 and is therefore searched by specifying
1194 <emphasis>structure</emphasis>=<emphasis>Numeric String</emphasis>.
1195 To match all those records with west-bounding-coordinate greater
1196 than -114 we use the following query:
1198 Z> find @attr 4=109 @attr 2=5 @attr gils 1=2038 -114
1203 The exact mapping between &acro.pqf; queries and &zebra; internal indexes
1204 and index types is explained in
1205 <xref linkend="querymodel-pqf-apt-mapping"/>.
1211 <section id="querymodel-bib1-truncation">
1212 <title>Truncation Attributes (type = 5)</title>
1215 The truncation attribute specifies whether variations of one or
1216 more characters are allowed between search term and hit terms, or
1217 not. Using non-default truncation attributes will broaden the
1218 document hit set of a search query.
1221 <table id="querymodel-bib1-truncation-table" frame="top">
1222 <title>Truncation Attributes (type 5)</title>
1226 <entry>Truncation</entry>
1227 <entry>Value</entry>
1228 <entry>Notes</entry>
1233 <entry>Right truncation </entry>
1235 <entry>supported</entry>
1238 <entry>Left truncation</entry>
1240 <entry>supported</entry>
1243 <entry>Left and right truncation</entry>
1245 <entry>supported</entry>
1248 <entry>Do not truncate</entry>
1250 <entry>default</entry>
1253 <entry>Process # in search term</entry>
1255 <entry>supported</entry>
1258 <entry>RegExpr-1 </entry>
1260 <entry>supported</entry>
1263 <entry>RegExpr-2</entry>
1265 <entry>supported</entry>
1272 The truncation attribute values 1-3 perform the obvious way:
1274 Z> scan @attr 1=Body-of-text schnittke
1280 Z> find @attr 1=Body-of-text @attr 5=1 schnittke
1282 Number of hits: 95, setno 7
1284 Z> find @attr 1=Body-of-text @attr 5=2 schnittke
1286 Number of hits: 81, setno 6
1288 Z> find @attr 1=Body-of-text @attr 5=3 schnittke
1290 Number of hits: 95, setno 8
1295 The truncation attribute value
1296 <literal>Process # in search term (101)</literal> is a
1297 poor-man's regular expression search. It maps
1298 each <literal>#</literal> to <literal>.*</literal>, and
1299 performs then a <literal>Regexp-1 (102)</literal> regular
1300 expression search. The following two queries are equivalent:
1302 Z> find @attr 1=Body-of-text @attr 5=101 schnit#ke
1303 Z> find @attr 1=Body-of-text @attr 5=102 schnit.*ke
1305 Number of hits: 89, setno 10
1310 The truncation attribute value
1311 <literal>Regexp-1 (102)</literal> is a normal regular search,
1312 see <xref linkend="querymodel-regular"/> for details.
1314 Z> find @attr 1=Body-of-text @attr 5=102 schnit+ke
1315 Z> find @attr 1=Body-of-text @attr 5=102 schni[a-t]+ke
1320 The truncation attribute value
1321 <literal>Regexp-2 (103) </literal> is a &zebra; specific extension
1322 which allows <emphasis>fuzzy</emphasis> matches. One single
1323 error in spelling of search terms is allowed, i.e., a document
1324 is hit if it includes a term which can be mapped to the used
1325 search term by one character substitution, addition, deletion or
1328 Z> find @attr 1=Body-of-text @attr 5=100 schnittke
1330 Number of hits: 81, setno 14
1332 Z> find @attr 1=Body-of-text @attr 5=103 schnittke
1334 Number of hits: 103, setno 15
1340 <section id="querymodel-bib1-completeness">
1341 <title>Completeness Attributes (type = 6)</title>
1345 The <literal>Completeness Attributes (type = 6)</literal>
1346 is used to specify that a given search term or term list is either
1347 part of the terms of a given index/field
1348 (<literal>Incomplete subfield (1)</literal>), or is
1349 what literally is found in the entire field's index
1350 (<literal>Complete field (3)</literal>).
1353 <table id="querymodel-bib1-completeness-table" frame="top">
1354 <title>Completeness Attributes (type = 6)</title>
1358 <entry>Completeness</entry>
1359 <entry>Value</entry>
1360 <entry>Notes</entry>
1365 <entry>Incomplete subfield</entry>
1367 <entry>default</entry>
1370 <entry>Complete subfield</entry>
1372 <entry>deprecated</entry>
1375 <entry>Complete field</entry>
1377 <entry>supported</entry>
1384 The <literal>Completeness Attributes (type = 6)</literal>
1385 is only partially and conditionally
1386 supported in the sense that it is ignored if the hit index is
1387 not of structure <literal>type="w"</literal> or
1388 <literal>type="p"</literal>.
1391 <literal>Incomplete subfield (1)</literal> is the default, and
1393 register <literal>type="w"</literal>, whereas
1394 <literal>Complete field (3)</literal> triggers
1395 search and scan in index <literal>type="p"</literal>.
1398 The <literal>Complete subfield (2)</literal> is a reminiscent
1399 from the happy &acro.marc;
1400 binary format days. &zebra; does not support it, but maps silently
1401 to <literal>Complete field (3)</literal>.
1406 The exact mapping between &acro.pqf; queries and &zebra; internal indexes
1407 and index types is explained in
1408 <xref linkend="querymodel-pqf-apt-mapping"/>.
1417 <section id="querymodel-zebra">
1418 <title>Extended &zebra; &acro.rpn; Features</title>
1420 The &zebra; internal query engine has been extended to specific needs
1421 not covered by the <literal>bib-1</literal> attribute set query
1422 model. These extensions are <emphasis>non-standard</emphasis>
1423 and <emphasis>non-portable</emphasis>: most functional extensions
1424 are modeled over the <literal>bib-1</literal> attribute set,
1425 defining type 7 and higher values.
1426 There are also the special
1427 <literal>string</literal> type index names for the
1428 <literal>idxpath</literal> attribute set.
1431 <section id="querymodel-zebra-attr-allrecords">
1432 <title>&zebra; specific retrieval of all records</title>
1434 &zebra; defines a hardwired <literal>string</literal> index name
1435 called <literal>_ALLRECORDS</literal>. It matches any record
1436 contained in the database, if used in conjunction with
1437 the relation attribute
1438 <literal>AlwaysMatches (103)</literal>.
1441 The <literal>_ALLRECORDS</literal> index name is used for total database
1442 export. The search term is ignored, it may be empty.
1444 Z> find @attr 1=_ALLRECORDS @attr 2=103 ""
1448 Combination with other index types can be made. For example, to
1449 find all records which are <emphasis>not</emphasis> indexed in
1450 the <literal>Title</literal> register, issue one of the two
1453 Z> find @not @attr 1=_ALLRECORDS @attr 2=103 "" @attr 1=Title @attr 2=103 ""
1454 Z> find @not @attr 1=_ALLRECORDS @attr 2=103 "" @attr 1=4 @attr 2=103 ""
1459 The special string index <literal>_ALLRECORDS</literal> is
1460 experimental, and the provided functionality and syntax may very
1461 well change in future releases of &zebra;.
1466 <section id="querymodel-zebra-attr-search">
1467 <title>&zebra; specific Search Extensions to all Attribute Sets</title>
1469 &zebra; extends the &acro.bib1; attribute types, and these extensions are
1470 recognized regardless of attribute
1471 set used in a <literal>search</literal> operation query.
1474 <table id="querymodel-zebra-attr-search-table" frame="top">
1475 <title>&zebra; Search Attribute Extensions</title>
1480 <entry>Value</entry>
1481 <entry>Operation</entry>
1482 <entry>&zebra; version</entry>
1487 <entry>Embedded Sort</entry>
1489 <entry>search</entry>
1493 <entry>Term Set</entry>
1495 <entry>search</entry>
1499 <entry>Rank Weight</entry>
1501 <entry>search</entry>
1505 <entry>Term Reference</entry>
1507 <entry>search</entry>
1511 <entry>Local Approx Limit</entry>
1513 <entry>search</entry>
1517 <entry>Global Approx Limit</entry>
1519 <entry>search</entry>
1520 <entry>2.0.8</entry>
1523 <entry>Maximum number of truncated terms (truncmax)</entry>
1525 <entry>search</entry>
1526 <entry>2.0.10</entry>
1530 Specifies whether un-indexed fields should be ignored.
1531 A zero value (default) throws a diagnostic when an un-indexed
1532 field is specified. A non-zero value makes it return 0 hits.
1535 <entry>search</entry>
1536 <entry>2.0.16</entry>
1542 <section id="querymodel-zebra-attr-sorting">
1543 <title>&zebra; Extension Embedded Sort Attribute (type 7)</title>
1545 The embedded sort is a way to specify sort within a query - thus
1546 removing the need to send a Sort Request separately. It is both
1547 faster and does not require clients to deal with the Sort
1552 All ordering operations are based on a lexicographical ordering,
1553 <emphasis>except</emphasis> when the
1554 <literal>structure attribute numeric (109)</literal> is used. In
1555 this case, ordering is numerical. See
1556 <xref linkend="querymodel-bib1-structure"/>.
1560 The possible values after attribute <literal>type 7</literal> are
1561 <literal>1</literal> ascending and
1562 <literal>2</literal> descending.
1563 The attributes+term (&acro.apt;) node is separate from the
1564 rest and must be <literal>@or</literal>'ed.
1565 The term associated with &acro.apt; is the sorting level in integers,
1566 where <literal>0</literal> means primary sort,
1567 <literal>1</literal> means secondary sort, and so forth.
1568 See also <xref linkend="administration-ranking"/>.
1571 For example, searching for water, sort by title (ascending)
1573 Z> find @or @attr 1=1016 water @attr 7=1 @attr 1=4 0
1577 Or, searching for water, sort by title ascending, then date descending
1579 Z> find @or @or @attr 1=1016 water @attr 7=1 @attr 1=4 0 @attr 7=2 @attr 1=30 1
1585 &zebra; Extension Term Set Attribute
1586 From the manual text, I can not see what is the point with this feature.
1587 I think it makes more sense when there are multiple terms in a query, or
1590 We decided 2006-06-03 to disable this feature, as it is covered by
1591 scan within a resultset. Better use ressources to upgrade this
1592 feature for good performance.
1596 <section id="querymodel-zebra-attr-estimation">
1597 <title>&zebra; Extension Term Set Attribute (type 8)</title>
1599 The Term Set feature is a facility that allows a search to store
1600 hitting terms in a "pseudo" resultset; thus a search (as usual) +
1601 a scan-like facility. Requires a client that can do named result
1602 sets since the search generates two result sets. The value for
1603 attribute 8 is the name of a result set (string). The terms in
1604 the named term set are returned as &acro.sutrs; records.
1607 For example, searching for u in title, right truncated, and
1608 storing the result in term set named 'aset'
1610 Z> find @attr 5=1 @attr 1=4 @attr 8=aset u
1614 The model has one serious flaw: we don't know the size of term
1615 set. Experimental. Do not use in production code.
1621 <section id="querymodel-zebra-attr-weight">
1622 <title>&zebra; Extension Rank Weight Attribute (type 9)</title>
1624 Rank weight is a way to pass a value to a ranking algorithm - so
1625 that one &acro.apt; has one value - while another as a different one.
1626 See also <xref linkend="administration-ranking"/>.
1629 For example, searching for utah in title with weight 30 as well
1630 as any with weight 20:
1632 Z> find @attr 2=102 @or @attr 9=30 @attr 1=4 utah @attr 9=20 utah
1637 <section id="querymodel-zebra-attr-termref">
1638 <title>&zebra; Extension Term Reference Attribute (type 10)</title>
1640 &zebra; supports the searchResult-1 facility.
1641 If the Term Reference Attribute (type 10) is
1642 given, that specifies a subqueryId value returned as part of the
1643 search result. It is a way for a client to name an &acro.apt; part of a
1649 Experimental. Do not use in production code.
1657 <section id="querymodel-zebra-local-attr-limit">
1658 <title>Local Approximative Limit Attribute (type 11)</title>
1660 &zebra; computes - unless otherwise configured -
1661 the exact hit count for every &acro.apt;
1662 (leaf) in the query tree. These hit counts are returned as part of
1663 the searchResult-1 facility in the binary encoded &acro.z3950; search
1667 By setting an estimation limit size of the resultset of the &acro.apt;
1668 leaves, &zebra; stops processing the result set when the limit
1670 Hit counts under this limit are still precise, but hit counts over it
1671 are estimated using the statistics gathered from the chopped
1675 Specifying a limit of <literal>0</literal> results in exact hit counts.
1678 For example, we might be interested in exact hit count for a, but
1679 for b we allow hit count estimates for 1000 and higher.
1681 Z> find @and a @attr 11=1000 b
1686 The estimated hit count facility makes searches faster, as one
1687 only needs to process large hit lists partially.
1688 It is mostly used in huge databases, where you you want trade
1689 exactness of hit counts against speed of execution.
1694 Do not use approximative hit count limits
1695 in conjunction with relevance ranking, as re-sorting of the
1696 result set only works when the entire result set has
1702 <section id="querymodel-zebra-global-attr-limit">
1703 <title>Global Approximative Limit Attribute (type 12)</title>
1705 By default &zebra; computes precise hit counts for a query as
1706 a whole. Setting attribute 12 makes it perform approximative
1707 hit counts instead. It has the same semantics as
1708 <literal>estimatehits</literal> for the <xref linkend="zebra-cfg"/>.
1711 The attribute (12) can occur anywhere in the query tree.
1712 Unlike regular attributes it does not relate to the leaf (&acro.apt;)
1713 - but to the whole query.
1717 Do not use approximative hit count limits
1718 in conjunction with relevance ranking, as re-sorting of the
1719 result set only works when the entire result set has
1727 <section id="querymodel-zebra-attr-scan">
1728 <title>&zebra; specific Scan Extensions to all Attribute Sets</title>
1730 &zebra; extends the Bib1 attribute types, and these extensions are
1731 recognized regardless of attribute
1732 set used in a scan operation query.
1734 <table id="querymodel-zebra-attr-scan-table" frame="top">
1735 <title>&zebra; Scan Attribute Extensions</title>
1741 <entry>Operation</entry>
1742 <entry>&zebra; version</entry>
1747 <entry>Result Set Narrow</entry>
1753 <entry>Approximative Limit</entry>
1756 <entry>2.0.20</entry>
1762 <section id="querymodel-zebra-attr-narrow">
1763 <title>&zebra; Extension Result Set Narrow (type 8)</title>
1765 If attribute Result Set Narrow (type 8)
1766 is given for scan, the value is the name of a
1767 result set. Each hit count in scan is
1768 <literal>@and</literal>'ed with the result set given.
1771 Consider for example
1772 the case of scanning all title fields around the
1773 scanterm <emphasis>mozart</emphasis>, then refining the scan by
1774 issuing a filtering query for <emphasis>amadeus</emphasis> to
1775 restrict the scan to the result set of the query:
1777 Z> scan @attr 1=4 mozart
1780 mozartforskningen (1)
1784 Z> f @attr 1=4 amadeus
1786 Number of hits: 15, setno 2
1788 Z> scan @attr 1=4 @attr 8=2 mozart
1791 mozartforskningen (0)
1799 &zebra; 2.0.2 and later is able to skip 0 hit counts. This, however,
1800 is known not to scale if the number of terms to skip is high.
1801 This most likely will happen if the result set is small (and
1802 result in many 0 hits).
1806 <section id="querymodel-zebra-attr-approx">
1807 <title>&zebra; Extension Approximative Limit (type 12)</title>
1809 The &zebra; Extension Approximative Limit (type 12) is a way to
1810 enable approximate hit counts for scan hit counts, in the same
1811 way as for search hit counts.
1816 <section id="querymodel-idxpath">
1817 <title>&zebra; special &acro.idxpath; Attribute Set for &acro.grs1; indexing</title>
1819 The attribute-set <literal>idxpath</literal> consists of a single
1820 Use (type 1) attribute. All non-use attributes behave as normal.
1823 This feature is enabled when defining the
1824 <literal>xpath enable</literal> option in the &acro.grs1; filter
1825 <filename>*.abs</filename> configuration files. If one wants to use
1826 the special <literal>idxpath</literal> numeric attribute set, the
1827 main &zebra; configuration file <filename>zebra.cfg</filename>
1828 directive <literal>attset: idxpath.att</literal> must be enabled.
1832 The <literal>idxpath</literal> is deprecated, may not be
1833 supported in future &zebra; versions, and should definitely
1834 not be used in production code.
1838 <section id="querymodel-idxpath-use">
1839 <title>&acro.idxpath; Use Attributes (type = 1)</title>
1841 This attribute set allows one to search &acro.grs1; filter indexed
1842 records by &acro.xpath; like structured index names.
1847 The <literal>idxpath</literal> option defines hard-coded
1848 index names, which might clash with your own index names.
1852 <table id="querymodel-idxpath-use-table" frame="top">
1853 <title>&zebra; specific &acro.idxpath; Use Attributes (type 1)</title>
1857 <entry>&acro.idxpath;</entry>
1858 <entry>Value</entry>
1859 <entry>String Index</entry>
1860 <entry>Notes</entry>
1865 <entry>&acro.xpath; Begin</entry>
1867 <entry>_XPATH_BEGIN</entry>
1868 <entry>deprecated</entry>
1871 <entry>&acro.xpath; End</entry>
1873 <entry>_XPATH_END</entry>
1874 <entry>deprecated</entry>
1877 <entry>&acro.xpath; CData</entry>
1879 <entry>_XPATH_CDATA</entry>
1880 <entry>deprecated</entry>
1883 <entry>&acro.xpath; Attribute Name</entry>
1885 <entry>_XPATH_ATTR_NAME</entry>
1886 <entry>deprecated</entry>
1889 <entry>&acro.xpath; Attribute CData</entry>
1891 <entry>_XPATH_ATTR_CDATA</entry>
1892 <entry>deprecated</entry>
1899 See <filename>tab/idxpath.att</filename> for more information.
1902 Search for all documents starting with root element
1903 <literal>/root</literal> (either using the numeric or the string
1906 Z> find @attrset idxpath @attr 1=1 @attr 4=3 root/
1907 Z> find @attr idxpath 1=1 @attr 4=3 root/
1908 Z> find @attr 1=_XPATH_BEGIN @attr 4=3 root/
1912 Search for all documents where specific nested &acro.xpath;
1913 <literal>/c1/c2/../cn</literal> exists. Notice the very
1914 counter-intuitive <emphasis>reverse</emphasis> notation!
1916 Z> find @attrset idxpath @attr 1=1 @attr 4=3 cn/cn-1/../c1/
1917 Z> find @attr 1=_XPATH_BEGIN @attr 4=3 cn/cn-1/../c1/
1921 Search for CDATA string <emphasis>text</emphasis> in any element
1923 Z> find @attrset idxpath @attr 1=1016 text
1924 Z> find @attr 1=_XPATH_CDATA text
1928 Search for CDATA string <emphasis>anothertext</emphasis> in any
1931 Z> find @attrset idxpath @attr 1=1015 anothertext
1932 Z> find @attr 1=_XPATH_ATTR_CDATA anothertext
1936 Search for all documents with have an &acro.xml; element node
1937 including an &acro.xml; attribute named <emphasis>creator</emphasis>
1939 Z> find @attrset idxpath @attr 1=3 @attr 4=3 creator
1940 Z> find @attr 1=_XPATH_ATTR_NAME @attr 4=3 creator
1944 Combining usual <literal>bib-1</literal> attribute set searches
1945 with <literal>idxpath</literal> attribute set searches:
1947 Z> find @and @attr idxpath 1=1 @attr 4=3 link/ @attr 1=4 mozart
1948 Z> find @and @attr 1=_XPATH_BEGIN @attr 4=3 link/ @attr 1=_XPATH_CDATA mozart
1952 Scanning is supported on all <literal>idxpath</literal>
1953 indexes, both specified as numeric use attributes, or as string
1956 Z> scan @attrset idxpath @attr 1=1016 text
1957 Z> scan @attr 1=_XPATH_ATTR_CDATA anothertext
1958 Z> scan @attrset idxpath @attr 1=3 @attr 4=3 ''
1966 <section id="querymodel-pqf-apt-mapping">
1967 <title>Mapping from &acro.pqf; atomic &acro.apt; queries to &zebra; internal
1968 register indexes</title>
1970 The rules for &acro.pqf; &acro.apt; mapping are rather tricky to grasp in the
1971 first place. We deal first with the rules for deciding which
1972 internal register or string index to use, according to the use
1973 attribute or access point specified in the query. Thereafter we
1974 deal with the rules for determining the correct structure type of
1978 <section id="querymodel-pqf-apt-mapping-accesspoint">
1979 <title>Mapping of &acro.pqf; &acro.apt; access points</title>
1981 &zebra; understands four fundamental different types of access
1982 points, of which only the
1983 <emphasis>numeric use attribute</emphasis> type access points
1984 are defined by the <ulink url="&url.z39.50;">&acro.z3950;</ulink>
1986 All other access point types are &zebra; specific, and non-portable.
1989 <table id="querymodel-zebra-mapping-accesspoint-types" frame="top">
1990 <title>Access point name mapping</title>
1994 <entry>Access Point</entry>
1996 <entry>Grammar</entry>
1997 <entry>Notes</entry>
2002 <entry>Use attribute</entry>
2003 <entry>numeric</entry>
2004 <entry>[1-9][1-9]*</entry>
2005 <entry>directly mapped to string index name</entry>
2008 <entry>String index name</entry>
2009 <entry>string</entry>
2010 <entry>[a-zA-Z](\-?[a-zA-Z0-9])*</entry>
2011 <entry>normalized name is used as internal string index name</entry>
2014 <entry>&zebra; internal index name</entry>
2015 <entry>zebra</entry>
2016 <entry>_[a-zA-Z](_?[a-zA-Z0-9])*</entry>
2017 <entry>hardwired internal string index name</entry>
2020 <entry>&acro.xpath; special index</entry>
2021 <entry>XPath</entry>
2023 <entry>special xpath search for &acro.grs1; indexed records</entry>
2030 <literal>Attribute set names</literal> and
2031 <literal>string index names</literal> are normalizes
2032 according to the following rules: all <emphasis>single</emphasis>
2033 hyphens <literal>'-'</literal> are stripped, and all upper case
2034 letters are folded to lower case.
2038 <emphasis>Numeric use attributes</emphasis> are mapped
2039 to the &zebra; internal
2040 string index according to the attribute set definition in use.
2041 The default attribute set is &acro.bib1;, and may be
2042 omitted in the &acro.pqf; query.
2046 According to normalization and numeric
2047 use attribute mapping, it follows that the following
2048 &acro.pqf; queries are considered equivalent (assuming the default
2049 configuration has not been altered):
2051 Z> find @attr 1=Body-of-text serenade
2052 Z> find @attr 1=bodyoftext serenade
2053 Z> find @attr 1=BodyOfText serenade
2054 Z> find @attr 1=bO-d-Y-of-tE-x-t serenade
2055 Z> find @attr 1=1010 serenade
2056 Z> find @attrset bib1 @attr 1=1010 serenade
2057 Z> find @attrset bib1 @attr 1=1010 serenade
2058 Z> find @attrset Bib1 @attr 1=1010 serenade
2059 Z> find @attrset b-I-b-1 @attr 1=1010 serenade
2064 The <emphasis>numerical</emphasis>
2065 <literal>use attributes (type 1)</literal>
2066 are interpreted according to the
2067 attribute sets which have been loaded in the
2068 <literal>zebra.cfg</literal> file, and are matched against specific
2069 fields as specified in the <literal>.abs</literal> file which
2070 describes the profile of the records which have been loaded.
2071 If no use attribute is provided, a default of
2072 &acro.bib1; Use Any (1016) is assumed.
2073 The predefined use attribute sets
2074 can be reconfigured by tweaking the configuration files
2075 <filename>tab/*.att</filename>, and
2076 new attribute sets can be defined by adding similar files in the
2077 configuration path <literal>profilePath</literal> of the server.
2081 String indexes can be accessed directly,
2082 independently which attribute set is in use. These are just
2083 ignored. The above mentioned name normalization applies.
2084 String index names are defined in the
2085 used indexing filter configuration files, for example in the
2087 <filename>*.abs</filename> configuration files, or in the
2088 <literal>alvis</literal> filter &acro.xslt; indexing stylesheets.
2092 &zebra; internal indexes can be accessed directly,
2093 according to the same rules as the user defined
2094 string indexes. The only difference is that
2095 &zebra; internal index names are hardwired,
2097 must start with the character <literal>'_'</literal>.
2101 Finally, &acro.xpath; access points are only
2102 available using the &acro.grs1; filter for indexing.
2103 These access point names must start with the character
2104 <literal>'/'</literal>, they are <emphasis>not
2105 normalized</emphasis>, but passed unaltered to the &zebra; internal
2106 &acro.xpath; engine. See <xref linkend="querymodel-use-xpath"/>.
2114 <section id="querymodel-pqf-apt-mapping-structuretype">
2115 <title>Mapping of &acro.pqf; &acro.apt; structure and completeness to
2116 register type</title>
2118 Internally &zebra; has in its default configuration several
2119 different types of registers or indexes, whose tokenization and
2120 character normalization rules differ. This reflects the fact that
2121 searching fundamental different tokens like dates, numbers,
2122 bitfields and string based text needs different rule sets.
2125 <table id="querymodel-zebra-mapping-structure-types" frame="top">
2126 <title>Structure and completeness mapping to register types</title>
2130 <entry>Structure</entry>
2131 <entry>Completeness</entry>
2132 <entry>Register type</entry>
2133 <entry>Notes</entry>
2139 phrase (@attr 4=1), word (@attr 4=2),
2140 word-list (@attr 4=6),
2141 free-form-text (@attr 4=105), or document-text (@attr 4=106)
2143 <entry>Incomplete field (@attr 6=1)</entry>
2144 <entry>Word ('w')</entry>
2145 <entry>Traditional tokenized and character normalized word index</entry>
2149 phrase (@attr 4=1), word (@attr 4=2),
2150 word-list (@attr 4=6),
2151 free-form-text (@attr 4=105), or document-text (@attr 4=106)
2153 <entry>complete field' (@attr 6=3)</entry>
2154 <entry>Phrase ('p')</entry>
2155 <entry>Character normalized, but not tokenized index for phrase
2160 <entry>urx (@attr 4=104)</entry>
2161 <entry>ignored</entry>
2162 <entry>URX/URL ('u')</entry>
2163 <entry>Special index for URL web addresses</entry>
2166 <entry>numeric (@attr 4=109)</entry>
2167 <entry>ignored</entry>
2168 <entry>Numeric ('n')</entry>
2169 <entry>Special index for digital numbers</entry>
2172 <entry>key (@attr 4=3)</entry>
2173 <entry>ignored</entry>
2174 <entry>Null bitmap ('0')</entry>
2175 <entry>Used for non-tokenized and non-normalized bit sequences</entry>
2178 <entry>year (@attr 4=4)</entry>
2179 <entry>ignored</entry>
2180 <entry>Year ('y')</entry>
2181 <entry>Non-tokenized and non-normalized 4 digit numbers</entry>
2184 <entry>date (@attr 4=5)</entry>
2185 <entry>ignored</entry>
2186 <entry>Date ('d')</entry>
2187 <entry>Non-tokenized and non-normalized ISO date strings</entry>
2190 <entry>ignored</entry>
2191 <entry>ignored</entry>
2192 <entry>Sort ('s')</entry>
2193 <entry>Used with special sort attribute set (@attr 7=1, @attr 7=2)</entry>
2196 <entry>overruled</entry>
2197 <entry>overruled</entry>
2198 <entry>special</entry>
2199 <entry>Internal record ID register, used whenever
2200 Relation Always Matches (@attr 2=103) is specified</entry>
2206 <!-- see in util/zebramap.c -->
2209 If a <emphasis>Structure</emphasis> attribute of
2210 <emphasis>Phrase</emphasis> is used in conjunction with a
2211 <emphasis>Completeness</emphasis> attribute of
2212 <emphasis>Complete (Sub)field</emphasis>, the term is matched
2213 against the contents of the phrase (long word) register, if one
2214 exists for the given <emphasis>Use</emphasis> attribute.
2215 A phrase register is created for those fields in the
2216 &acro.grs1; <filename>*.abs</filename> file that contains a
2217 <literal>p</literal>-specifier.
2219 Z> scan @attr 1=Title @attr 4=1 @attr 6=3 beethoven
2221 bayreuther festspiele (1)
2222 * beethoven bibliography database (1)
2225 Z> find @attr 1=Title @attr 4=1 @attr 6=3 "beethoven bibliography"
2227 Number of hits: 0, setno 5
2229 Z> find @attr 1=Title @attr 4=1 @attr 6=3 "beethoven bibliography database"
2231 Number of hits: 1, setno 6
2236 If <emphasis>Structure</emphasis>=<emphasis>Phrase</emphasis> is
2237 used in conjunction with <emphasis>Incomplete Field</emphasis> - the
2238 default value for <emphasis>Completeness</emphasis>, the
2239 search is directed against the normal word registers, but if the term
2240 contains multiple words, the term will only match if all of the words
2241 are found immediately adjacent, and in the given order.
2242 The word search is performed on those fields that are indexed as
2243 type <literal>w</literal> in the &acro.grs1; <filename>*.abs</filename> file.
2245 Z> scan @attr 1=Title @attr 4=1 @attr 6=1 beethoven
2251 Z> find @attr 1=Title @attr 4=1 @attr 6=1 beethoven
2253 Number of hits: 18, setno 1
2255 Z> find @attr 1=Title @attr 4=1 @attr 6=1 "beethoven bibliography"
2257 Number of hits: 2, setno 2
2263 If the <emphasis>Structure</emphasis> attribute is
2264 <emphasis>Word List</emphasis>,
2265 <emphasis>Free-form Text</emphasis>, or
2266 <emphasis>Document Text</emphasis>, the term is treated as a
2267 natural-language, relevance-ranked query.
2268 This search type uses the word register, i.e. those fields
2269 that are indexed as type <literal>w</literal> in the
2270 &acro.grs1; <filename>*.abs</filename> file.
2274 If the <emphasis>Structure</emphasis> attribute is
2275 <emphasis>Numeric String</emphasis> the term is treated as an integer.
2276 The search is performed on those fields that are indexed
2277 as type <literal>n</literal> in the &acro.grs1;
2278 <filename>*.abs</filename> file.
2282 If the <emphasis>Structure</emphasis> attribute is
2283 <emphasis>URX</emphasis> the term is treated as a URX (URL) entity.
2284 The search is performed on those fields that are indexed as type
2285 <literal>u</literal> in the <filename>*.abs</filename> file.
2289 If the <emphasis>Structure</emphasis> attribute is
2290 <emphasis>Local Number</emphasis> the term is treated as
2291 native &zebra; Record Identifier.
2295 If the <emphasis>Relation</emphasis> attribute is
2296 <emphasis>Equals</emphasis> (default), the term is matched
2297 in a normal fashion (modulo truncation and processing of
2298 individual words, if required).
2299 If <emphasis>Relation</emphasis> is <emphasis>Less Than</emphasis>,
2300 <emphasis>Less Than or Equal</emphasis>,
2301 <emphasis>Greater than</emphasis>, or <emphasis>Greater than or
2302 Equal</emphasis>, the term is assumed to be numerical, and a
2303 standard regular expression is constructed to match the given
2305 If <emphasis>Relation</emphasis> is <emphasis>Relevance</emphasis>,
2306 the standard natural-language query processor is invoked.
2310 For the <emphasis>Truncation</emphasis> attribute,
2311 <emphasis>No Truncation</emphasis> is the default.
2312 <emphasis>Left Truncation</emphasis> is not supported.
2313 <emphasis>Process # in search term</emphasis> is supported, as is
2314 <emphasis>Regxp-1</emphasis>.
2315 <emphasis>Regxp-2</emphasis> enables the fault-tolerant (fuzzy)
2316 search. As a default, a single error (deletion, insertion,
2317 replacement) is accepted when terms are matched against the register
2324 <section id="querymodel-regular">
2325 <title>&zebra; Regular Expressions in Truncation Attribute (type = 5)</title>
2328 Each term in a query is interpreted as a regular expression if
2329 the truncation value is either <emphasis>Regxp-1 (@attr 5=102)</emphasis>
2330 or <emphasis>Regxp-2 (@attr 5=103)</emphasis>.
2331 Both query types follow the same syntax with the operands:
2334 <table id="querymodel-regular-operands-table" frame="top">
2335 <title>Regular Expression Operands</title>
2339 <entry><literal>x</literal></entry>
2340 <entry>Matches the character <literal>x</literal>.</entry>
2343 <entry><literal>.</literal></entry>
2344 <entry>Matches any character.</entry>
2347 <entry><literal>[ .. ]</literal></entry>
2348 <entry>Matches the set of characters specified;
2349 such as <literal>[abc]</literal> or <literal>[a-c]</literal>.</entry>
2356 The above operands can be combined with the following operators:
2359 <table id="querymodel-regular-operators-table" frame="top">
2360 <title>Regular Expression Operators</title>
2364 <entry><literal>x*</literal></entry>
2365 <entry>Matches <literal>x</literal> zero or more times.
2366 Priority: high.</entry>
2369 <entry><literal>x+</literal></entry>
2370 <entry>Matches <literal>x</literal> one or more times.
2371 Priority: high.</entry>
2374 <entry><literal>x?</literal></entry>
2375 <entry> Matches <literal>x</literal> zero or once.
2376 Priority: high.</entry>
2379 <entry><literal>xy</literal></entry>
2380 <entry> Matches <literal>x</literal>, then <literal>y</literal>.
2381 Priority: medium.</entry>
2384 <entry><literal>x|y</literal></entry>
2385 <entry> Matches either <literal>x</literal> or <literal>y</literal>.
2386 Priority: low.</entry>
2389 <entry><literal>( )</literal></entry>
2390 <entry>The order of evaluation may be changed by using parentheses.</entry>
2397 If the first character of the <literal>Regxp-2</literal> query
2398 is a plus character (<literal>+</literal>) it marks the
2399 beginning of a section with non-standard specifiers.
2400 The next plus character marks the end of the section.
2401 Currently &zebra; only supports one specifier, the error tolerance,
2402 which consists one digit.
2403 <!-- TODO Nice thing, but what does
2404 that error tolerance digit *mean*? Maybe an example would be nice? -->
2408 Since the plus operator is normally a suffix operator the addition to
2409 the query syntax doesn't violate the syntax for standard regular
2414 For example, a phrase search with regular expressions in
2415 the title-register is performed like this:
2417 Z> find @attr 1=4 @attr 5=102 "informat.* retrieval"
2422 Combinations with other attributes are possible. For example, a
2423 ranked search with a regular expression:
2425 Z> find @attr 1=4 @attr 5=102 @attr 2=102 "informat.* retrieval"
2433 The RecordType parameter in the <literal>zebra.cfg</literal> file, or
2434 the <literal>-t</literal> option to the indexer tells &zebra; how to
2435 process input records.
2436 Two basic types of processing are available - raw text and structured
2437 data. Raw text is just that, and it is selected by providing the
2438 argument <literal>text</literal> to &zebra;. Structured records are
2439 all handled internally using the basic mechanisms described in the
2440 subsequent sections.
2441 &zebra; can read structured records in many different formats.
2447 <section id="querymodel-cql-to-pqf">
2448 <title>Server Side &acro.cql; to &acro.pqf; Query Translation</title>
2451 <literal><cql2rpn>l2rpn.txt</cql2rpn></literal>
2452 &yaz; Frontend Virtual
2453 Hosts option, one can configure
2454 the &yaz; Frontend &acro.cql;-to-&acro.pqf;
2455 converter, specifying the interpretation of various
2456 <ulink url="&url.cql;">&acro.cql;</ulink>
2457 indexes, relations, etc. in terms of Type-1 query attributes.
2458 <!-- The yaz-client config file -->
2461 For example, using server-side &acro.cql;-to-&acro.pqf; conversion, one might
2462 query a zebra server like this:
2465 yaz-client localhost:9999
2467 Z> find text=(plant and soil)
2470 and - if properly configured - even static relevance ranking can
2471 be performed using &acro.cql; query syntax:
2474 Z> find text = /relevant (plant and soil)
2480 By the way, the same configuration can be used to
2481 search using client-side &acro.cql;-to-&acro.pqf; conversion:
2482 (the only difference is <literal>querytype cql2rpn</literal>
2484 <literal>querytype cql</literal>, and the call specifying a local
2488 yaz-client -q local/cql2pqf.txt localhost:9999
2489 Z> querytype cql2rpn
2490 Z> find text=(plant and soil)
2496 Exhaustive information can be found in the
2497 Section <ulink url="&url.yaz.cql2pqf;">&acro.cql; to &acro.rpn; conversion</ulink>
2498 in the &yaz; manual.
2503 <ulink url="http://www.loc.gov/z3950/agency/zing/cql/dc-indexes.html"/>
2504 for the Maintenance Agency's work-in-progress mapping of Dublin Core
2505 indexes to Attribute Architecture (util, XD and BIB-2)
2513 <!-- Keep this comment at the end of the file
2518 sgml-minimize-attributes:nil
2519 sgml-always-quote-attributes:t
2522 sgml-parent-document: "idzebra.xml"
2523 sgml-local-catalogs: nil
2524 sgml-namecase-general:t