1 <chapter id="tools"><title>Supporting Tools</title>
4 In support of the service API - primarily the ASN module, which
5 provides the pro-grammatic interface to the Z39.50 APDUs, &yaz; contains
6 a collection of tools that support the development of applications.
9 <sect1 id="tools.query"><title>Query Syntax Parsers</title>
12 Since the type-1 (RPN) query structure has no direct, useful string
13 representation, every origin application needs to provide some form of
14 mapping from a local query notation or representation to a
15 <token>Z_RPNQuery</token> structure. Some programmers will prefer to
16 construct the query manually, perhaps using
17 <function>odr_malloc()</function> to simplify memory management.
18 The &yaz; distribution includes three separate, query-generating tools
19 that may be of use to you.
22 <sect2 id="PQF"><title>Prefix Query Format</title>
25 Since RPN or reverse polish notation is really just a fancy way of
26 describing a suffix notation format (operator follows operands), it
27 would seem that the confusion is total when we now introduce a prefix
28 notation for RPN. The reason is one of simple laziness - it's somewhat
29 simpler to interpret a prefix format, and this utility was designed
30 for maximum simplicity, to provide a baseline representation for use
31 in simple test applications and scripting environments (like Tcl). The
32 demonstration client included with YAZ uses the PQF.
37 The PQF have been adopted by other parties developing Z39.50
38 software. It is often referred to as Prefix Query Notation
43 The PQF is defined by the pquery module in the YAZ library.
44 There are two sets of function that have similar behavior. First
45 set operates on a PQF parser handle, second set doesn't. First set
46 set of functions are more flexible than the second set. Second set
47 is obsolete and is only provided to ensure backwards compatibility.
50 First set of functions all operate on a PQF parser handle:
53 #include <yaz/pquery.h>
55 YAZ_PQF_Parser yaz_pqf_create(void);
57 void yaz_pqf_destroy(YAZ_PQF_Parser p);
59 Z_RPNQuery *yaz_pqf_parse(YAZ_PQF_Parser p, ODR o, const char *qbuf);
61 Z_AttributesPlusTerm *yaz_pqf_scan(YAZ_PQF_Parser p, ODR o,
62 Odr_oid **attributeSetId, const char *qbuf);
64 int yaz_pqf_error(YAZ_PQF_Parser p, const char **msg, size_t *off);
67 A PQF parser is created and destructed by functions
68 <function>yaz_pqf_create</function> and
69 <function>yaz_pqf_destroy</function> respectively.
70 Function <function>yaz_pqf_parse</function> parses query given
71 by string <literal>qbuf</literal>. If parsing was successful,
72 a Z39.50 RPN Query is returned which is created using ODR stream
73 <literal>o</literal>. If parsing failed, a NULL pointer is
75 Function <function>yaz_pqf_scan</function> takes a scan query in
76 <literal>qbuf</literal>. If parsing was successful, the function
77 returns attributes plus term pointer and modifies
78 <literal>attributeSetId</literal> to hold attribute set for the
79 scan request - both allocated using ODR stream <literal>o</literal>.
80 If parsing failed, yaz_pqf_scan returns a NULL pointer.
81 Error information for bad queries can be obtained by a call to
82 <function>yaz_pqf_error</function> which returns an error code and
83 modifies <literal>*msg</literal> to point to an error description,
84 and modifies <literal>*off</literal> to the offset within last
85 query were parsing failed.
88 The second set of functions are declared as follows:
91 #include <yaz/pquery.h>
93 Z_RPNQuery *p_query_rpn(ODR o, oid_proto proto, const char *qbuf);
95 Z_AttributesPlusTerm *p_query_scan(ODR o, oid_proto proto,
96 Odr_oid **attributeSetP, const char *qbuf);
98 int p_query_attset(const char *arg);
101 The function <function>p_query_rpn()</function> takes as arguments an
102 &odr; stream (see section <link linkend="odr">The ODR Module</link>)
103 to provide a memory source (the structure created is released on
104 the next call to <function>odr_reset()</function> on the stream), a
105 protocol identifier (one of the constants <token>PROTO_Z3950</token> and
106 <token>PROTO_SR</token>), an attribute set reference, and
107 finally a null-terminated string holding the query string.
110 If the parse went well, <function>p_query_rpn()</function> returns a
111 pointer to a <literal>Z_RPNQuery</literal> structure which can be
112 placed directly into a <literal>Z_SearchRequest</literal>.
113 If parsing failed, due to syntax error, a NULL pointer is returned.
116 The <literal>p_query_attset</literal> specifies which attribute set
117 to use if the query doesn't specify one by the
118 <literal>@attrset</literal> operator.
119 The <literal>p_query_attset</literal> returns 0 if the argument is a
120 valid attribute set specifier; otherwise the function returns -1.
124 The grammar of the PQF is as follows:
128 query ::= top-set query-struct.
130 top-set ::= [ '@attrset' string ]
132 query-struct ::= attr-spec | simple | complex | '@term' term-type query
134 attr-spec ::= '@attr' [ string ] string query-struct
136 complex ::= operator query-struct query-struct.
138 operator ::= '@and' | '@or' | '@not' | '@prox' proximity.
140 simple ::= result-set | term.
142 result-set ::= '@set' string.
146 proximity ::= exclusion distance ordered relation which-code unit-code.
148 exclusion ::= '1' | '0' | 'void'.
150 distance ::= integer.
152 ordered ::= '1' | '0'.
154 relation ::= integer.
156 which-code ::= 'known' | 'private' | integer.
158 unit-code ::= integer.
160 term-type ::= 'general' | 'numeric' | 'string' | 'oid' | 'datetime' | 'null'.
164 You will note that the syntax above is a fairly faithful
165 representation of RPN, except for the Attribute, which has been
166 moved a step away from the term, allowing you to associate one or more
167 attributes with an entire query structure. The parser will
168 automatically apply the given attributes to each term as required.
172 The @attr operator is followed by an attribute specification
173 (<literal>attr-spec</literal> above). The specification consists
174 of an optional attribute set, an attribute type-value pair and
175 a sub-query. The attribute type-value pair is packed in one string:
176 an attribute type, an equals sign, and an attribute value, like this:
177 <literal>@attr 1=1003</literal>.
178 The type is always an integer but the value may be either an
179 integer or a string (if it doesn't start with a digit character).
180 A string attribute-value is encoded as a Type-1 ``complex''
181 attribute with the list of values containing the single string
182 specified, and including no semantic indicators.
186 Version 3 of the Z39.50 specification defines various encoding of terms.
187 Use <literal>@term </literal> <replaceable>type</replaceable>
188 <replaceable>string</replaceable>,
189 where type is one of: <literal>general</literal>,
190 <literal>numeric</literal> or <literal>string</literal>
191 (for InternationalString).
192 If no term type has been given, the <literal>general</literal> form
193 is used. This is the only encoding allowed in both versions 2 and 3
194 of the Z39.50 standard.
197 <sect3 id="PQF-prox">
198 <title>Using Proximity Operators with PQF</title>
201 This is an advanced topic, describing how to construct
202 queries that make very specific requirements on the
203 relative location of their operands.
204 You may wish to skip this section and go straight to
205 <link linkend="pqf-examples">the example PQF queries</link>.
210 Most Z39.50 servers do not support proximity searching, or
211 support only a small subset of the full functionality that
212 can be expressed using the PQF proximity operator. Be
213 aware that the ability to <emphasis>express</emphasis> a
214 query in PQF is no guarantee that any given server will
215 be able to <emphasis>execute</emphasis> it.
221 The proximity operator <literal>@prox</literal> is a special
222 and more restrictive version of the conjunction operator
223 <literal>@and</literal>. Its semantics are described in
224 section 3.7.2 (Proximity) of Z39.50 the standard itself, which
225 can be read on-line at
226 <ulink url="&url.z39.50.proximity;"/>
229 In PQF, the proximity operation is represented by a sequence
232 @prox <replaceable>exclusion</replaceable> <replaceable>distance</replaceable> <replaceable>ordered</replaceable> <replaceable>relation</replaceable> <replaceable>which-code</replaceable> <replaceable>unit-code</replaceable>
234 in which the meanings of the parameters are as described in in
235 the standard, and they can take the following values:
237 <listitem><formalpara><title>exclusion</title><para>
238 0 = false (i.e. the proximity condition specified by the
239 remaining parameters must be satisfied) or
240 1 = true (the proximity condition specified by the
241 remaining parameters must <emphasis>not</emphasis> be
243 </para></formalpara></listitem>
244 <listitem><formalpara><title>distance</title><para>
245 An integer specifying the difference between the locations
246 of the operands: e.g. two adjacent words would have
247 distance=1 since their locations differ by one unit.
248 </para></formalpara></listitem>
249 <listitem><formalpara><title>ordered</title><para>
250 1 = ordered (the operands must occur in the order the
251 query specifies them) or
252 0 = unordered (they may appear in either order).
253 </para></formalpara></listitem>
254 <listitem><formalpara><title>relation</title><para>
255 Recognised values are
259 4 (greaterThanOrEqual),
262 </para></formalpara></listitem>
263 <listitem><formalpara><title>which-code</title><para>
264 <literal>known</literal>
267 (the unit-code parameter is taken from the well-known list
268 of alternatives described in below) or
269 <literal>private</literal>
272 (the unit-code paramater has semantics specific to an
273 out-of-band agreement such as a profile).
274 </para></formalpara></listitem>
275 <listitem><formalpara><title>unit-code</title><para>
276 If the which-code parameter is <literal>known</literal>
277 then the recognised values are
289 If which-code is <literal>private</literal> then the
290 acceptable values are determined by the profile.
291 </para></formalpara></listitem>
293 (The numeric values of the relation and well-known unit-code
294 parameters are taken straight from
295 <ulink url="&url.z39.50.proximity.asn1;"
296 >the ASN.1</ulink> of the proximity structure in the standard.)
300 <sect3 id="pqf-examples"><title>PQF queries</title>
302 <example id="example.pqf.simple.terms">
303 <title>PQF queries using simple terms</title>
312 <example id="pqf.example.pqf.boolean.operators">
313 <title>PQF boolean operators</title>
316 @or "dylan" "zimmerman"
318 @and @or dylan zimmerman when
320 @and when @or dylan zimmerman
324 <example id="example.pqf.result.sets">
325 <title>PQF references to result sets</title>
330 @and @set seta @set setb
334 <example id="example.pqf.attributes">
335 <title>Attributes for terms</title>
340 @attr 1=4 @attr 4=1 "self portrait"
342 @attrset exp1 @attr 1=1 CategoryList
344 @attr gils 1=2008 Copenhagen
346 @attr 1=/book/title computer
350 <example id="example.pqf.proximity">
351 <title>PQF Proximity queries</title>
354 @prox 0 3 1 2 k 2 dylan zimmerman
357 Here the parameters 0, 3, 1, 2, k and 2 represent exclusion,
358 distance, ordered, relation, which-code and unit-code, in that
362 exclusion = 0: the proximity condition must hold
365 distance = 3: the terms must be three units apart
368 ordered = 1: they must occur in the order they are specified
371 relation = 2: lessThanOrEqual (to the distance of 3 units)
374 which-code is ``known'', so the standard unit-codes are used
380 So the whole proximity query means that the words
381 <literal>dylan</literal> and <literal>zimmerman</literal> must
382 both occur in the record, in that order, differing in position
383 by three or fewer words (i.e. with two or fewer words between
384 them.) The query would find ``Bob Dylan, aka. Robert
385 Zimmerman'', but not ``Bob Dylan, born as Robert Zimmerman''
386 since the distance in this case is four.
390 <example id="example.pqf.search.term.type">
391 <title>PQF specification of search term type</title>
394 @term string "a UTF-8 string, maybe?"
398 <example id="example.pqf.mixed.queries">
399 <title>PQF mixed queries</title>
402 @or @and bob dylan @set Result-1
404 @attr 4=1 @and @attr 1=1 "bob dylan" @attr 1=4 "slow train coming"
406 @and @attr 2=4 @attr gils 1=2038 -114 @attr 2=2 @attr gils 1=2039 -109
410 The last of these examples is a spatial search: in
411 <ulink url="http://www.gils.net/prof_v2.html#sec_7_4"
412 >the GILS attribute set</ulink>,
414 2038 indicates West Bounding Coordinate and
415 2030 indicates East Bounding Coordinate,
416 so the query is for areas extending from -114 degrees
417 to no more than -109 degrees.
424 <sect2 id="CCL"><title>CCL</title>
427 Not all users enjoy typing in prefix query structures and numerical
428 attribute values, even in a minimalistic test client. In the library
429 world, the more intuitive Common Command Language - CCL (ISO 8777)
430 has enjoyed some popularity - especially before the widespread
431 availability of graphical interfaces. It is still useful in
432 applications where you for some reason or other need to provide a
433 symbolic language for expressing boolean query structures.
436 <sect3 id="ccl.syntax">
437 <title>CCL Syntax</title>
440 The CCL parser obeys the following grammar for the FIND argument.
441 The syntax is annotated by in the lines prefixed by
442 <literal>--</literal>.
446 CCL-Find ::= CCL-Find Op Elements
449 Op ::= "and" | "or" | "not"
450 -- The above means that Elements are separated by boolean operators.
452 Elements ::= '(' CCL-Find ')'
455 | Qualifiers Relation Terms
456 | Qualifiers Relation '(' CCL-Find ')'
457 | Qualifiers '=' string '-' string
458 -- Elements is either a recursive definition, a result set reference, a
459 -- list of terms, qualifiers followed by terms, qualifiers followed
460 -- by a recursive definition or qualifiers in a range (lower - upper).
462 Set ::= 'set' = string
463 -- Reference to a result set
465 Terms ::= Terms Prox Term
467 -- Proximity of terms.
471 -- This basically means that a term may include a blank
473 Qualifiers ::= Qualifiers ',' string
475 -- Qualifiers is a list of strings separated by comma
477 Relation ::= '=' | '>=' | '<=' | '<>' | '>' | '<'
478 -- Relational operators. This really doesn't follow the ISO8777
482 -- Proximity operator
486 <example id="example.ccl.queries">
487 <title>CCL queries</title>
489 The following queries are all valid:
501 (dylan and bob) or set=1
511 Assuming that the qualifiers <literal>ti</literal>,
512 <literal>au</literal>
513 and <literal>date</literal> are defined we may use:
519 au=(bob dylan and slow train coming)
521 date>1980 and (ti=((self portrait)))
527 <sect3 id="ccl.qualifiers">
528 <title>CCL Qualifiers</title>
531 Qualifiers are used to direct the search to a particular searchable
532 index, such as title (ti) and author indexes (au). The CCL standard
533 itself doesn't specify a particular set of qualifiers, but it does
534 suggest a few short-hand notations. You can customize the CCL parser
535 to support a particular set of qualifiers to reflect the current target
536 profile. Traditionally, a qualifier would map to a particular
537 use-attribute within the BIB-1 attribute set. It is also
538 possible to set other attributes, such as the structure
543 A CCL profile is a set of predefined CCL qualifiers that may be
544 read from a file or set in the CCL API.
545 The YAZ client reads its CCL qualifiers from a file named
546 <filename>default.bib</filename>. There are four types of
547 lines in a CCL profile: qualifier specification,
548 qualifier alias, comments and directives.
550 <sect4 id="ccl.qualifier.specification">
551 <title>Qualifier specification</title>
553 A qualifier specification is of the form:
557 <replaceable>qualifier-name</replaceable>
558 [<replaceable>attributeset</replaceable><literal>,</literal>]<replaceable>type</replaceable><literal>=</literal><replaceable>val</replaceable>
559 [<replaceable>attributeset</replaceable><literal>,</literal>]<replaceable>type</replaceable><literal>=</literal><replaceable>val</replaceable> ...
563 where <replaceable>qualifier-name</replaceable> is the name of the
564 qualifier to be used (eg. <literal>ti</literal>),
565 <replaceable>type</replaceable> is attribute type in the attribute
566 set (Bib-1 is used if no attribute set is given) and
567 <replaceable>val</replaceable> is attribute value.
568 The <replaceable>type</replaceable> can be specified as an
569 integer or as it be specified either as a single-letter:
570 <literal>u</literal> for use,
571 <literal>r</literal> for relation,<literal>p</literal> for position,
572 <literal>s</literal> for structure,<literal>t</literal> for truncation
573 or <literal>c</literal> for completeness.
574 The attributes for the special qualifier name <literal>term</literal>
575 are used when no CCL qualifier is given in a query.
576 <table id="ccl.common.bib1.attributes">
577 <title>Common Bib-1 attributes</title>
579 <colspec colwidth="2*" colname="type"></colspec>
580 <colspec colwidth="9*" colname="description"></colspec>
584 <entry>Description</entry>
589 <entry><literal>u=</literal><replaceable>value</replaceable></entry>
591 Use attribute (1). Common use attributes are
592 1 Personal-name, 4 Title, 7 ISBN, 8 ISSN, 30 Date,
593 62 Subject, 1003 Author), 1016 Any. Specify value
599 <entry><literal>r=</literal><replaceable>value</replaceable></entry>
601 Relation attribute (2). Common values are
602 1 <, 2 <=, 3 =, 4 >=, 5 >, 6 <>,
603 100 phonetic, 101 stem, 102 relevance, 103 always matches.
608 <entry><literal>p=</literal><replaceable>value</replaceable></entry>
610 Position attribute (3). Values: 1 first in field, 2
611 first in any subfield, 3 any position in field.
616 <entry><literal>s=</literal><replaceable>value</replaceable></entry>
618 Structure attribute (4). Values: 1 phrase, 2 word,
619 3 key, 4 year, 5 date, 6 word list, 100 date (un),
620 101 name (norm), 102 name (un), 103 structure, 104 urx,
621 105 free-form-text, 106 document-text, 107 local-number,
622 108 string, 109 numeric string.
627 <entry><literal>t=</literal><replaceable>value</replaceable></entry>
629 Truncation attribute (5). Values: 1 right, 2 left,
630 3 left& right, 100 none, 101 process #, 102 regular-1,
631 103 regular-2, 104 CCL.
636 <entry><literal>c=</literal><replaceable>value</replaceable></entry>
638 Completeness attribute (6). Values: 1 incomplete subfield,
639 2 complete subfield, 3 complete field.
648 Refer to <xref linkend="bib1"/> or the complete
649 <ulink url="&url.z39.50.attset.bib1;">list of Bib-1 attributes</ulink>
652 It is also possible to specify non-numeric attribute values,
653 which are used in combination with certain types.
654 The special combinations are:
656 <table id="ccl.special.attribute.combos">
657 <title>Special attribute combos</title>
659 <colspec colwidth="2*" colname="name"></colspec>
660 <colspec colwidth="9*" colname="description"></colspec>
664 <entry>Description</entry>
669 <entry><literal>s=pw</literal></entry><entry>
670 The structure is set to either word or phrase depending
671 on the number of tokens in a term (phrase-word).
675 <entry><literal>s=al</literal></entry><entry>
676 Each token in the term is ANDed. (and-list).
677 This does not set the structure at all.
681 <row><entry><literal>s=ol</literal></entry><entry>
682 Each token in the term is ORed. (or-list).
683 This does not set the structure at all.
687 <row><entry><literal>s=ag</literal></entry><entry>
688 Tokens that appears as phrases (with blank in them) gets
689 structure phrase attached (4=1). Tokens that appear to be words
690 gets structure word attached (4=2). Phrases and words are
691 ANDed. This is a variant of s=al and s=pw, with the main
692 difference that words are not split (with operator AND)
693 but instead kept in one RPN token. This facility appeared
698 <row><entry><literal>r=o</literal></entry><entry>
699 Allows ranges and the operators greather-than, less-than, ...
701 This sets Bib-1 relation attribute accordingly (relation
702 ordered). A query construct is only treated as a range if
703 dash is used and that is surrounded by white-space. So
704 <literal>-1980</literal> is treated as term
705 <literal>"-1980"</literal> not <literal><= 1980</literal>.
706 If <literal>- 1980</literal> is used, however, that is
711 <row><entry><literal>r=r</literal></entry><entry>
712 Similar to <literal>r=o</literal> but assumes that terms
713 are non-negative (not prefixed with <literal>-</literal>).
714 Thus, a dash will always be treated as a range.
715 The construct <literal>1980-1990</literal> is
716 treated as a range with <literal>r=r</literal> but as a
717 single term <literal>"1980-1990"</literal> with
718 <literal>r=o</literal>. The special attribute
719 <literal>r=r</literal> is available in YAZ 2.0.24 or later.
723 <row><entry><literal>t=l</literal></entry><entry>
724 Allows term to be left-truncated.
725 If term is of the form <literal>?x</literal>, the resulting
726 Type-1 term is <literal>x</literal> and truncation is left.
730 <row><entry><literal>t=r</literal></entry><entry>
731 Allows term to be right-truncated.
732 If term is of the form <literal>x?</literal>, the resulting
733 Type-1 term is <literal>x</literal> and truncation is right.
737 <row><entry><literal>t=n</literal></entry><entry>
738 If term is does not include <literal>?</literal>, the
739 truncation attribute is set to none (100).
743 <row><entry><literal>t=b</literal></entry><entry>
744 Allows term to be both left&right truncated.
745 If term is of the form <literal>?x?</literal>, the
746 resulting term is <literal>x</literal> and trunctation is
747 set to both left&right.
751 <row><entry><literal>t=x</literal></entry><entry>
752 Allows masking anywhere in a term, thus fully supporting
753 # (mask one character) and ? (zero or more of any).
754 If masking is used, trunction is set to 102 (regexp-1 in term)
755 and the term is converted accordingly to a regular expression.
759 <row><entry><literal>t=z</literal></entry><entry>
760 Allows masking anywhere in a term, thus fully supporting
761 # (mask one character) and ? (zero or more of any).
762 If masking is used, trunction is set to 104 (Z39.58 in term)
763 and the term is converted accordingly to Z39.58 masking term -
764 actually the same truncation as CCL itself.
772 <example id="example.ccl.profile"><title>CCL profile</title>
774 Consider the following definition:
785 <literal>ti</literal> and <literal>au</literal> both set
786 structure attribute to phrase (s=1).
787 <literal>ti</literal>
788 sets the use-attribute to 4. <literal>au</literal> sets the
790 When no qualifiers are used in the query the structure-attribute is
791 set to free-form-text (105) (rule for <literal>term</literal>).
792 The <literal>date</literal> sets the relation attribute to
793 the relation used in the CCL query and sets the use attribute
797 You can combine attributes. To Search for "ranked title" you
800 ti,ranked=knuth computer
802 which will set relation=ranked, use=title, structure=phrase.
809 is a valid query. But
817 <sect4 id="ccl.qualifier.alias">
818 <title>Qualifier alias</title>
820 A qualifier alias is of the form:
823 <replaceable>q</replaceable>
824 <replaceable>q1</replaceable> <replaceable>q2</replaceable> ..
827 which declares <replaceable>q</replaceable> to
828 be an alias for <replaceable>q1</replaceable>,
829 <replaceable>q2</replaceable>... such that the CCL
830 query <replaceable>q=x</replaceable> is equivalent to
831 <replaceable>q1=x or q2=x or ...</replaceable>.
835 <sect4 id="ccl.comments">
836 <title>Comments</title>
838 Lines with white space or lines that begin with
839 character <literal>#</literal> are treated as comments.
843 <sect4 id="ccl.directives">
844 <title>Directives</title>
846 Directive specifications takes the form
848 <para><literal>@</literal><replaceable>directive</replaceable> <replaceable>value</replaceable>
850 <table id="ccl.directives.table">
851 <title>CCL directives</title>
853 <colspec colwidth="2*" colname="name"></colspec>
854 <colspec colwidth="8*" colname="description"></colspec>
855 <colspec colwidth="1*" colname="default"></colspec>
859 <entry>Description</entry>
860 <entry>Default</entry>
865 <entry>truncation</entry>
866 <entry>Truncation character</entry>
867 <entry><literal>?</literal></entry>
871 <entry>Masking character. Requires YAZ 4.2.58 or later</entry>
872 <entry><literal>#</literal></entry>
876 <entry>Specifies how multiple fields are to be
877 combined. There are two modes: <literal>or</literal>:
878 multiple qualifier fields are ORed,
879 <literal>merge</literal>: attributes for the qualifier
880 fields are merged and assigned to one term.
882 <entry><literal>merge</literal></entry>
886 <entry>Specifies if CCL operators and qualifiers should be
887 compared with case sensitivity or not. Specify 1 for
888 case sensitive; 0 for case insensitive.</entry>
889 <entry><literal>1</literal></entry>
894 <entry>Specifies token for CCL operator AND.</entry>
895 <entry><literal>and</literal></entry>
900 <entry>Specifies token for CCL operator OR.</entry>
901 <entry><literal>or</literal></entry>
906 <entry>Specifies token for CCL operator NOT.</entry>
907 <entry><literal>not</literal></entry>
912 <entry>Specifies token for CCL operator SET.</entry>
913 <entry><literal>set</literal></entry>
921 <title>CCL API</title>
923 All public definitions can be found in the header file
924 <filename>ccl.h</filename>. A profile identifier is of type
925 <literal>CCL_bibset</literal>. A profile must be created with the call
926 to the function <function>ccl_qual_mk</function> which returns a profile
927 handle of type <literal>CCL_bibset</literal>.
931 To read a file containing qualifier definitions the function
932 <function>ccl_qual_file</function> may be convenient. This function
933 takes an already opened <literal>FILE</literal> handle pointer as
934 argument along with a <literal>CCL_bibset</literal> handle.
938 To parse a simple string with a FIND query use the function
941 struct ccl_rpn_node *ccl_find_str(CCL_bibset bibset, const char *str,
942 int *error, int *pos);
945 which takes the CCL profile (<literal>bibset</literal>) and query
946 (<literal>str</literal>) as input. Upon successful completion the RPN
947 tree is returned. If an error occur, such as a syntax error, the integer
948 pointed to by <literal>error</literal> holds the error code and
949 <literal>pos</literal> holds the offset inside query string in which
954 An English representation of the error may be obtained by calling
955 the <literal>ccl_err_msg</literal> function. The error codes are
956 listed in <filename>ccl.h</filename>.
960 To convert the CCL RPN tree (type
961 <literal>struct ccl_rpn_node *</literal>)
962 to the Z_RPNQuery of YAZ the function <function>ccl_rpn_query</function>
963 must be used. This function which is part of YAZ is implemented in
964 <filename>yaz-ccl.c</filename>.
965 After calling this function the CCL RPN tree is probably no longer
966 needed. The <literal>ccl_rpn_delete</literal> destroys the CCL RPN tree.
970 A CCL profile may be destroyed by calling the
971 <function>ccl_qual_rm</function> function.
975 The token names for the CCL operators may be changed by setting the
976 globals (all type <literal>char *</literal>)
977 <literal>ccl_token_and</literal>, <literal>ccl_token_or</literal>,
978 <literal>ccl_token_not</literal> and <literal>ccl_token_set</literal>.
979 An operator may have aliases, i.e. there may be more than one name for
980 the operator. To do this, separate each alias with a space character.
984 <sect2 id="cql"><title>CQL</title>
986 <ulink url="&url.cql;">CQL</ulink>
987 - Common Query Language - was defined for the
988 <ulink url="&url.sru;">SRU</ulink> protocol.
989 In many ways CQL has a similar syntax to CCL.
990 The objective of CQL is different. Where CCL aims to be
991 an end-user language, CQL is <emphasis>the</emphasis> protocol
992 query language for SRU.
996 If you are new to CQL, read the
997 <ulink url="&url.cql.intro;">Gentle Introduction</ulink>.
1001 The CQL parser in &yaz; provides the following:
1005 It parses and validates a CQL query.
1010 It generates a C structure that allows you to convert
1011 a CQL query to some other query language, such as SQL.
1016 The parser converts a valid CQL query to PQF, thus providing a
1017 way to use CQL for both SRU servers and Z39.50 targets at the
1023 The parser converts CQL to XCQL.
1024 XCQL is an XML representation of CQL.
1025 XCQL is part of the SRU specification. However, since SRU
1026 supports CQL only, we don't expect XCQL to be widely used.
1027 Furthermore, CQL has the advantage over XCQL that it is
1033 <sect3 id="cql.parsing"><title>CQL parsing</title>
1035 A CQL parser is represented by the <literal>CQL_parser</literal>
1036 handle. Its contents should be considered &yaz; internal (private).
1038 #include <yaz/cql.h>
1040 typedef struct cql_parser *CQL_parser;
1042 CQL_parser cql_parser_create(void);
1043 void cql_parser_destroy(CQL_parser cp);
1045 A parser is created by <function>cql_parser_create</function> and
1046 is destroyed by <function>cql_parser_destroy</function>.
1049 To parse a CQL query string, the following function
1052 int cql_parser_string(CQL_parser cp, const char *str);
1054 A CQL query is parsed by the <function>cql_parser_string</function>
1055 which takes a query <parameter>str</parameter>.
1056 If the query was valid (no syntax errors), then zero is returned;
1057 otherwise -1 is returned to indicate a syntax error.
1061 int cql_parser_stream(CQL_parser cp,
1062 int (*getbyte)(void *client_data),
1063 void (*ungetbyte)(int b, void *client_data),
1066 int cql_parser_stdio(CQL_parser cp, FILE *f);
1068 The functions <function>cql_parser_stream</function> and
1069 <function>cql_parser_stdio</function> parses a CQL query
1070 - just like <function>cql_parser_string</function>.
1071 The only difference is that the CQL query can be
1072 fed to the parser in different ways.
1073 The <function>cql_parser_stream</function> uses a generic
1074 byte stream as input. The <function>cql_parser_stdio</function>
1075 uses a <literal>FILE</literal> handle which is opened for reading.
1079 <sect3 id="cql.tree"><title>CQL tree</title>
1081 The the query string is valid, the CQL parser
1082 generates a tree representing the structure of the
1087 struct cql_node *cql_parser_result(CQL_parser cp);
1089 <function>cql_parser_result</function> returns the
1090 a pointer to the root node of the resulting tree.
1093 Each node in a CQL tree is represented by a
1094 <literal>struct cql_node</literal>.
1095 It is defined as follows:
1097 #define CQL_NODE_ST 1
1098 #define CQL_NODE_BOOL 2
1099 #define CQL_NODE_SORT 3
1109 struct cql_node *modifiers;
1113 struct cql_node *left;
1114 struct cql_node *right;
1115 struct cql_node *modifiers;
1119 struct cql_node *next;
1120 struct cql_node *modifiers;
1121 struct cql_node *search;
1126 There are three node types: search term (ST), boolean (BOOL)
1128 A modifier is treated as a search term too.
1131 The search term node has five members:
1135 <literal>index</literal>: index for search term.
1136 If an index is unspecified for a search term,
1137 <literal>index</literal> will be NULL.
1142 <literal>index_uri</literal>: index URi for search term
1143 or NULL if none could be resolved for the index.
1148 <literal>term</literal>: the search term itself.
1153 <literal>relation</literal>: relation for search term.
1158 <literal>relation_uri</literal>: relation URI for search term.
1163 <literal>modifiers</literal>: relation modifiers for search
1164 term. The <literal>modifiers</literal> list itself of cql_nodes
1165 each of type <literal>ST</literal>.
1172 The boolean node represents <literal>and</literal>,
1173 <literal>or</literal>, <literal>not</literal> +
1178 <literal>left</literal> and <literal>right</literal>: left
1179 - and right operand respectively.
1184 <literal>modifiers</literal>: proximity arguments.
1191 The sort node represents both the SORTBY clause.
1195 <sect3 id="cql.to.pqf"><title>CQL to PQF conversion</title>
1197 Conversion to PQF (and Z39.50 RPN) is tricky by the fact
1198 that the resulting RPN depends on the Z39.50 target
1199 capabilities (combinations of supported attributes).
1200 In addition, the CQL and SRU operates on index prefixes
1201 (URI or strings), whereas the RPN uses Object Identifiers
1205 The CQL library of &yaz; defines a <literal>cql_transform_t</literal>
1206 type. It represents a particular mapping between CQL and RPN.
1207 This handle is created and destroyed by the functions:
1209 cql_transform_t cql_transform_open_FILE (FILE *f);
1210 cql_transform_t cql_transform_open_fname(const char *fname);
1211 void cql_transform_close(cql_transform_t ct);
1213 The first two functions create a tranformation handle from
1214 either an already open FILE or from a filename respectively.
1217 The handle is destroyed by <function>cql_transform_close</function>
1218 in which case no further reference of the handle is allowed.
1221 When a <literal>cql_transform_t</literal> handle has been created
1222 you can convert to RPN.
1224 int cql_transform_buf(cql_transform_t ct,
1225 struct cql_node *cn, char *out, int max);
1227 This function converts the CQL tree <literal>cn</literal>
1228 using handle <literal>ct</literal>.
1229 For the resulting PQF, you supply a buffer <literal>out</literal>
1230 which must be able to hold at at least <literal>max</literal>
1234 If conversion failed, <function>cql_transform_buf</function>
1235 returns a non-zero SRU error code; otherwise zero is returned
1236 (conversion successful). The meanings of the numeric error
1237 codes are listed in the SRU specification somewhere (no
1238 direct link anymore).
1241 If conversion fails, more information can be obtained by calling
1243 int cql_transform_error(cql_transform_t ct, char **addinfop);
1245 This function returns the most recently returned numeric
1246 error-code and sets the string-pointer at
1247 <literal>*addinfop</literal> to point to a string containing
1248 additional information about the error that occurred: for
1249 example, if the error code is 15 (``Illegal or unsupported context
1250 set''), the additional information is the name of the requested
1251 context set that was not recognised.
1254 The SRU error-codes may be translated into brief human-readable
1255 error messages using
1257 const char *cql_strerror(int code);
1261 If you wish to be able to produce a PQF result in a different
1262 way, there are two alternatives.
1264 void cql_transform_pr(cql_transform_t ct,
1265 struct cql_node *cn,
1266 void (*pr)(const char *buf, void *client_data),
1269 int cql_transform_FILE(cql_transform_t ct,
1270 struct cql_node *cn, FILE *f);
1272 The former function produces output to a user-defined
1273 output stream. The latter writes the result to an already
1274 open <literal>FILE</literal>.
1277 <sect3 id="cql.to.rpn">
1278 <title>Specification of CQL to RPN mappings</title>
1280 The file supplied to functions
1281 <function>cql_transform_open_FILE</function>,
1282 <function>cql_transform_open_fname</function> follows
1283 a structure found in many Unix utilities.
1284 It consists of mapping specifications - one per line.
1285 Lines starting with <literal>#</literal> are ignored (comments).
1288 Each line is of the form
1290 <replaceable>CQL pattern</replaceable><literal> = </literal> <replaceable> RPN equivalent</replaceable>
1294 An RPN pattern is a simple attribute list. Each attribute pair
1297 [<replaceable>set</replaceable>] <replaceable>type</replaceable><literal>=</literal><replaceable>value</replaceable>
1299 The attribute <replaceable>set</replaceable> is optional.
1300 The <replaceable>type</replaceable> is the attribute type,
1301 <replaceable>value</replaceable> the attribute value.
1304 The character <literal>*</literal> (asterisk) has special meaning
1305 when used in the RPN pattern.
1306 Each occurrence of <literal>*</literal> is substituted with the
1307 CQL matching name (index, relation, qualifier etc).
1308 This facility can be used to copy a CQL name verbatim to the RPN result.
1311 The following CQL patterns are recognized:
1313 <varlistentry><term>
1314 <literal>index.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
1318 This pattern is invoked when a CQL index, such as
1319 dc.title is converted. <replaceable>set</replaceable>
1320 and <replaceable>name</replaceable> are the context set and index
1322 Typically, the RPN specifies an equivalent use attribute.
1325 For terms not bound by an index the pattern
1326 <literal>index.cql.serverChoice</literal> is used.
1327 Here, the prefix <literal>cql</literal> is defined as
1328 <literal>http://www.loc.gov/zing/cql/cql-indexes/v1.0/</literal>.
1329 If this pattern is not defined, the mapping will fail.
1333 <literal>index.</literal><replaceable>set</replaceable><literal>.*</literal>
1334 is used when no other index pattern is matched.
1338 <varlistentry><term>
1339 <literal>qualifier.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
1344 For backwards compatibility, this is recognised as a synonym of
1345 <literal>index.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
1349 <varlistentry><term>
1350 <literal>relation.</literal><replaceable>relation</replaceable>
1354 This pattern specifies how a CQL relation is mapped to RPN.
1355 <replaceable>pattern</replaceable> is name of relation
1356 operator. Since <literal>=</literal> is used as
1357 separator between CQL pattern and RPN, CQL relations
1358 including <literal>=</literal> cannot be
1359 used directly. To avoid a conflict, the names
1360 <literal>ge</literal>,
1361 <literal>eq</literal>,
1362 <literal>le</literal>,
1363 must be used for CQL operators, greater-than-or-equal,
1364 equal, less-than-or-equal respectively.
1365 The RPN pattern is supposed to include a relation attribute.
1368 For terms not bound by a relation, the pattern
1369 <literal>relation.scr</literal> is used. If the pattern
1370 is not defined, the mapping will fail.
1373 The special pattern, <literal>relation.*</literal> is used
1374 when no other relation pattern is matched.
1379 <varlistentry><term>
1380 <literal>relationModifier.</literal><replaceable>mod</replaceable>
1384 This pattern specifies how a CQL relation modifier is mapped to RPN.
1385 The RPN pattern is usually a relation attribute.
1390 <varlistentry><term>
1391 <literal>structure.</literal><replaceable>type</replaceable>
1395 This pattern specifies how a CQL structure is mapped to RPN.
1396 Note that this CQL pattern is somewhat to similar to
1397 CQL pattern <literal>relation</literal>.
1398 The <replaceable>type</replaceable> is a CQL relation.
1401 The pattern, <literal>structure.*</literal> is used
1402 when no other structure pattern is matched.
1403 Usually, the RPN equivalent specifies a structure attribute.
1408 <varlistentry><term>
1409 <literal>position.</literal><replaceable>type</replaceable>
1413 This pattern specifies how the anchor (position) of
1414 CQL is mapped to RPN.
1415 The <replaceable>type</replaceable> is one
1416 of <literal>first</literal>, <literal>any</literal>,
1417 <literal>last</literal>, <literal>firstAndLast</literal>.
1420 The pattern, <literal>position.*</literal> is used
1421 when no other position pattern is matched.
1426 <varlistentry><term>
1427 <literal>set.</literal><replaceable>prefix</replaceable>
1431 This specification defines a CQL context set for a given prefix.
1432 The value on the right hand side is the URI for the set -
1433 <emphasis>not</emphasis> RPN. All prefixes used in
1434 index patterns must be defined this way.
1439 <varlistentry><term>
1440 <literal>set</literal>
1444 This specification defines a default CQL context set for index names.
1445 The value on the right hand side is the URI for the set.
1452 <example id="example.cql.to.rpn.mapping">
1453 <title>CQL to RPN mapping file</title>
1455 This simple file defines two context sets, three indexes and three
1456 relations, a position pattern and a default structure.
1458 <programlisting><![CDATA[
1459 set.cql = http://www.loc.gov/zing/cql/context-sets/cql/v1.1/
1460 set.dc = http://www.loc.gov/zing/cql/dc-indexes/v1.0/
1462 index.cql.serverChoice = 1=1016
1463 index.dc.title = 1=4
1464 index.dc.subject = 1=21
1470 position.any = 3=3 6=1
1476 With the mappings above, the CQL query
1480 is converted to the PQF:
1482 @attr 1=1016 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "computer"
1484 by rules <literal>index.cql.serverChoice</literal>,
1485 <literal>relation.scr</literal>, <literal>structure.*</literal>,
1486 <literal>position.any</literal>.
1493 is rejected, since <literal>position.right</literal> is
1499 >my = "http://www.loc.gov/zing/cql/dc-indexes/v1.0/" my.title = x
1503 @attr 1=4 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "x"
1507 <example id="example.cql.to.rpn.string">
1508 <title>CQL to RPN string attributes</title>
1510 In this example we allow any index to be passed to RPN as
1513 <programlisting><![CDATA[
1514 # Identifiers for prefixes used in this file. (index.*)
1515 set.cql = info:srw/cql-context-set/1/cql-v1.1
1516 set.rpn = http://bogus/rpn
1517 set = http://bogus/rpn
1519 # The default index when none is specified by the query
1520 index.cql.serverChoice = 1=any
1529 The <literal>http://bogus/rpn</literal> context set is also the default
1530 so we can make queries such as
1534 which is converted to
1536 @attr 2=3 @attr 4=1 @attr 3=3 @attr 1=title "a"
1540 <example id="example.cql.to.rpn.bathprofile">
1541 <title>CQL to RPN using Bath Profile</title>
1543 The file <filename>etc/pqf.properties</filename> has mappings from
1544 the Bath Profile and Dublin Core to RPN.
1545 If YAZ is installed as a package it's usually located
1546 in <filename>/usr/share/yaz/etc</filename> and part of the
1547 development package, such as <literal>libyaz-dev</literal>.
1551 <sect3 id="cql.xcql"><title>CQL to XCQL conversion</title>
1553 Conversion from CQL to XCQL is trivial and does not
1554 require a mapping to be defined.
1555 There three functions to choose from depending on the
1556 way you wish to store the resulting output (XML buffer
1559 int cql_to_xml_buf(struct cql_node *cn, char *out, int max);
1560 void cql_to_xml(struct cql_node *cn,
1561 void (*pr)(const char *buf, void *client_data),
1563 void cql_to_xml_stdio(struct cql_node *cn, FILE *f);
1565 Function <function>cql_to_xml_buf</function> converts
1566 to XCQL and stores result in a user supplied buffer of a given
1570 <function>cql_to_xml</function> writes the result in
1571 a user defined output stream.
1572 <function>cql_to_xml_stdio</function> writes to a
1576 <sect3 id="rpn.to.cql">
1577 <title>PQF to CQL conversion</title>
1579 Conversion from PQF to CQL is offered by the two functions shown
1580 below. The former uses a generic stream for result. The latter
1581 puts result in a WRBUF (string container).
1583 #include <yaz/rpn2cql.h>
1585 int cql_transform_rpn2cql_stream(cql_transform_t ct,
1586 void (*pr)(const char *buf, void *client_data),
1590 int cql_transform_rpn2cql_wrbuf(cql_transform_t ct,
1594 The configuration is the same as used in CQL to PQF conversions.
1599 <sect1 id="tools.oid"><title>Object Identifiers</title>
1602 The basic YAZ representation of an OID is an array of integers,
1603 terminated with the value -1. This integer is of type
1604 <literal>Odr_oid</literal>.
1607 Fundamental OID operations and the type <literal>Odr_oid</literal>
1608 are defined in <filename>yaz/oid_util.h</filename>.
1611 An OID can either be declared as a automatic variable or it can
1612 allocated using the memory utilities or ODR/NMEM. It's
1613 guaranteed that an OID can fit in <literal>OID_SIZE</literal> integers.
1615 <example id="tools.oid.bib1.1"><title>Create OID on stack</title>
1617 We can create an OID for the Bib-1 attribute set with:
1619 Odr_oid bib1[OID_SIZE];
1631 And OID may also be filled from a string-based representation using
1632 dots (.). This is achieved by function
1634 int oid_dotstring_to_oid(const char *name, Odr_oid *oid);
1636 This functions returns 0 if name could be converted; -1 otherwise.
1638 <example id="tools.oid.bib1.2"><title>Using oid_oiddotstring_to_oid</title>
1640 We can fill the Bib-1 attribute set OID easier with:
1642 Odr_oid bib1[OID_SIZE];
1643 oid_oiddotstring_to_oid("1.2.840.10003.3.1", bib1);
1648 We can also allocate an OID dynamically on a ODR stream with:
1650 Odr_oid *odr_getoidbystr(ODR o, const char *str);
1652 This creates an OID from string-based representation using dots.
1653 This function take an &odr; stream as parameter. This stream is used to
1654 allocate memory for the data elements, which is released on a
1655 subsequent call to <function>odr_reset()</function> on that stream.
1658 <example id="tools.oid.bib1.3"><title>Using odr_getoidbystr</title>
1660 We can create a OID for the Bib-1 attribute set with:
1662 Odr_oid *bib1 = odr_getoidbystr(odr, "1.2.840.10003.3.1");
1670 char *oid_oid_to_dotstring(const Odr_oid *oid, char *oidbuf)
1672 does the reverse of <function>oid_oiddotstring_to_oid</function>. It
1673 converts an OID to the string-based representation using dots.
1674 The supplied char buffer <literal>oidbuf</literal> holds the resulting
1675 string and must be at least <literal>OID_STR_MAX</literal> in size.
1679 OIDs can be copied with <function>oid_oidcpy</function> which takes
1680 two OID lists as arguments. Alternativly, an OID copy can be allocated
1681 on a ODR stream with:
1683 Odr_oid *odr_oiddup(ODR odr, const Odr_oid *o);
1688 OIDs can be compared with <function>oid_oidcmp</function> which returns
1689 zero if the two OIDs provided are identical; non-zero otherwise.
1692 <sect2 id="tools.oid.database"><title>OID database</title>
1694 From YAZ version 3 and later, the oident system has been replaced
1695 by an OID database. OID database is a misnomer .. the old odient
1696 system was also a database.
1699 The OID database is really just a map between named Object Identifiers
1700 (string) and their OID raw equivalents. Most operations either
1701 convert from string to OID or other way around.
1704 Unfortunately, whenever we supply a string we must also specify the
1705 <emphasis>OID class</emphasis>. The class is necessary because some
1706 strings correspond to multiple OIDs. An example of such a string is
1707 <literal>Bib-1</literal> which may either be an attribute-set
1708 or a diagnostic-set.
1711 Applications using the YAZ database should include
1712 <filename>yaz/oid_db.h</filename>.
1715 A YAZ database handle is of type <literal>yaz_oid_db_t</literal>.
1716 Actually that's a pointer. You need not think deal with that.
1717 YAZ has a built-in database which can be considered "constant" for
1719 We can get hold that by using function <function>yaz_oid_std</function>.
1722 All functions with prefix <function>yaz_string_to_oid</function>
1723 converts from class + string to OID. We have variants of this
1724 operation due to different memory allocation strategies.
1727 All functions with prefix
1728 <function>yaz_oid_to_string</function> converts from OID to string
1732 <example id="tools.oid.bib1.4"><title>Create OID with YAZ DB</title>
1734 We can create an OID for the Bib-1 attribute set on the ODR stream
1738 yaz_string_to_oid_odr(yaz_oid_std(), CLASS_ATTSET, "Bib-1", odr);
1740 This is more complex than using <function>odr_getoidbystr</function>.
1741 You would only use <function>yaz_string_to_oid_odr</function> when the
1742 string (here Bib-1) is supplied by a user or configuration.
1747 <sect2 id="tools.oid.std"><title>Standard OIDs</title>
1750 All the object identifers in the standard OID database as returned
1751 by <function>yaz_oid_std</function> can referenced directly in a
1752 program as a constant OID.
1753 Each constant OID is prefixed with <literal>yaz_oid_</literal> -
1754 followed by OID class (lowercase) - then by OID name (normalized and
1758 See <xref linkend="list-oids"/> for list of all object identifiers
1760 These are declared in <filename>yaz/oid_std.h</filename> but are
1761 included by <filename>yaz/oid_db.h</filename> as well.
1764 <example id="tools.oid.bib1.5"><title>Use a built-in OID</title>
1766 We can allocate our own OID filled with the constant OID for
1769 Odr_oid *bib1 = odr_oiddup(o, yaz_oid_attset_bib1);
1775 <sect1 id="tools.nmem"><title>Nibble Memory</title>
1778 Sometimes when you need to allocate and construct a large,
1779 interconnected complex of structures, it can be a bit of a pain to
1780 release the associated memory again. For the structures describing the
1781 Z39.50 PDUs and related structures, it is convenient to use the
1782 memory-management system of the &odr; subsystem (see
1783 <xref linkend="odr.use"/>). However, in some circumstances
1784 where you might otherwise benefit from using a simple nibble memory
1785 management system, it may be impractical to use
1786 <function>odr_malloc()</function> and <function>odr_reset()</function>.
1787 For this purpose, the memory manager which also supports the &odr;
1788 streams is made available in the NMEM module. The external interface
1789 to this module is given in the <filename>nmem.h</filename> file.
1793 The following prototypes are given:
1797 NMEM nmem_create(void);
1798 void nmem_destroy(NMEM n);
1799 void *nmem_malloc(NMEM n, size_t size);
1800 void nmem_reset(NMEM n);
1801 size_t nmem_total(NMEM n);
1802 void nmem_init(void);
1803 void nmem_exit(void);
1807 The <function>nmem_create()</function> function returns a pointer to a
1808 memory control handle, which can be released again by
1809 <function>nmem_destroy()</function> when no longer needed.
1810 The function <function>nmem_malloc()</function> allocates a block of
1811 memory of the requested size. A call to <function>nmem_reset()</function>
1812 or <function>nmem_destroy()</function> will release all memory allocated
1813 on the handle since it was created (or since the last call to
1814 <function>nmem_reset()</function>. The function
1815 <function>nmem_total()</function> returns the number of bytes currently
1816 allocated on the handle.
1820 The nibble memory pool is shared amongst threads. POSIX
1821 mutex'es and WIN32 Critical sections are introduced to keep the
1822 module thread safe. Function <function>nmem_init()</function>
1823 initializes the nibble memory library and it is called automatically
1824 the first time the <literal>YAZ.DLL</literal> is loaded. &yaz; uses
1825 function <function>DllMain</function> to achieve this. You should
1826 <emphasis>not</emphasis> call <function>nmem_init</function> or
1827 <function>nmem_exit</function> unless you're absolute sure what
1828 you're doing. Note that in previous &yaz; versions you'd have to call
1829 <function>nmem_init</function> yourself.
1834 <sect1 id="tools.log"><title>Log</title>
1836 &yaz; has evolved a fairly complex log system which should be useful both
1837 for debugging &yaz; itself, debugging applications that use &yaz;, and for
1838 production use of those applications.
1841 The log functions are declared in header <filename>yaz/log.h</filename>
1842 and implemented in <filename>src/log.c</filename>.
1843 Due to name clash with syslog and some math utilities the logging
1844 interface has been modified as of YAZ 2.0.29. The obsolete interface
1845 is still available if in header file <filename>yaz/log.h</filename>.
1846 The key points of the interface are:
1849 void yaz_log(int level, const char *fmt, ...)
1851 void yaz_log_init(int level, const char *prefix, const char *name);
1852 void yaz_log_init_file(const char *fname);
1853 void yaz_log_init_level(int level);
1854 void yaz_log_init_prefix(const char *prefix);
1855 void yaz_log_time_format(const char *fmt);
1856 void yaz_log_init_max_size(int mx);
1858 int yaz_log_mask_str(const char *str);
1859 int yaz_log_module_level(const char *name);
1863 The reason for the whole log module is the <function>yaz_log</function>
1864 function. It takes a bitmask indicating the log levels, a
1865 <literal>printf</literal>-like format string, and a variable number of
1870 The <literal>log level</literal> is a bit mask, that says on which level(s)
1871 the log entry should be made, and optionally set some behaviour of the
1872 logging. In the most simple cases, it can be one of <literal>YLOG_FATAL,
1873 YLOG_DEBUG, YLOG_WARN, YLOG_LOG</literal>. Those can be combined with bits
1874 that modify the way the log entry is written:<literal>YLOG_ERRNO,
1875 YLOG_NOTIME, YLOG_FLUSH</literal>.
1876 Most of the rest of the bits are deprecated, and should not be used. Use
1877 the dynamic log levels instead.
1881 Applications that use &yaz;, should not use the LOG_LOG for ordinary
1882 messages, but should make use of the dynamic loglevel system. This consists
1883 of two parts, defining the loglevel and checking it.
1887 To define the log levels, the (main) program should pass a string to
1888 <function>yaz_log_mask_str</function> to define which log levels are to be
1889 logged. This string should be a comma-separated list of log level names,
1890 and can contain both hard-coded names and dynamic ones. The log level
1891 calculation starts with <literal>YLOG_DEFAULT_LEVEL</literal> and adds a bit
1892 for each word it meets, unless the word starts with a '-', in which case it
1893 clears the bit. If the string <literal>'none'</literal> is found,
1894 all bits are cleared. Typically this string comes from the command-line,
1895 often identified by <literal>-v</literal>. The
1896 <function>yaz_log_mask_str</function> returns a log level that should be
1897 passed to <function>yaz_log_init_level</function> for it to take effect.
1901 Each module should check what log bits it should be used, by calling
1902 <function>yaz_log_module_level</function> with a suitable name for the
1903 module. The name is cleared from a preceding path and an extension, if any,
1904 so it is quite possible to use <literal>__FILE__</literal> for it. If the
1905 name has been passed to <function>yaz_log_mask_str</function>, the routine
1906 returns a non-zero bitmask, which should then be used in consequent calls
1907 to yaz_log. (It can also be tested, so as to avoid unnecessary calls to
1908 yaz_log, in time-critical places, or when the log entry would take time
1913 Yaz uses the following dynamic log levels:
1914 <literal>server, session, request, requestdetail</literal> for the server
1916 <literal>zoom</literal> for the zoom client api.
1917 <literal>ztest</literal> for the simple test server.
1918 <literal>malloc, nmem, odr, eventl</literal> for internal debugging of yaz itself.
1919 Of course, any program using yaz is welcome to define as many new ones, as
1924 By default the log is written to stderr, but this can be changed by a call
1925 to <function>yaz_log_init_file</function> or
1926 <function>yaz_log_init</function>. If the log is directed to a file, the
1927 file size is checked at every write, and if it exceeds the limit given in
1928 <function>yaz_log_init_max_size</function>, the log is rotated. The
1929 rotation keeps one old version (with a <literal>.1</literal> appended to
1930 the name). The size defaults to 1GB. Setting it to zero will disable the
1935 A typical yaz-log looks like this
1936 13:23:14-23/11 yaz-ztest(1) [session] Starting session from tcp:127.0.0.1 (pid=30968)
1937 13:23:14-23/11 yaz-ztest(1) [request] Init from 'YAZ' (81) (ver 2.0.28) OK
1938 13:23:17-23/11 yaz-ztest(1) [request] Search Z: @attrset Bib-1 foo OK:7 hits
1939 13:23:22-23/11 yaz-ztest(1) [request] Present: [1] 2+2 OK 2 records returned
1940 13:24:13-23/11 yaz-ztest(1) [request] Close OK
1944 The log entries start with a time stamp. This can be omitted by setting the
1945 <literal>YLOG_NOTIME</literal> bit in the loglevel. This way automatic tests
1946 can be hoped to produce identical log files, that are easy to diff. The
1947 format of the time stamp can be set with
1948 <function>yaz_log_time_format</function>, which takes a format string just
1949 like <function>strftime</function>.
1953 Next in a log line comes the prefix, often the name of the program. For
1954 yaz-based servers, it can also contain the session number. Then
1955 comes one or more logbits in square brackets, depending on the logging
1956 level set by <function>yaz_log_init_level</function> and the loglevel
1957 passed to <function>yaz_log_init_level</function>. Finally comes the format
1958 string and additional values passed to <function>yaz_log</function>
1962 The log level <literal>YLOG_LOGLVL</literal>, enabled by the string
1963 <literal>loglevel</literal>, will log all the log-level affecting
1964 operations. This can come in handy if you need to know what other log
1965 levels would be useful. Grep the logfile for <literal>[loglevel]</literal>.
1969 The log system is almost independent of the rest of &yaz;, the only
1970 important dependence is of <filename>nmem</filename>, and that only for
1971 using the semaphore definition there.
1975 The dynamic log levels and log rotation were introduced in &yaz; 2.0.28. At
1976 the same time, the log bit names were changed from
1977 <literal>LOG_something</literal> to <literal>YLOG_something</literal>,
1978 to avoid collision with <filename>syslog.h</filename>.
1983 <sect1 id="marc"><title>MARC</title>
1986 YAZ provides a fast utility for working with MARC records.
1987 Early versions of the MARC utility only allowed decoding of ISO2709.
1988 Today the utility may both encode - and decode to a varity of formats.
1991 #include <yaz/marcdisp.h>
1993 /* create handler */
1994 yaz_marc_t yaz_marc_create(void);
1996 void yaz_marc_destroy(yaz_marc_t mt);
1998 /* set XML mode YAZ_MARC_LINE, YAZ_MARC_SIMPLEXML, ... */
1999 void yaz_marc_xml(yaz_marc_t mt, int xmlmode);
2000 #define YAZ_MARC_LINE 0
2001 #define YAZ_MARC_SIMPLEXML 1
2002 #define YAZ_MARC_OAIMARC 2
2003 #define YAZ_MARC_MARCXML 3
2004 #define YAZ_MARC_ISO2709 4
2005 #define YAZ_MARC_XCHANGE 5
2006 #define YAZ_MARC_CHECK 6
2007 #define YAZ_MARC_TURBOMARC 7
2008 #define YAZ_MARC_JSON 8
2010 /* supply iconv handle for character set conversion .. */
2011 void yaz_marc_iconv(yaz_marc_t mt, yaz_iconv_t cd);
2013 /* set debug level, 0=none, 1=more, 2=even more, .. */
2014 void yaz_marc_debug(yaz_marc_t mt, int level);
2016 /* decode MARC in buf of size bsize. Returns >0 on success; <=0 on failure.
2017 On success, result in *result with size *rsize. */
2018 int yaz_marc_decode_buf(yaz_marc_t mt, const char *buf, int bsize,
2019 const char **result, size_t *rsize);
2021 /* decode MARC in buf of size bsize. Returns >0 on success; <=0 on failure.
2022 On success, result in WRBUF */
2023 int yaz_marc_decode_wrbuf(yaz_marc_t mt, const char *buf,
2024 int bsize, WRBUF wrbuf);
2029 The synopsis is just a basic subset of all functionality. Refer
2030 to the actual header file <filename>marcdisp.h</filename> for
2035 A MARC conversion handle must be created by using
2036 <function>yaz_marc_create</function> and destroyed
2037 by calling <function>yaz_marc_destroy</function>.
2040 All other function operate on a <literal>yaz_marc_t</literal> handle.
2041 The output is specified by a call to <function>yaz_marc_xml</function>.
2042 The <literal>xmlmode</literal> must be one of
2045 <term>YAZ_MARC_LINE</term>
2048 A simple line-by-line format suitable for display but not
2049 recommend for further (machine) processing.
2055 <term>YAZ_MARC_MARCXML</term>
2058 <ulink url="&url.marcxml;">MARCXML</ulink>.
2064 <term>YAZ_MARC_ISO2709</term>
2067 ISO2709 (sometimes just referred to as "MARC").
2073 <term>YAZ_MARC_XCHANGE</term>
2076 <ulink url="&url.marcxchange;">MarcXchange</ulink>.
2082 <term>YAZ_MARC_CHECK</term>
2085 Pseudo format for validation only. Does not generate
2086 any real output except diagnostics.
2092 <term>YAZ_MARC_TURBOMARC</term>
2095 XML format with same semantics as MARCXML but more compact
2096 and geared towards fast processing with XSLT. Refer to
2097 <xref linkend="tools.turbomarc"/> for more information.
2103 <term>YAZ_MARC_JSON</term>
2106 <ulink url="&url.marc_in_json;">MARC-in_JSON</ulink> format.
2114 The actual conversion functions are
2115 <function>yaz_marc_decode_buf</function> and
2116 <function>yaz_marc_decode_wrbuf</function> which decodes and encodes
2117 a MARC record. The former function operates on simple buffers, the
2118 stores the resulting record in a WRBUF handle (WRBUF is a simple string
2121 <example id="example.marc.display">
2122 <title>Display of MARC record</title>
2124 The following program snippet illustrates how the MARC API may
2125 be used to convert a MARC record to the line-by-line format:
2126 <programlisting><![CDATA[
2127 void print_marc(const char *marc_buf, int marc_buf_size)
2129 char *result; /* for result buf */
2130 size_t result_len; /* for size of result */
2131 yaz_marc_t mt = yaz_marc_create();
2132 yaz_marc_xml(mt, YAZ_MARC_LINE);
2133 yaz_marc_decode_buf(mt, marc_buf, marc_buf_size,
2134 &result, &result_len);
2135 fwrite(result, result_len, 1, stdout);
2136 yaz_marc_destroy(mt); /* note that result is now freed... */
2142 <sect2 id="tools.turbomarc">
2143 <title>TurboMARC</title>
2145 TurboMARC is yet another XML encoding of a MARC record. The format
2146 was designed for fast processing with XSLT.
2150 Pazpar2 uses XSLT to convert an XML encoded MARC record to an internal
2151 representation. This conversion mostly check the tag of a MARC field
2152 to determine the basic rules in the conversion. This check is
2153 costly when that is tag is encoded as an attribute in MARCXML.
2154 By having the tag value as the element instead, makes processing
2155 many times faster (at least for Libxslt).
2158 TurboMARC is encoded as follows:
2161 Record elements is part of namespace
2162 "<literal>http://www.indexdata.com/turbomarc</literal>".
2165 A record is enclosed in element <literal>r</literal>.
2168 A collection of records is enclosed in element
2169 <literal>collection</literal>.
2172 The leader is encoded as element <literal>l</literal> with the
2173 leader content as its (text) value.
2176 A control field is encoded as element <literal>c</literal> concatenated
2177 with the tag value of the control field if the tag value
2178 matches the regular expression <literal>[a-zA-Z0-9]*</literal>.
2179 If the tag value do not match the regular expression
2180 <literal>[a-zA-Z0-9]*</literal> the control field is encoded
2181 as element <literal>c</literal> and attribute <literal>code</literal>
2182 will hold the tag value.
2183 This rule ensure that in the rare cases where a tag value might
2184 result in a non-wellformed XML YAZ encode it as a coded attribute
2188 The control field content is the the text value of this element.
2189 Indicators are encoded as attribute names
2190 <literal>i1</literal>, <literal>i2</literal>, etc.. and
2191 corresponding values for each indicator.
2194 A data field is encoded as element <literal>d</literal> concatenated
2195 with the tag value of the data field or using the attribute
2196 <literal>code</literal> as described in the rules for control fields.
2197 The children of the data field element is subfield elements.
2198 Each subfield element is encoded as <literal>s</literal>
2199 concatenated with the sub field code.
2200 The text of the subfield element is the contents of the subfield.
2201 Indicators are encoded as attributes for the data field element similar
2202 to the encoding for control fields.
2209 <sect1 id="tools.retrieval">
2210 <title>Retrieval Facility</title>
2212 YAZ version 2.1.20 or later includes a Retrieval facility tool
2213 which allows a SRU/Z39.50 to describe itself and perform record
2214 conversions. The idea is the following:
2219 An SRU/Z39.50 client sends a retrieval request which includes
2220 a combination of the following parameters: syntax (format),
2221 schema (or element set name).
2227 The retrieval facility is invoked with parameters in a
2228 server/proxy. The retrieval facility matches the parameters a set of
2229 "supported" retrieval types.
2230 If there is no match, the retrieval signals an error
2231 (syntax and / or schema not supported).
2237 For a successful match, the backend is invoked with the same
2238 or altered retrieval parameters (syntax, schema). If
2239 a record is received from the backend, it is converted to the
2240 frontend name / syntax.
2246 The resulting record is sent back the client and tagged with
2247 the frontend syntax / schema.
2254 The Retrieval facility is driven by an XML configuration. The
2255 configuration is neither Z39.50 ZeeRex or SRU ZeeRex. But it
2256 should be easy to generate both of them from the XML configuration.
2257 (unfortunately the two versions
2258 of ZeeRex differ substantially in this regard).
2260 <sect2 id="tools.retrieval.format">
2261 <title>Retrieval XML format</title>
2263 All elements should be covered by namespace
2264 <literal>http://indexdata.com/yaz</literal> .
2265 The root element node must be <literal>retrievalinfo</literal>.
2268 The <literal>retrievalinfo</literal> must include one or
2269 more <literal>retrieval</literal> elements. Each
2270 <literal>retrieval</literal> defines specific combination of
2271 syntax, name and identifier supported by this retrieval service.
2274 The <literal>retrieval</literal> element may include any of the
2275 following attributes:
2277 <varlistentry><term><literal>syntax</literal> (REQUIRED)</term>
2280 Defines the record syntax. Possible values is any
2281 of the names defined in YAZ' OID database or a raw
2286 <varlistentry><term><literal>name</literal> (OPTIONAL)</term>
2289 Defines the name of the retrieval format. This can be
2290 any string. For SRU, the value, is equivalent to schema (short-hand);
2291 for Z39.50 it's equivalent to simple element set name.
2292 For YAZ 3.0.24 and later this name may be specified as a glob
2293 expression with operators
2294 <literal>*</literal> and <literal>?</literal>.
2298 <varlistentry><term><literal>identifier</literal> (OPTIONAL)</term>
2301 Defines the URI schema name of the retrieval format. This can be
2302 any string. For SRU, the value, is equivalent to URI schema.
2303 For Z39.50, there is no equivalent.
2310 The <literal>retrieval</literal> may include one
2311 <literal>backend</literal> element. If a <literal>backend</literal>
2312 element is given, it specifies how the records are retrieved by
2313 some backend and how the records are converted from the backend to
2317 The attributes, <literal>name</literal> and <literal>syntax</literal>
2318 may be specified for the <literal>backend</literal> element. These
2319 semantics of these attributes is equivalent to those for the
2320 <literal>retrieval</literal>. However, these values are passed to
2324 The <literal>backend</literal> element may includes one or more
2325 conversion instructions (as children elements). The supported
2328 <varlistentry><term><literal>marc</literal></term>
2331 The <literal>marc</literal> element specifies a conversion
2332 to - and from ISO2709 encoded MARC and
2333 <ulink url="&url.marcxml;">&acro.marcxml;</ulink>/MarcXchange.
2334 The following attributes may be specified:
2337 <varlistentry><term><literal>inputformat</literal> (REQUIRED)</term>
2340 Format of input. Supported values are
2341 <literal>marc</literal> (for ISO2709), <literal>xml</literal>
2342 (MARCXML/MarcXchange) and <literal>json</literal>
2343 (<ulink url="&url.marc_in_json;">MARC-in_JSON</ulink>).
2348 <varlistentry><term><literal>outputformat</literal> (REQUIRED)</term>
2351 Format of output. Supported values are
2352 <literal>line</literal> (MARC line format);
2353 <literal>marcxml</literal> (for MARCXML),
2354 <literal>marc</literal> (ISO2709),
2355 <literal>marcxhcange</literal> (for MarcXchange),
2356 or <literal>json</literal>
2357 (<ulink url="&url.marc_in_json;">MARC-in_JSON </ulink>).
2362 <varlistentry><term><literal>inputcharset</literal> (OPTIONAL)</term>
2365 Encoding of input. For XML input formats, this need not
2366 be given, but for ISO2709 based inputformats, this should
2367 be set to the encoding used. For MARC21 records, a common
2368 inputcharset value would be <literal>marc-8</literal>.
2373 <varlistentry><term><literal>outputcharset</literal> (OPTIONAL)</term>
2376 Encoding of output. If outputformat is XML based, it is
2377 strongly recommened to use <literal>utf-8</literal>.
2386 <varlistentry><term><literal>xslt</literal></term>
2389 The <literal>xslt</literal> element specifies a conversion
2390 via &acro.xslt;. The following attributes may be specified:
2393 <varlistentry><term><literal>stylesheet</literal> (REQUIRED)</term>
2408 <sect2 id="tools.retrieval.examples">
2409 <title>Retrieval Facility Examples</title>
2410 <example id="tools.retrieval.marc21">
2411 <title>MARC21 backend</title>
2413 A typical way to use the retrieval facility is to enable XML
2414 for servers that only supports ISO2709 encoded MARC21 records.
2416 <programlisting><![CDATA[
2418 <retrieval syntax="usmarc" name="F"/>
2419 <retrieval syntax="usmarc" name="B"/>
2420 <retrieval syntax="xml" name="marcxml"
2421 identifier="info:srw/schema/1/marcxml-v1.1">
2422 <backend syntax="usmarc" name="F">
2423 <marc inputformat="marc" outputformat="marcxml"
2424 inputcharset="marc-8"/>
2427 <retrieval syntax="xml" name="dc">
2428 <backend syntax="usmarc" name="F">
2429 <marc inputformat="marc" outputformat="marcxml"
2430 inputcharset="marc-8"/>
2431 <xslt stylesheet="MARC21slim2DC.xsl"/>
2438 This means that our frontend supports:
2442 MARC21 F(ull) records.
2447 MARC21 B(rief) records.
2459 Dublin core records.
2466 <example id="tools.retrieval.marcxml">
2467 <title>MARCXML backend</title>
2469 SRW/SRU and Solr backends returns records in XML.
2470 If they return MARCXML or MarcXchange, the retrieval module
2471 can convert those into ISO2709 formats, most commonly USMARC
2473 In this example, the backend returns MARCXML for schema="marcxml".
2475 <programlisting><![CDATA[
2477 <retrieval syntax="usmarc">
2478 <backend syntax="xml" name="marcxml">
2479 <marc inputformat="xml" outputformat="marc"
2480 outputcharset="marc-8"/>
2483 <retrieval syntax="xml" name="marcxml"
2484 identifier="info:srw/schema/1/marcxml-v1.1"/>
2485 <retrieval syntax="xml" name="dc">
2486 <backend syntax="xml" name="marcxml">
2487 <xslt stylesheet="MARC21slim2DC.xsl"/>
2494 This means that our frontend supports:
2498 MARC21 records (any element set name) in MARC-8 encoding.
2503 MARCXML records for element-set=marcxml
2508 Dublin core records for element-set=dc.
2516 <sect2 id="tools.retrieval.api">
2519 It should be easy to use the retrieval systems from applications. Refer
2521 <filename>yaz/retrieval.h</filename> and
2522 <filename>yaz/record_conv.h</filename>.
2526 <sect1><title>Sorting</title>
2528 This chapter describes sorting and how it is supported in YAZ.
2529 Sorting applies to a result-set.
2530 The <ulink url="http://www.loc.gov/z3950/agency/markup/05.html#3.2.7">
2531 Z39.50 sorting facility
2533 takes one or more input result-sets
2534 and one result-set as output. The most simple case is that
2535 the input-set is the same as the output-set.
2538 Z39.50 sorting has a separate APDU (service) that is, thus, performed
2539 following a search (two phases).
2542 In SRU/Solr, however, the model is different. Here, sorting is specified
2543 during the the search operation. Note, however, that SRU might
2544 perform sort as separate search, by referring to an existing result-set
2545 in the query (result-set reference).
2547 <sect2><title>Using the Z39.50 sort service</title>
2549 yaz-client and the ZOOM API supports the Z39.50 sort facility. In any
2550 case the sort sequence or sort critiera is using a string notation.
2551 This notation is a one-line notation suitable for being manually
2552 entered or generated and allows for easy logging (one liner).
2553 For the ZOOM API, the sort is specified in the call to ZOOM_query_sortby
2554 function. For yaz-client the sort is performed and specified using
2555 the sort and sort+ commands. For description of the sort criteria notation
2556 refer to the <link linkend="sortspec">sort command</link> in the
2560 The ZOOM API might choose one of several sort strategies for
2561 sorting. Refer to <xref linkend="zoom-sort-strategy"/>.
2564 <sect2><title>Type-7 sort</title>
2566 Type-7 sort is an extension to the Bib-1 based RPN query where the
2567 sort specification is embedded as an Attribute-Plus-Term.
2570 The objectives for introducing Type-7 sorting is that it allows
2571 a client to perform sorting even if it does not implement/support
2572 Z39.50 sort. Virtually all Z39.50 client software supports
2573 RPN queries. It also may improve performance because the sort
2574 critieria is specified along with the search query.
2577 The sort is triggered by the presence of type 7 and the value of type 7
2579 <ulink url="http://www.loc.gov/z3950/agency/asn1.html#SortKeySpec">
2582 The value for type 7 is 1 for ascending and 2 for descending.
2584 <ulink url="http://www.loc.gov/z3950/agency/asn1.html#SortElement">
2587 only the generic part is handled. If generic sortKey is of type
2588 sortField, then attribute type 1 is present and the value is
2589 sortField (InternationalString). If generic sortKey is of type
2590 sortAttributes, then the attributes in list is used . generic sortKey
2591 of type elementSpec is not supported.
2594 The term in the sorting Attribute-Plus-Term combo should hold
2595 an integer. The value is 0 for primary sorting criteria, 1 for second
2602 <!-- Keep this comment at the end of the file
2607 sgml-minimize-attributes:nil
2608 sgml-always-quote-attributes:t
2611 sgml-parent-document: "yaz.xml"
2612 sgml-local-catalogs: nil
2613 sgml-namecase-general:t