1 <chapter id="tools"><title>Supporting Tools</title>
4 In support of the service API - primarily the ASN module, which
5 provides the pro-grammatic interface to the Z39.50 APDUs, &yaz; contains
6 a collection of tools that support the development of applications.
9 <sect1 id="tools.query"><title>Query Syntax Parsers</title>
12 Since the type-1 (RPN) query structure has no direct, useful string
13 representation, every origin application needs to provide some form of
14 mapping from a local query notation or representation to a
15 <token>Z_RPNQuery</token> structure. Some programmers will prefer to
16 construct the query manually, perhaps using
17 <function>odr_malloc()</function> to simplify memory management.
18 The &yaz; distribution includes three separate, query-generating tools
19 that may be of use to you.
22 <sect2 id="PQF"><title>Prefix Query Format</title>
25 Since RPN or reverse polish notation is really just a fancy way of
26 describing a suffix notation format (operator follows operands), it
27 would seem that the confusion is total when we now introduce a prefix
28 notation for RPN. The reason is one of simple laziness - it's somewhat
29 simpler to interpret a prefix format, and this utility was designed
30 for maximum simplicity, to provide a baseline representation for use
31 in simple test applications and scripting environments (like Tcl). The
32 demonstration client included with YAZ uses the PQF.
37 The PQF have been adopted by other parties developing Z39.50
38 software. It is often referred to as Prefix Query Notation
43 The PQF is defined by the pquery module in the YAZ library.
44 There are two sets of function that have similar behavior. First
45 set operates on a PQF parser handle, second set doesn't. First set
46 set of functions are more flexible than the second set. Second set
47 is obsolete and is only provided to ensure backwards compatibility.
50 First set of functions all operate on a PQF parser handle:
53 #include <yaz/pquery.h>
55 YAZ_PQF_Parser yaz_pqf_create (void);
57 void yaz_pqf_destroy (YAZ_PQF_Parser p);
59 Z_RPNQuery *yaz_pqf_parse (YAZ_PQF_Parser p, ODR o, const char *qbuf);
61 Z_AttributesPlusTerm *yaz_pqf_scan (YAZ_PQF_Parser p, ODR o,
62 Odr_oid **attributeSetId, const char *qbuf);
65 int yaz_pqf_error (YAZ_PQF_Parser p, const char **msg, size_t *off);
68 A PQF parser is created and destructed by functions
69 <function>yaz_pqf_create</function> and
70 <function>yaz_pqf_destroy</function> respectively.
71 Function <function>yaz_pqf_parse</function> parses query given
72 by string <literal>qbuf</literal>. If parsing was successful,
73 a Z39.50 RPN Query is returned which is created using ODR stream
74 <literal>o</literal>. If parsing failed, a NULL pointer is
76 Function <function>yaz_pqf_scan</function> takes a scan query in
77 <literal>qbuf</literal>. If parsing was successful, the function
78 returns attributes plus term pointer and modifies
79 <literal>attributeSetId</literal> to hold attribute set for the
80 scan request - both allocated using ODR stream <literal>o</literal>.
81 If parsing failed, yaz_pqf_scan returns a NULL pointer.
82 Error information for bad queries can be obtained by a call to
83 <function>yaz_pqf_error</function> which returns an error code and
84 modifies <literal>*msg</literal> to point to an error description,
85 and modifies <literal>*off</literal> to the offset within last
86 query were parsing failed.
89 The second set of functions are declared as follows:
92 #include <yaz/pquery.h>
94 Z_RPNQuery *p_query_rpn (ODR o, oid_proto proto, const char *qbuf);
96 Z_AttributesPlusTerm *p_query_scan (ODR o, oid_proto proto,
97 Odr_oid **attributeSetP, const char *qbuf);
99 int p_query_attset (const char *arg);
102 The function <function>p_query_rpn()</function> takes as arguments an
103 &odr; stream (see section <link linkend="odr">The ODR Module</link>)
104 to provide a memory source (the structure created is released on
105 the next call to <function>odr_reset()</function> on the stream), a
106 protocol identifier (one of the constants <token>PROTO_Z3950</token> and
107 <token>PROTO_SR</token>), an attribute set reference, and
108 finally a null-terminated string holding the query string.
111 If the parse went well, <function>p_query_rpn()</function> returns a
112 pointer to a <literal>Z_RPNQuery</literal> structure which can be
113 placed directly into a <literal>Z_SearchRequest</literal>.
114 If parsing failed, due to syntax error, a NULL pointer is returned.
117 The <literal>p_query_attset</literal> specifies which attribute set
118 to use if the query doesn't specify one by the
119 <literal>@attrset</literal> operator.
120 The <literal>p_query_attset</literal> returns 0 if the argument is a
121 valid attribute set specifier; otherwise the function returns -1.
125 The grammar of the PQF is as follows:
129 query ::= top-set query-struct.
131 top-set ::= [ '@attrset' string ]
133 query-struct ::= attr-spec | simple | complex | '@term' term-type query
135 attr-spec ::= '@attr' [ string ] string query-struct
137 complex ::= operator query-struct query-struct.
139 operator ::= '@and' | '@or' | '@not' | '@prox' proximity.
141 simple ::= result-set | term.
143 result-set ::= '@set' string.
147 proximity ::= exclusion distance ordered relation which-code unit-code.
149 exclusion ::= '1' | '0' | 'void'.
151 distance ::= integer.
153 ordered ::= '1' | '0'.
155 relation ::= integer.
157 which-code ::= 'known' | 'private' | integer.
159 unit-code ::= integer.
161 term-type ::= 'general' | 'numeric' | 'string' | 'oid' | 'datetime' | 'null'.
165 You will note that the syntax above is a fairly faithful
166 representation of RPN, except for the Attribute, which has been
167 moved a step away from the term, allowing you to associate one or more
168 attributes with an entire query structure. The parser will
169 automatically apply the given attributes to each term as required.
173 The @attr operator is followed by an attribute specification
174 (<literal>attr-spec</literal> above). The specification consists
175 of an optional attribute set, an attribute type-value pair and
176 a sub-query. The attribute type-value pair is packed in one string:
177 an attribute type, an equals sign, and an attribute value, like this:
178 <literal>@attr 1=1003</literal>.
179 The type is always an integer but the value may be either an
180 integer or a string (if it doesn't start with a digit character).
181 A string attribute-value is encoded as a Type-1 ``complex''
182 attribute with the list of values containing the single string
183 specified, and including no semantic indicators.
187 Version 3 of the Z39.50 specification defines various encoding of terms.
188 Use <literal>@term </literal> <replaceable>type</replaceable>
189 <replaceable>string</replaceable>,
190 where type is one of: <literal>general</literal>,
191 <literal>numeric</literal> or <literal>string</literal>
192 (for InternationalString).
193 If no term type has been given, the <literal>general</literal> form
194 is used. This is the only encoding allowed in both versions 2 and 3
195 of the Z39.50 standard.
198 <sect3 id="PQF-prox">
199 <title>Using Proximity Operators with PQF</title>
202 This is an advanced topic, describing how to construct
203 queries that make very specific requirements on the
204 relative location of their operands.
205 You may wish to skip this section and go straight to
206 <link linkend="pqf-examples">the example PQF queries</link>.
211 Most Z39.50 servers do not support proximity searching, or
212 support only a small subset of the full functionality that
213 can be expressed using the PQF proximity operator. Be
214 aware that the ability to <emphasis>express</emphasis> a
215 query in PQF is no guarantee that any given server will
216 be able to <emphasis>execute</emphasis> it.
222 The proximity operator <literal>@prox</literal> is a special
223 and more restrictive version of the conjunction operator
224 <literal>@and</literal>. Its semantics are described in
225 section 3.7.2 (Proximity) of Z39.50 the standard itself, which
226 can be read on-line at
227 <ulink url="&url.z39.50.proximity;"/>
230 In PQF, the proximity operation is represented by a sequence
233 @prox <replaceable>exclusion</replaceable> <replaceable>distance</replaceable> <replaceable>ordered</replaceable> <replaceable>relation</replaceable> <replaceable>which-code</replaceable> <replaceable>unit-code</replaceable>
235 in which the meanings of the parameters are as described in in
236 the standard, and they can take the following values:
238 <listitem><formalpara><title>exclusion</title><para>
239 0 = false (i.e. the proximity condition specified by the
240 remaining parameters must be satisfied) or
241 1 = true (the proximity condition specified by the
242 remaining parameters must <emphasis>not</emphasis> be
244 </para></formalpara></listitem>
245 <listitem><formalpara><title>distance</title><para>
246 An integer specifying the difference between the locations
247 of the operands: e.g. two adjacent words would have
248 distance=1 since their locations differ by one unit.
249 </para></formalpara></listitem>
250 <listitem><formalpara><title>ordered</title><para>
251 1 = ordered (the operands must occur in the order the
252 query specifies them) or
253 0 = unordered (they may appear in either order).
254 </para></formalpara></listitem>
255 <listitem><formalpara><title>relation</title><para>
256 Recognised values are
260 4 (greaterThanOrEqual),
263 </para></formalpara></listitem>
264 <listitem><formalpara><title>which-code</title><para>
265 <literal>known</literal>
268 (the unit-code parameter is taken from the well-known list
269 of alternatives described in below) or
270 <literal>private</literal>
273 (the unit-code paramater has semantics specific to an
274 out-of-band agreement such as a profile).
275 </para></formalpara></listitem>
276 <listitem><formalpara><title>unit-code</title><para>
277 If the which-code parameter is <literal>known</literal>
278 then the recognised values are
290 If which-code is <literal>private</literal> then the
291 acceptable values are determined by the profile.
292 </para></formalpara></listitem>
294 (The numeric values of the relation and well-known unit-code
295 parameters are taken straight from
296 <ulink url="&url.z39.50.proximity.asn1;"
297 >the ASN.1</ulink> of the proximity structure in the standard.)
301 <sect3 id="pqf-examples"><title>PQF queries</title>
303 <example id="example.pqf.simple.terms">
304 <title>PQF queries using simple terms</title>
313 <example id="pqf.example.pqf.boolean.operators">
314 <title>PQF boolean operators</title>
317 @or "dylan" "zimmerman"
319 @and @or dylan zimmerman when
321 @and when @or dylan zimmerman
325 <example id="example.pqf.result.sets">
326 <title>PQF references to result sets</title>
331 @and @set seta @set setb
335 <example id="example.pqf.attributes">
336 <title>Attributes for terms</title>
341 @attr 1=4 @attr 4=1 "self portrait"
343 @attrset exp1 @attr 1=1 CategoryList
345 @attr gils 1=2008 Copenhagen
347 @attr 1=/book/title computer
351 <example id="example.pqf.proximity">
352 <title>PQF Proximity queries</title>
355 @prox 0 3 1 2 k 2 dylan zimmerman
358 Here the parameters 0, 3, 1, 2, k and 2 represent exclusion,
359 distance, ordered, relation, which-code and unit-code, in that
363 exclusion = 0: the proximity condition must hold
366 distance = 3: the terms must be three units apart
369 ordered = 1: they must occur in the order they are specified
372 relation = 2: lessThanOrEqual (to the distance of 3 units)
375 which-code is ``known'', so the standard unit-codes are used
381 So the whole proximity query means that the words
382 <literal>dylan</literal> and <literal>zimmerman</literal> must
383 both occur in the record, in that order, differing in position
384 by three or fewer words (i.e. with two or fewer words between
385 them.) The query would find ``Bob Dylan, aka. Robert
386 Zimmerman'', but not ``Bob Dylan, born as Robert Zimmerman''
387 since the distance in this case is four.
391 <example id="example.pqf.search.term.type">
392 <title>PQF specification of search term type</title>
395 @term string "a UTF-8 string, maybe?"
399 <example id="example.pqf.mixed.queries">
400 <title>PQF mixed queries</title>
403 @or @and bob dylan @set Result-1
405 @attr 4=1 @and @attr 1=1 "bob dylan" @attr 1=4 "slow train coming"
407 @and @attr 2=4 @attr gils 1=2038 -114 @attr 2=2 @attr gils 1=2039 -109
411 The last of these examples is a spatial search: in
412 <ulink url="http://www.gils.net/prof_v2.html#sec_7_4"
413 >the GILS attribute set</ulink>,
415 2038 indicates West Bounding Coordinate and
416 2030 indicates East Bounding Coordinate,
417 so the query is for areas extending from -114 degrees
418 to no more than -109 degrees.
425 <sect2 id="CCL"><title>CCL</title>
428 Not all users enjoy typing in prefix query structures and numerical
429 attribute values, even in a minimalistic test client. In the library
430 world, the more intuitive Common Command Language - CCL (ISO 8777)
431 has enjoyed some popularity - especially before the widespread
432 availability of graphical interfaces. It is still useful in
433 applications where you for some reason or other need to provide a
434 symbolic language for expressing boolean query structures.
437 <sect3 id="ccl.syntax">
438 <title>CCL Syntax</title>
441 The CCL parser obeys the following grammar for the FIND argument.
442 The syntax is annotated by in the lines prefixed by
443 <literal>--</literal>.
447 CCL-Find ::= CCL-Find Op Elements
450 Op ::= "and" | "or" | "not"
451 -- The above means that Elements are separated by boolean operators.
453 Elements ::= '(' CCL-Find ')'
456 | Qualifiers Relation Terms
457 | Qualifiers Relation '(' CCL-Find ')'
458 | Qualifiers '=' string '-' string
459 -- Elements is either a recursive definition, a result set reference, a
460 -- list of terms, qualifiers followed by terms, qualifiers followed
461 -- by a recursive definition or qualifiers in a range (lower - upper).
463 Set ::= 'set' = string
464 -- Reference to a result set
466 Terms ::= Terms Prox Term
468 -- Proximity of terms.
472 -- This basically means that a term may include a blank
474 Qualifiers ::= Qualifiers ',' string
476 -- Qualifiers is a list of strings separated by comma
478 Relation ::= '=' | '>=' | '<=' | '<>' | '>' | '<'
479 -- Relational operators. This really doesn't follow the ISO8777
483 -- Proximity operator
487 <example id="example.ccl.queries">
488 <title>CCL queries</title>
490 The following queries are all valid:
502 (dylan and bob) or set=1
506 Assuming that the qualifiers <literal>ti</literal>,
507 <literal>au</literal>
508 and <literal>date</literal> are defined we may use:
514 au=(bob dylan and slow train coming)
516 date>1980 and (ti=((self portrait)))
522 <sect3 id="ccl.qualifiers">
523 <title>CCL Qualifiers</title>
526 Qualifiers are used to direct the search to a particular searchable
527 index, such as title (ti) and author indexes (au). The CCL standard
528 itself doesn't specify a particular set of qualifiers, but it does
529 suggest a few short-hand notations. You can customize the CCL parser
530 to support a particular set of qualifiers to reflect the current target
531 profile. Traditionally, a qualifier would map to a particular
532 use-attribute within the BIB-1 attribute set. It is also
533 possible to set other attributes, such as the structure
538 A CCL profile is a set of predefined CCL qualifiers that may be
539 read from a file or set in the CCL API.
540 The YAZ client reads its CCL qualifiers from a file named
541 <filename>default.bib</filename>. There are four types of
542 lines in a CCL profile: qualifier specification,
543 qualifier alias, comments and directives.
545 <sect4 id="ccl.qualifier.specification">
546 <title>Qualifier specification</title>
548 A qualifier specification is of the form:
552 <replaceable>qualifier-name</replaceable>
553 [<replaceable>attributeset</replaceable><literal>,</literal>]<replaceable>type</replaceable><literal>=</literal><replaceable>val</replaceable>
554 [<replaceable>attributeset</replaceable><literal>,</literal>]<replaceable>type</replaceable><literal>=</literal><replaceable>val</replaceable> ...
558 where <replaceable>qualifier-name</replaceable> is the name of the
559 qualifier to be used (eg. <literal>ti</literal>),
560 <replaceable>type</replaceable> is attribute type in the attribute
561 set (Bib-1 is used if no attribute set is given) and
562 <replaceable>val</replaceable> is attribute value.
563 The <replaceable>type</replaceable> can be specified as an
564 integer or as it be specified either as a single-letter:
565 <literal>u</literal> for use,
566 <literal>r</literal> for relation,<literal>p</literal> for position,
567 <literal>s</literal> for structure,<literal>t</literal> for truncation
568 or <literal>c</literal> for completeness.
569 The attributes for the special qualifier name <literal>term</literal>
570 are used when no CCL qualifier is given in a query.
571 <table id="ccl.common.bib1.attributes">
572 <title>Common Bib-1 attributes</title>
574 <colspec colwidth="2*" colname="type"></colspec>
575 <colspec colwidth="9*" colname="description"></colspec>
579 <entry>Description</entry>
584 <entry><literal>u=</literal><replaceable>value</replaceable></entry>
586 Use attribute (1). Common use attributes are
587 1 Personal-name, 4 Title, 7 ISBN, 8 ISSN, 30 Date,
588 62 Subject, 1003 Author), 1016 Any. Specify value
594 <entry><literal>r=</literal><replaceable>value</replaceable></entry>
596 Relation attribute (2). Common values are
597 1 <, 2 <=, 3 =, 4 >=, 5 >, 6 <>,
598 100 phonetic, 101 stem, 102 relevance, 103 always matches.
603 <entry><literal>p=</literal><replaceable>value</replaceable></entry>
605 Position attribute (3). Values: 1 first in field, 2
606 first in any subfield, 3 any position in field.
611 <entry><literal>s=</literal><replaceable>value</replaceable></entry>
613 Structure attribute (4). Values: 1 phrase, 2 word,
614 3 key, 4 year, 5 date, 6 word list, 100 date (un),
615 101 name (norm), 102 name (un), 103 structure, 104 urx,
616 105 free-form-text, 106 document-text, 107 local-number,
617 108 string, 109 numeric string.
622 <entry><literal>t=</literal><replaceable>value</replaceable></entry>
624 Truncation attribute (5). Values: 1 right, 2 left,
625 3 left& right, 100 none, 101 process #, 102 regular-1,
626 103 regular-2, 104 CCL.
631 <entry><literal>c=</literal><replaceable>value</replaceable></entry>
633 Completeness attribute (6). Values: 1 incomplete subfield,
634 2 complete subfield, 3 complete field.
643 Refer to <xref linkend="bib1"/> or the complete
644 <ulink url="&url.z39.50.attset.bib1;">list of Bib-1 attributes</ulink>
647 It is also possible to specify non-numeric attribute values,
648 which are used in combination with certain types.
649 The special combinations are:
651 <table id="ccl.special.attribute.combos">
652 <title>Special attribute combos</title>
654 <colspec colwidth="2*" colname="name"></colspec>
655 <colspec colwidth="9*" colname="description"></colspec>
659 <entry>Description</entry>
664 <entry><literal>s=pw</literal></entry><entry>
665 The structure is set to either word or phrase depending
666 on the number of tokens in a term (phrase-word).
670 <entry><literal>s=al</literal></entry><entry>
671 Each token in the term is ANDed. (and-list).
672 This does not set the structure at all.
676 <row><entry><literal>s=ol</literal></entry><entry>
677 Each token in the term is ORed. (or-list).
678 This does not set the structure at all.
682 <row><entry><literal>r=o</literal></entry><entry>
683 Allows ranges and the operators greather-than, less-than, ...
685 This sets Bib-1 relation attribute accordingly (relation
686 ordered). A query construct is only treated as a range if
687 dash is used and that is surrounded by white-space. So
688 <literal>-1980</literal> is treated as term
689 <literal>"-1980"</literal> not <literal><= 1980</literal>.
690 If <literal>- 1980</literal> is used, however, that is
695 <row><entry><literal>r=r</literal></entry><entry>
696 Similar to <literal>r=o</literal> but assumes that terms
697 are non-negative (not prefixed with <literal>-</literal>).
698 Thus, a dash will always be treated as a range.
699 The construct <literal>1980-1990</literal> is
700 treated as a range with <literal>r=r</literal> but as a
701 single term <literal>"1980-1990"</literal> with
702 <literal>r=o</literal>. The special attribute
703 <literal>r=r</literal> is available in YAZ 2.0.24 or later.
707 <row><entry><literal>t=l</literal></entry><entry>
708 Allows term to be left-truncated.
709 If term is of the form <literal>?x</literal>, the resulting
710 Type-1 term is <literal>x</literal> and truncation is left.
714 <row><entry><literal>t=r</literal></entry><entry>
715 Allows term to be right-truncated.
716 If term is of the form <literal>x?</literal>, the resulting
717 Type-1 term is <literal>x</literal> and truncation is right.
721 <row><entry><literal>t=n</literal></entry><entry>
722 If term is does not include <literal>?</literal>, the
723 truncation attribute is set to none (100).
727 <row><entry><literal>t=b</literal></entry><entry>
728 Allows term to be both left&right truncated.
729 If term is of the form <literal>?x?</literal>, the
730 resulting term is <literal>x</literal> and trunctation is
731 set to both left&right.
735 <row><entry><literal>t=x</literal></entry><entry>
736 Allows masking anywhere in a term, thus fully supporting
737 # (mask one character) and ? (zero or more of any).
738 If masking is used, trunction is set to 102 (regexp-1 in term)
739 and the term is converted accordingly to a regular expression.
743 <row><entry><literal>t=z</literal></entry><entry>
744 Allows masking anywhere in a term, thus fully supporting
745 # (mask one character) and ? (zero or more of any).
746 If masking is used, trunction is set to 104 (Z39.58 in term)
747 and the term is converted accordingly to Z39.58 masking term -
748 actually the same truncation as CCL itself.
756 <example id="example.ccl.profile"><title>CCL profile</title>
758 Consider the following definition:
769 <literal>ti</literal> and <literal>au</literal> both set
770 structure attribute to phrase (s=1).
771 <literal>ti</literal>
772 sets the use-attribute to 4. <literal>au</literal> sets the
774 When no qualifiers are used in the query the structure-attribute is
775 set to free-form-text (105) (rule for <literal>term</literal>).
776 The <literal>date</literal> sets the relation attribute to
777 the relation used in the CCL query and sets the use attribute
781 You can combine attributes. To Search for "ranked title" you
784 ti,ranked=knuth computer
786 which will set relation=ranked, use=title, structure=phrase.
793 is a valid query. But
801 <sect4 id="ccl.qualifier.alias">
802 <title>Qualifier alias</title>
804 A qualifier alias is of the form:
807 <replaceable>q</replaceable>
808 <replaceable>q1</replaceable> <replaceable>q2</replaceable> ..
811 which declares <replaceable>q</replaceable> to
812 be an alias for <replaceable>q1</replaceable>,
813 <replaceable>q2</replaceable>... such that the CCL
814 query <replaceable>q=x</replaceable> is equivalent to
815 <replaceable>q1=x or q2=x or ...</replaceable>.
819 <sect4 id="ccl.comments">
820 <title>Comments</title>
822 Lines with white space or lines that begin with
823 character <literal>#</literal> are treated as comments.
827 <sect4 id="ccl.directives">
828 <title>Directives</title>
830 Directive specifications takes the form
832 <para><literal>@</literal><replaceable>directive</replaceable> <replaceable>value</replaceable>
834 <table id="ccl.directives.table">
835 <title>CCL directives</title>
837 <colspec colwidth="2*" colname="name"></colspec>
838 <colspec colwidth="8*" colname="description"></colspec>
839 <colspec colwidth="1*" colname="default"></colspec>
843 <entry>Description</entry>
844 <entry>Default</entry>
849 <entry>truncation</entry>
850 <entry>Truncation character</entry>
851 <entry><literal>?</literal></entry>
855 <entry>Specifies how multiple fields are to be
856 combined. There are two modes: <literal>or</literal>:
857 multiple qualifier fields are ORed,
858 <literal>merge</literal>: attributes for the qualifier
859 fields are merged and assigned to one term.
861 <entry><literal>merge</literal></entry>
865 <entry>Specificies if CCL operatores and qualifiers should be
866 compared with case sensitivity or not. Specify 0 for
867 case sensitive; 1 for case insensitive.</entry>
868 <entry><literal>0</literal></entry>
873 <entry>Specifies token for CCL operator AND.</entry>
874 <entry><literal>and</literal></entry>
879 <entry>Specifies token for CCL operator OR.</entry>
880 <entry><literal>or</literal></entry>
885 <entry>Specifies token for CCL operator NOT.</entry>
886 <entry><literal>not</literal></entry>
891 <entry>Specifies token for CCL operator SET.</entry>
892 <entry><literal>set</literal></entry>
900 <title>CCL API</title>
902 All public definitions can be found in the header file
903 <filename>ccl.h</filename>. A profile identifier is of type
904 <literal>CCL_bibset</literal>. A profile must be created with the call
905 to the function <function>ccl_qual_mk</function> which returns a profile
906 handle of type <literal>CCL_bibset</literal>.
910 To read a file containing qualifier definitions the function
911 <function>ccl_qual_file</function> may be convenient. This function
912 takes an already opened <literal>FILE</literal> handle pointer as
913 argument along with a <literal>CCL_bibset</literal> handle.
917 To parse a simple string with a FIND query use the function
920 struct ccl_rpn_node *ccl_find_str (CCL_bibset bibset, const char *str,
921 int *error, int *pos);
924 which takes the CCL profile (<literal>bibset</literal>) and query
925 (<literal>str</literal>) as input. Upon successful completion the RPN
926 tree is returned. If an error occur, such as a syntax error, the integer
927 pointed to by <literal>error</literal> holds the error code and
928 <literal>pos</literal> holds the offset inside query string in which
933 An English representation of the error may be obtained by calling
934 the <literal>ccl_err_msg</literal> function. The error codes are
935 listed in <filename>ccl.h</filename>.
939 To convert the CCL RPN tree (type
940 <literal>struct ccl_rpn_node *</literal>)
941 to the Z_RPNQuery of YAZ the function <function>ccl_rpn_query</function>
942 must be used. This function which is part of YAZ is implemented in
943 <filename>yaz-ccl.c</filename>.
944 After calling this function the CCL RPN tree is probably no longer
945 needed. The <literal>ccl_rpn_delete</literal> destroys the CCL RPN tree.
949 A CCL profile may be destroyed by calling the
950 <function>ccl_qual_rm</function> function.
954 The token names for the CCL operators may be changed by setting the
955 globals (all type <literal>char *</literal>)
956 <literal>ccl_token_and</literal>, <literal>ccl_token_or</literal>,
957 <literal>ccl_token_not</literal> and <literal>ccl_token_set</literal>.
958 An operator may have aliases, i.e. there may be more than one name for
959 the operator. To do this, separate each alias with a space character.
963 <sect2 id="cql"><title>CQL</title>
965 <ulink url="&url.cql;">CQL</ulink>
966 - Common Query Language - was defined for the
967 <ulink url="&url.sru;">SRU</ulink> protocol.
968 In many ways CQL has a similar syntax to CCL.
969 The objective of CQL is different. Where CCL aims to be
970 an end-user language, CQL is <emphasis>the</emphasis> protocol
971 query language for SRU.
975 If you are new to CQL, read the
976 <ulink url="&url.cql.intro;">Gentle Introduction</ulink>.
980 The CQL parser in &yaz; provides the following:
984 It parses and validates a CQL query.
989 It generates a C structure that allows you to convert
990 a CQL query to some other query language, such as SQL.
995 The parser converts a valid CQL query to PQF, thus providing a
996 way to use CQL for both SRU servers and Z39.50 targets at the
1002 The parser converts CQL to
1003 <ulink url="&url.xcql;">XCQL</ulink>.
1004 XCQL is an XML representation of CQL.
1005 XCQL is part of the SRU specification. However, since SRU
1006 supports CQL only, we don't expect XCQL to be widely used.
1007 Furthermore, CQL has the advantage over XCQL that it is
1013 <sect3 id="cql.parsing"><title>CQL parsing</title>
1015 A CQL parser is represented by the <literal>CQL_parser</literal>
1016 handle. Its contents should be considered &yaz; internal (private).
1018 #include <yaz/cql.h>
1020 typedef struct cql_parser *CQL_parser;
1022 CQL_parser cql_parser_create(void);
1023 void cql_parser_destroy(CQL_parser cp);
1025 A parser is created by <function>cql_parser_create</function> and
1026 is destroyed by <function>cql_parser_destroy</function>.
1029 To parse a CQL query string, the following function
1032 int cql_parser_string(CQL_parser cp, const char *str);
1034 A CQL query is parsed by the <function>cql_parser_string</function>
1035 which takes a query <parameter>str</parameter>.
1036 If the query was valid (no syntax errors), then zero is returned;
1037 otherwise -1 is returned to indicate a syntax error.
1041 int cql_parser_stream(CQL_parser cp,
1042 int (*getbyte)(void *client_data),
1043 void (*ungetbyte)(int b, void *client_data),
1046 int cql_parser_stdio(CQL_parser cp, FILE *f);
1048 The functions <function>cql_parser_stream</function> and
1049 <function>cql_parser_stdio</function> parses a CQL query
1050 - just like <function>cql_parser_string</function>.
1051 The only difference is that the CQL query can be
1052 fed to the parser in different ways.
1053 The <function>cql_parser_stream</function> uses a generic
1054 byte stream as input. The <function>cql_parser_stdio</function>
1055 uses a <literal>FILE</literal> handle which is opened for reading.
1059 <sect3 id="cql.tree"><title>CQL tree</title>
1061 The the query string is valid, the CQL parser
1062 generates a tree representing the structure of the
1067 struct cql_node *cql_parser_result(CQL_parser cp);
1069 <function>cql_parser_result</function> returns the
1070 a pointer to the root node of the resulting tree.
1073 Each node in a CQL tree is represented by a
1074 <literal>struct cql_node</literal>.
1075 It is defined as follows:
1077 #define CQL_NODE_ST 1
1078 #define CQL_NODE_BOOL 2
1088 struct cql_node *modifiers;
1092 struct cql_node *left;
1093 struct cql_node *right;
1094 struct cql_node *modifiers;
1099 There are two node types: search term (ST) and boolean (BOOL).
1100 A modifier is treated as a search term too.
1103 The search term node has five members:
1107 <literal>index</literal>: index for search term.
1108 If an index is unspecified for a search term,
1109 <literal>index</literal> will be NULL.
1114 <literal>index_uri</literal>: index URi for search term
1115 or NULL if none could be resolved for the index.
1120 <literal>term</literal>: the search term itself.
1125 <literal>relation</literal>: relation for search term.
1130 <literal>relation_uri</literal>: relation URI for search term.
1135 <literal>modifiers</literal>: relation modifiers for search
1136 term. The <literal>modifiers</literal> list itself of cql_nodes
1137 each of type <literal>ST</literal>.
1144 The boolean node represents both <literal>and</literal>,
1145 <literal>or</literal>, not as well as
1150 <literal>left</literal> and <literal>right</literal>: left
1151 - and right operand respectively.
1156 <literal>modifiers</literal>: proximity arguments.
1163 <sect3 id="cql.to.pqf"><title>CQL to PQF conversion</title>
1165 Conversion to PQF (and Z39.50 RPN) is tricky by the fact
1166 that the resulting RPN depends on the Z39.50 target
1167 capabilities (combinations of supported attributes).
1168 In addition, the CQL and SRU operates on index prefixes
1169 (URI or strings), whereas the RPN uses Object Identifiers
1173 The CQL library of &yaz; defines a <literal>cql_transform_t</literal>
1174 type. It represents a particular mapping between CQL and RPN.
1175 This handle is created and destroyed by the functions:
1177 cql_transform_t cql_transform_open_FILE (FILE *f);
1178 cql_transform_t cql_transform_open_fname(const char *fname);
1179 void cql_transform_close(cql_transform_t ct);
1181 The first two functions create a tranformation handle from
1182 either an already open FILE or from a filename respectively.
1185 The handle is destroyed by <function>cql_transform_close</function>
1186 in which case no further reference of the handle is allowed.
1189 When a <literal>cql_transform_t</literal> handle has been created
1190 you can convert to RPN.
1192 int cql_transform_buf(cql_transform_t ct,
1193 struct cql_node *cn, char *out, int max);
1195 This function converts the CQL tree <literal>cn</literal>
1196 using handle <literal>ct</literal>.
1197 For the resulting PQF, you supply a buffer <literal>out</literal>
1198 which must be able to hold at at least <literal>max</literal>
1202 If conversion failed, <function>cql_transform_buf</function>
1203 returns a non-zero SRU error code; otherwise zero is returned
1204 (conversion successful). The meanings of the numeric error
1205 codes are listed in the SRU specifications at
1206 <ulink url="&url.sru.diagnostics.list;"/>
1209 If conversion fails, more information can be obtained by calling
1211 int cql_transform_error(cql_transform_t ct, char **addinfop);
1213 This function returns the most recently returned numeric
1214 error-code and sets the string-pointer at
1215 <literal>*addinfop</literal> to point to a string containing
1216 additional information about the error that occurred: for
1217 example, if the error code is 15 (``Illegal or unsupported context
1218 set''), the additional information is the name of the requested
1219 context set that was not recognised.
1222 The SRU error-codes may be translated into brief human-readable
1223 error messages using
1225 const char *cql_strerror(int code);
1229 If you wish to be able to produce a PQF result in a different
1230 way, there are two alternatives.
1232 void cql_transform_pr(cql_transform_t ct,
1233 struct cql_node *cn,
1234 void (*pr)(const char *buf, void *client_data),
1237 int cql_transform_FILE(cql_transform_t ct,
1238 struct cql_node *cn, FILE *f);
1240 The former function produces output to a user-defined
1241 output stream. The latter writes the result to an already
1242 open <literal>FILE</literal>.
1245 <sect3 id="cql.to.rpn">
1246 <title>Specification of CQL to RPN mappings</title>
1248 The file supplied to functions
1249 <function>cql_transform_open_FILE</function>,
1250 <function>cql_transform_open_fname</function> follows
1251 a structure found in many Unix utilities.
1252 It consists of mapping specifications - one per line.
1253 Lines starting with <literal>#</literal> are ignored (comments).
1256 Each line is of the form
1258 <replaceable>CQL pattern</replaceable><literal> = </literal> <replaceable> RPN equivalent</replaceable>
1262 An RPN pattern is a simple attribute list. Each attribute pair
1265 [<replaceable>set</replaceable>] <replaceable>type</replaceable><literal>=</literal><replaceable>value</replaceable>
1267 The attribute <replaceable>set</replaceable> is optional.
1268 The <replaceable>type</replaceable> is the attribute type,
1269 <replaceable>value</replaceable> the attribute value.
1272 The character <literal>*</literal> (asterisk) has special meaning
1273 when used in the RPN pattern.
1274 Each occurrence of <literal>*</literal> is substituted with the
1275 CQL matching name (index, relation, qualifier etc).
1276 This facility can be used to copy a CQL name verbatim to the RPN result.
1279 The following CQL patterns are recognized:
1281 <varlistentry><term>
1282 <literal>index.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
1286 This pattern is invoked when a CQL index, such as
1287 dc.title is converted. <replaceable>set</replaceable>
1288 and <replaceable>name</replaceable> are the context set and index
1290 Typically, the RPN specifies an equivalent use attribute.
1293 For terms not bound by an index the pattern
1294 <literal>index.cql.serverChoice</literal> is used.
1295 Here, the prefix <literal>cql</literal> is defined as
1296 <literal>http://www.loc.gov/zing/cql/cql-indexes/v1.0/</literal>.
1297 If this pattern is not defined, the mapping will fail.
1301 <literal>index.</literal><replaceable>set</replaceable><literal>.*</literal>
1302 is used when no other index pattern is matched.
1306 <varlistentry><term>
1307 <literal>qualifier.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
1312 For backwards compatibility, this is recognised as a synonym of
1313 <literal>index.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
1317 <varlistentry><term>
1318 <literal>relation.</literal><replaceable>relation</replaceable>
1322 This pattern specifies how a CQL relation is mapped to RPN.
1323 <replaceable>pattern</replaceable> is name of relation
1324 operator. Since <literal>=</literal> is used as
1325 separator between CQL pattern and RPN, CQL relations
1326 including <literal>=</literal> cannot be
1327 used directly. To avoid a conflict, the names
1328 <literal>ge</literal>,
1329 <literal>eq</literal>,
1330 <literal>le</literal>,
1331 must be used for CQL operators, greater-than-or-equal,
1332 equal, less-than-or-equal respectively.
1333 The RPN pattern is supposed to include a relation attribute.
1336 For terms not bound by a relation, the pattern
1337 <literal>relation.scr</literal> is used. If the pattern
1338 is not defined, the mapping will fail.
1341 The special pattern, <literal>relation.*</literal> is used
1342 when no other relation pattern is matched.
1347 <varlistentry><term>
1348 <literal>relationModifier.</literal><replaceable>mod</replaceable>
1352 This pattern specifies how a CQL relation modifier is mapped to RPN.
1353 The RPN pattern is usually a relation attribute.
1358 <varlistentry><term>
1359 <literal>structure.</literal><replaceable>type</replaceable>
1363 This pattern specifies how a CQL structure is mapped to RPN.
1364 Note that this CQL pattern is somewhat to similar to
1365 CQL pattern <literal>relation</literal>.
1366 The <replaceable>type</replaceable> is a CQL relation.
1369 The pattern, <literal>structure.*</literal> is used
1370 when no other structure pattern is matched.
1371 Usually, the RPN equivalent specifies a structure attribute.
1376 <varlistentry><term>
1377 <literal>position.</literal><replaceable>type</replaceable>
1381 This pattern specifies how the anchor (position) of
1382 CQL is mapped to RPN.
1383 The <replaceable>type</replaceable> is one
1384 of <literal>first</literal>, <literal>any</literal>,
1385 <literal>last</literal>, <literal>firstAndLast</literal>.
1388 The pattern, <literal>position.*</literal> is used
1389 when no other position pattern is matched.
1394 <varlistentry><term>
1395 <literal>set.</literal><replaceable>prefix</replaceable>
1399 This specification defines a CQL context set for a given prefix.
1400 The value on the right hand side is the URI for the set -
1401 <emphasis>not</emphasis> RPN. All prefixes used in
1402 index patterns must be defined this way.
1407 <varlistentry><term>
1408 <literal>set</literal>
1412 This specification defines a default CQL context set for index names.
1413 The value on the right hand side is the URI for the set.
1420 <example id="example.cql.to.rpn.mapping">
1421 <title>CQL to RPN mapping file</title>
1423 This simple file defines two context sets, three indexes and three
1424 relations, a position pattern and a default structure.
1426 <programlisting><![CDATA[
1427 set.cql = http://www.loc.gov/zing/cql/context-sets/cql/v1.1/
1428 set.dc = http://www.loc.gov/zing/cql/dc-indexes/v1.0/
1430 index.cql.serverChoice = 1=1016
1431 index.dc.title = 1=4
1432 index.dc.subject = 1=21
1438 position.any = 3=3 6=1
1444 With the mappings above, the CQL query
1448 is converted to the PQF:
1450 @attr 1=1016 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "computer"
1452 by rules <literal>index.cql.serverChoice</literal>,
1453 <literal>relation.scr</literal>, <literal>structure.*</literal>,
1454 <literal>position.any</literal>.
1461 is rejected, since <literal>position.right</literal> is
1467 >my = "http://www.loc.gov/zing/cql/dc-indexes/v1.0/" my.title = x
1471 @attr 1=4 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "x"
1475 <example id="example.cql.to.rpn.string">
1476 <title>CQL to RPN string attributes</title>
1478 In this example we allow any index to be passed to RPN as
1481 <programlisting><![CDATA[
1482 # Identifiers for prefixes used in this file. (index.*)
1483 set.cql = info:srw/cql-context-set/1/cql-v1.1
1484 set.rpn = http://bogus/rpn
1485 set = http://bogus/rpn
1487 # The default index when none is specified by the query
1488 index.cql.serverChoice = 1=any
1497 The <literal>http://bogus/rpn</literal> context set is also the default
1498 so we can make queries such as
1502 which is converted to
1504 @attr 2=3 @attr 4=1 @attr 3=3 @attr 1=title "a"
1508 <example id="example.cql.to.rpn.bathprofile">
1509 <title>CQL to RPN using Bath Profile</title>
1511 The file <filename>etc/pqf.properties</filename> has mappings from
1512 the Bath Profile and Dublin Core to RPN.
1513 If YAZ is installed as a package it's usually located
1514 in <filename>/usr/share/yaz/etc</filename> and part of the
1515 development package, such as <literal>libyaz-dev</literal>.
1519 <sect3 id="cql.xcql"><title>CQL to XCQL conversion</title>
1521 Conversion from CQL to XCQL is trivial and does not
1522 require a mapping to be defined.
1523 There three functions to choose from depending on the
1524 way you wish to store the resulting output (XML buffer
1527 int cql_to_xml_buf(struct cql_node *cn, char *out, int max);
1528 void cql_to_xml(struct cql_node *cn,
1529 void (*pr)(const char *buf, void *client_data),
1531 void cql_to_xml_stdio(struct cql_node *cn, FILE *f);
1533 Function <function>cql_to_xml_buf</function> converts
1534 to XCQL and stores result in a user supplied buffer of a given
1538 <function>cql_to_xml</function> writes the result in
1539 a user defined output stream.
1540 <function>cql_to_xml_stdio</function> writes to a
1546 <sect1 id="tools.oid"><title>Object Identifiers</title>
1549 The basic YAZ representation of an OID is an array of integers,
1550 terminated with the value -1. This integer is of type
1551 <literal>Odr_oid</literal>.
1554 Fundamental OID operations and the type <literal>Odr_oid</literal>
1555 are defined in <filename>yaz/oid_util.h</filename>.
1558 An OID can either be declared as a automatic variable or it can
1559 allocated using the memory utilities or ODR/NMEM. It's
1560 guaranteed that an OID can fit in <literal>OID_SIZE</literal> integers.
1562 <example id="tools.oid.bib1.1"><title>Create OID on stack</title>
1564 We can create an OID for the Bib-1 attribute set with:
1566 Odr_oid bib1[OID_SIZE];
1578 And OID may also be filled from a string-based representation using
1579 dots (.). This is achieved by function
1581 int oid_dotstring_to_oid(const char *name, Odr_oid *oid);
1583 This functions returns 0 if name could be converted; -1 otherwise.
1585 <example id="tools.oid.bib1.2"><title>Using oid_oiddotstring_to_oid</title>
1587 We can fill the Bib-1 attribute set OID easier with:
1589 Odr_oid bib1[OID_SIZE];
1590 oid_oiddotstring_to_oid("1.2.840.10003.3.1", bib1);
1595 We can also allocate an OID dynamically on a ODR stream with:
1597 Odr_oid *odr_getoidbystr(ODR o, const char *str);
1599 This creates an OID from string-based representation using dots.
1600 This function take an &odr; stream as parameter. This stream is used to
1601 allocate memory for the data elements, which is released on a
1602 subsequent call to <function>odr_reset()</function> on that stream.
1605 <example id="tools.oid.bib1.3"><title>Using odr_getoidbystr</title>
1607 We can create a OID for the Bib-1 attribute set with:
1609 Odr_oid *bib1 = odr_getoidbystr(odr, "1.2.840.10003.3.1");
1617 char *oid_oid_to_dotstring(const Odr_oid *oid, char *oidbuf)
1619 does the reverse of <function>oid_oiddotstring_to_oid</function>. It
1620 converts an OID to the string-based representation using dots.
1621 The supplied char buffer <literal>oidbuf</literal> holds the resulting
1622 string and must be at least <literal>OID_STR_MAX</literal> in size.
1626 OIDs can be copied with <function>oid_oidcpy</function> which takes
1627 two OID lists as arguments. Alternativly, an OID copy can be allocated
1628 on a ODR stream with:
1630 Odr_oid *odr_oiddup(ODR odr, const Odr_oid *o);
1635 OIDs can be compared with <function>oid_oidcmp</function> which returns
1636 zero if the two OIDs provided are identical; non-zero otherwise.
1639 <sect2 id="tools.oid.database"><title>OID database</title>
1641 From YAZ version 3 and later, the oident system has been replaced
1642 by an OID database. OID database is a misnomer .. the old odient
1643 system was also a database.
1646 The OID database is really just a map between named Object Identifiers
1647 (string) and their OID raw equivalents. Most operations either
1648 convert from string to OID or other way around.
1651 Unfortunately, whenever we supply a string we must also specify the
1652 <emphasis>OID class</emphasis>. The class is necessary because some
1653 strings correspond to multiple OIDs. An example of such a string is
1654 <literal>Bib-1</literal> which may either be an attribute-set
1655 or a diagnostic-set.
1658 Applications using the YAZ database should include
1659 <filename>yaz/oid_db.h</filename>.
1662 A YAZ database handle is of type <literal>yaz_oid_db_t</literal>.
1663 Actually that's a pointer. You need not think deal with that.
1664 YAZ has a built-in database which can be considered "constant" for
1666 We can get hold that by using function <function>yaz_oid_std</function>.
1669 All functions with prefix <function>yaz_string_to_oid</function>
1670 converts from class + string to OID. We have variants of this
1671 operation due to different memory allocation strategies.
1674 All functions with prefix
1675 <function>yaz_oid_to_string</function> converts from OID to string
1679 <example id="tools.oid.bib1.4"><title>Create OID with YAZ DB</title>
1681 We can create an OID for the Bib-1 attribute set on the ODR stream
1685 yaz_string_to_oid_odr(yaz_oid_std(), CLASS_ATTSET, "Bib-1", odr);
1687 This is more complex than using <function>odr_getoidbystr</function>.
1688 You would only use <function>yaz_string_to_oid_odr</function> when the
1689 string (here Bib-1) is supplied by a user or configuration.
1694 <sect2 id="tools.oid.std"><title>Standard OIDs</title>
1697 All the object identifers in the standard OID database as returned
1698 by <function>yaz_oid_std</function> can referenced directly in a
1699 program as a constant OID.
1700 Each constant OID is prefixed with <literal>yaz_oid_</literal> -
1701 followed by OID class (lowercase) - then by OID name (normalized and
1705 See <xref linkend="list-oids"/> for list of all object identifiers
1707 These are declared in <filename>yaz/oid_std.h</filename> but are
1708 included by <filename>yaz/oid_db.h</filename> as well.
1711 <example id="tools.oid.bib1.5"><title>Use a built-in OID</title>
1713 We can allocate our own OID filled with the constant OID for
1716 Odr_oid *bib1 = odr_oiddup(o, yaz_oid_attset_bib1);
1722 <sect1 id="tools.nmem"><title>Nibble Memory</title>
1725 Sometimes when you need to allocate and construct a large,
1726 interconnected complex of structures, it can be a bit of a pain to
1727 release the associated memory again. For the structures describing the
1728 Z39.50 PDUs and related structures, it is convenient to use the
1729 memory-management system of the &odr; subsystem (see
1730 <xref linkend="odr.use"/>). However, in some circumstances
1731 where you might otherwise benefit from using a simple nibble memory
1732 management system, it may be impractical to use
1733 <function>odr_malloc()</function> and <function>odr_reset()</function>.
1734 For this purpose, the memory manager which also supports the &odr;
1735 streams is made available in the NMEM module. The external interface
1736 to this module is given in the <filename>nmem.h</filename> file.
1740 The following prototypes are given:
1744 NMEM nmem_create(void);
1745 void nmem_destroy(NMEM n);
1746 void *nmem_malloc(NMEM n, size_t size);
1747 void nmem_reset(NMEM n);
1748 size_t nmem_total(NMEM n);
1749 void nmem_init(void);
1750 void nmem_exit(void);
1754 The <function>nmem_create()</function> function returns a pointer to a
1755 memory control handle, which can be released again by
1756 <function>nmem_destroy()</function> when no longer needed.
1757 The function <function>nmem_malloc()</function> allocates a block of
1758 memory of the requested size. A call to <function>nmem_reset()</function>
1759 or <function>nmem_destroy()</function> will release all memory allocated
1760 on the handle since it was created (or since the last call to
1761 <function>nmem_reset()</function>. The function
1762 <function>nmem_total()</function> returns the number of bytes currently
1763 allocated on the handle.
1767 The nibble memory pool is shared amongst threads. POSIX
1768 mutex'es and WIN32 Critical sections are introduced to keep the
1769 module thread safe. Function <function>nmem_init()</function>
1770 initializes the nibble memory library and it is called automatically
1771 the first time the <literal>YAZ.DLL</literal> is loaded. &yaz; uses
1772 function <function>DllMain</function> to achieve this. You should
1773 <emphasis>not</emphasis> call <function>nmem_init</function> or
1774 <function>nmem_exit</function> unless you're absolute sure what
1775 you're doing. Note that in previous &yaz; versions you'd have to call
1776 <function>nmem_init</function> yourself.
1781 <sect1 id="tools.log"><title>Log</title>
1783 &yaz; has evolved a fairly complex log system which should be useful both
1784 for debugging &yaz; itself, debugging applications that use &yaz;, and for
1785 production use of those applications.
1788 The log functions are declared in header <filename>yaz/log.h</filename>
1789 and implemented in <filename>src/log.c</filename>.
1790 Due to name clash with syslog and some math utilities the logging
1791 interface has been modified as of YAZ 2.0.29. The obsolete interface
1792 is still available if in header file <filename>yaz/log.h</filename>.
1793 The key points of the interface are:
1796 void yaz_log(int level, const char *fmt, ...)
1798 void yaz_log_init(int level, const char *prefix, const char *name);
1799 void yaz_log_init_file(const char *fname);
1800 void yaz_log_init_level(int level);
1801 void yaz_log_init_prefix(const char *prefix);
1802 void yaz_log_time_format(const char *fmt);
1803 void yaz_log_init_max_size(int mx);
1805 int yaz_log_mask_str(const char *str);
1806 int yaz_log_module_level(const char *name);
1810 The reason for the whole log module is the <function>yaz_log</function>
1811 function. It takes a bitmask indicating the log levels, a
1812 <literal>printf</literal>-like format string, and a variable number of
1817 The <literal>log level</literal> is a bit mask, that says on which level(s)
1818 the log entry should be made, and optionally set some behaviour of the
1819 logging. In the most simple cases, it can be one of <literal>YLOG_FATAL,
1820 YLOG_DEBUG, YLOG_WARN, YLOG_LOG</literal>. Those can be combined with bits
1821 that modify the way the log entry is written:<literal>YLOG_ERRNO,
1822 YLOG_NOTIME, YLOG_FLUSH</literal>.
1823 Most of the rest of the bits are deprecated, and should not be used. Use
1824 the dynamic log levels instead.
1828 Applications that use &yaz;, should not use the LOG_LOG for ordinary
1829 messages, but should make use of the dynamic loglevel system. This consists
1830 of two parts, defining the loglevel and checking it.
1834 To define the log levels, the (main) program should pass a string to
1835 <function>yaz_log_mask_str</function> to define which log levels are to be
1836 logged. This string should be a comma-separated list of log level names,
1837 and can contain both hard-coded names and dynamic ones. The log level
1838 calculation starts with <literal>YLOG_DEFAULT_LEVEL</literal> and adds a bit
1839 for each word it meets, unless the word starts with a '-', in which case it
1840 clears the bit. If the string <literal>'none'</literal> is found,
1841 all bits are cleared. Typically this string comes from the command-line,
1842 often identified by <literal>-v</literal>. The
1843 <function>yaz_log_mask_str</function> returns a log level that should be
1844 passed to <function>yaz_log_init_level</function> for it to take effect.
1848 Each module should check what log bits it should be used, by calling
1849 <function>yaz_log_module_level</function> with a suitable name for the
1850 module. The name is cleared from a preceding path and an extension, if any,
1851 so it is quite possible to use <literal>__FILE__</literal> for it. If the
1852 name has been passed to <function>yaz_log_mask_str</function>, the routine
1853 returns a non-zero bitmask, which should then be used in consequent calls
1854 to yaz_log. (It can also be tested, so as to avoid unnecessary calls to
1855 yaz_log, in time-critical places, or when the log entry would take time
1860 Yaz uses the following dynamic log levels:
1861 <literal>server, session, request, requestdetail</literal> for the server
1863 <literal>zoom</literal> for the zoom client api.
1864 <literal>ztest</literal> for the simple test server.
1865 <literal>malloc, nmem, odr, eventl</literal> for internal debugging of yaz itself.
1866 Of course, any program using yaz is welcome to define as many new ones, as
1871 By default the log is written to stderr, but this can be changed by a call
1872 to <function>yaz_log_init_file</function> or
1873 <function>yaz_log_init</function>. If the log is directed to a file, the
1874 file size is checked at every write, and if it exceeds the limit given in
1875 <function>yaz_log_init_max_size</function>, the log is rotated. The
1876 rotation keeps one old version (with a <literal>.1</literal> appended to
1877 the name). The size defaults to 1GB. Setting it to zero will disable the
1882 A typical yaz-log looks like this
1883 13:23:14-23/11 yaz-ztest(1) [session] Starting session from tcp:127.0.0.1 (pid=30968)
1884 13:23:14-23/11 yaz-ztest(1) [request] Init from 'YAZ' (81) (ver 2.0.28) OK
1885 13:23:17-23/11 yaz-ztest(1) [request] Search Z: @attrset Bib-1 foo OK:7 hits
1886 13:23:22-23/11 yaz-ztest(1) [request] Present: [1] 2+2 OK 2 records returned
1887 13:24:13-23/11 yaz-ztest(1) [request] Close OK
1891 The log entries start with a time stamp. This can be omitted by setting the
1892 <literal>YLOG_NOTIME</literal> bit in the loglevel. This way automatic tests
1893 can be hoped to produce identical log files, that are easy to diff. The
1894 format of the time stamp can be set with
1895 <function>yaz_log_time_format</function>, which takes a format string just
1896 like <function>strftime</function>.
1900 Next in a log line comes the prefix, often the name of the program. For
1901 yaz-based servers, it can also contain the session number. Then
1902 comes one or more logbits in square brackets, depending on the logging
1903 level set by <function>yaz_log_init_level</function> and the loglevel
1904 passed to <function>yaz_log_init_level</function>. Finally comes the format
1905 string and additional values passed to <function>yaz_log</function>
1909 The log level <literal>YLOG_LOGLVL</literal>, enabled by the string
1910 <literal>loglevel</literal>, will log all the log-level affecting
1911 operations. This can come in handy if you need to know what other log
1912 levels would be useful. Grep the logfile for <literal>[loglevel]</literal>.
1916 The log system is almost independent of the rest of &yaz;, the only
1917 important dependence is of <filename>nmem</filename>, and that only for
1918 using the semaphore definition there.
1922 The dynamic log levels and log rotation were introduced in &yaz; 2.0.28. At
1923 the same time, the log bit names were changed from
1924 <literal>LOG_something</literal> to <literal>YLOG_something</literal>,
1925 to avoid collision with <filename>syslog.h</filename>.
1930 <sect1 id="marc"><title>MARC</title>
1933 YAZ provides a fast utility for working with MARC records.
1934 Early versions of the MARC utility only allowed decoding of ISO2709.
1935 Today the utility may both encode - and decode to a varity of formats.
1938 #include <yaz/marcdisp.h>
1940 /* create handler */
1941 yaz_marc_t yaz_marc_create(void);
1943 void yaz_marc_destroy(yaz_marc_t mt);
1945 /* set XML mode YAZ_MARC_LINE, YAZ_MARC_SIMPLEXML, ... */
1946 void yaz_marc_xml(yaz_marc_t mt, int xmlmode);
1947 #define YAZ_MARC_LINE 0
1948 #define YAZ_MARC_SIMPLEXML 1
1949 #define YAZ_MARC_OAIMARC 2
1950 #define YAZ_MARC_MARCXML 3
1951 #define YAZ_MARC_ISO2709 4
1952 #define YAZ_MARC_XCHANGE 5
1953 #define YAZ_MARC_CHECK 6
1954 #define YAZ_MARC_TURBOMARC 7
1956 /* supply iconv handle for character set conversion .. */
1957 void yaz_marc_iconv(yaz_marc_t mt, yaz_iconv_t cd);
1959 /* set debug level, 0=none, 1=more, 2=even more, .. */
1960 void yaz_marc_debug(yaz_marc_t mt, int level);
1962 /* decode MARC in buf of size bsize. Returns >0 on success; <=0 on failure.
1963 On success, result in *result with size *rsize. */
1964 int yaz_marc_decode_buf(yaz_marc_t mt, const char *buf, int bsize,
1965 const char **result, size_t *rsize);
1967 /* decode MARC in buf of size bsize. Returns >0 on success; <=0 on failure.
1968 On success, result in WRBUF */
1969 int yaz_marc_decode_wrbuf(yaz_marc_t mt, const char *buf,
1970 int bsize, WRBUF wrbuf);
1975 The synopsis is just a basic subset of all functionality. Refer
1976 to the actual header file <filename>marcdisp.h</filename> for
1981 A MARC conversion handle must be created by using
1982 <function>yaz_marc_create</function> and destroyed
1983 by calling <function>yaz_marc_destroy</function>.
1986 All other function operate on a <literal>yaz_marc_t</literal> handle.
1987 The output is specified by a call to <function>yaz_marc_xml</function>.
1988 The <literal>xmlmode</literal> must be one of
1991 <term>YAZ_MARC_LINE</term>
1994 A simple line-by-line format suitable for display but not
1995 recommend for further (machine) processing.
2001 <term>YAZ_MARC_MARCXML</term>
2004 <ulink url="&url.marcxml;">MARCXML</ulink>.
2010 <term>YAZ_MARC_ISO2709</term>
2013 ISO2709 (sometimes just referred to as "MARC").
2019 <term>YAZ_MARC_XCHANGE</term>
2022 <ulink url="&url.marcxchange;">MarcXchange</ulink>.
2028 <term>YAZ_MARC_CHECK</term>
2031 Pseudo format for validation only. Does not generate
2032 any real output except diagnostics.
2038 <term>YAZ_MARC_TURBOMARC</term>
2041 XML format with same semantics as MARCXML but more compact
2042 and geared towards fast processing with XSLT. Refer to
2043 <xref linkend="tools.turbomarc"/> for more information.
2051 The actual conversion functions are
2052 <function>yaz_marc_decode_buf</function> and
2053 <function>yaz_marc_decode_wrbuf</function> which decodes and encodes
2054 a MARC record. The former function operates on simple buffers, the
2055 stores the resulting record in a WRBUF handle (WRBUF is a simple string
2058 <example id="example.marc.display">
2059 <title>Display of MARC record</title>
2061 The following program snippet illustrates how the MARC API may
2062 be used to convert a MARC record to the line-by-line format:
2063 <programlisting><![CDATA[
2064 void print_marc(const char *marc_buf, int marc_buf_size)
2066 char *result; /* for result buf */
2067 size_t result_len; /* for size of result */
2068 yaz_marc_t mt = yaz_marc_create();
2069 yaz_marc_xml(mt, YAZ_MARC_LINE);
2070 yaz_marc_decode_buf(mt, marc_buf, marc_buf_size,
2071 &result, &result_len);
2072 fwrite(result, result_len, 1, stdout);
2073 yaz_marc_destroy(mt); /* note that result is now freed... */
2079 <sect2 id="tools.turbomarc">
2080 <title>TurboMARC</title>
2082 TurboMARC is yet another XML encoding of a MARC record. The format
2083 was designed for fast processing with XSLT.
2087 Pazpar2 uses XSLT to convert an XML encoded MARC record to an internal
2088 representation. This conversion mostly check the tag of a MARC field
2089 to determine the basic rules in the conversion. This check is
2090 costly when that is tag is encoded as an attribute in MARCXML.
2091 By having the tag value as the element instead, makes processing
2092 many times faster (at least for Libxslt).
2095 TurboMARC is encoded as follows:
2098 Record elements is part of namespace
2099 "<literal>http://www.indexdata.com/turbomarc</literal>".
2102 A record is enclosed in element <literal>r</literal>.
2105 A collection of records is enclosed in element
2106 <literal>collection</literal>.
2109 The leader is encoded as element <literal>l</literal> with the
2110 leader content as its (text) value.
2113 A control field is encoded as element <literal>c</literal> concatenated
2114 with the tag value of the control field if the tag value
2115 matches the regular expression <literal>[a-zA-Z0-9]*</literal>.
2116 If the tag value do not match the regular expression
2117 <literal>[a-zA-Z0-9]*</literal> the control field is encoded
2118 as element <literal>c</literal> and attribute <literal>code</literal>
2119 will hold the tag value.
2120 This rule ensure that in the rare cases where a tag value might
2121 result in a non-wellformed XML YAZ encode it as a coded attribute
2125 The control field content is the the text value of this element.
2126 Indicators are encoded as attribute names
2127 <literal>i1</literal>, <literal>i2</literal>, etc.. and
2128 corresponding values for each indicator.
2131 A data field is encoded as element <literal>d</literal> concatenated
2132 with the tag value of the data field or using the attribute
2133 <literal>code</literal> as described in the rules for control fields.
2134 The children of the data field element is subfield elements.
2135 Each subfield element is encoded as <literal>s</literal>
2136 concatenated with the sub field code.
2137 The text of the subfield element is the contents of the subfield.
2138 Indicators are encoded as attributes for the data field element similar
2139 to the encoding for control fields.
2146 <sect1 id="tools.retrieval">
2147 <title>Retrieval Facility</title>
2149 YAZ version 2.1.20 or later includes a Retrieval facility tool
2150 which allows a SRU/Z39.50 to describe itself and perform record
2151 conversions. The idea is the following:
2156 An SRU/Z39.50 client sends a retrieval request which includes
2157 a combination of the following parameters: syntax (format),
2158 schema (or element set name).
2164 The retrieval facility is invoked with parameters in a
2165 server/proxy. The retrieval facility matches the parameters a set of
2166 "supported" retrieval types.
2167 If there is no match, the retrieval signals an error
2168 (syntax and / or schema not supported).
2174 For a successful match, the backend is invoked with the same
2175 or altered retrieval parameters (syntax, schema). If
2176 a record is received from the backend, it is converted to the
2177 frontend name / syntax.
2183 The resulting record is sent back the client and tagged with
2184 the frontend syntax / schema.
2191 The Retrieval facility is driven by an XML configuration. The
2192 configuration is neither Z39.50 ZeeRex or SRU ZeeRex. But it
2193 should be easy to generate both of them from the XML configuration.
2194 (unfortunately the two versions
2195 of ZeeRex differ substantially in this regard).
2197 <sect2 id="tools.retrieval.format">
2198 <title>Retrieval XML format</title>
2200 All elements should be covered by namespace
2201 <literal>http://indexdata.com/yaz</literal> .
2202 The root element node must be <literal>retrievalinfo</literal>.
2205 The <literal>retrievalinfo</literal> must include one or
2206 more <literal>retrieval</literal> elements. Each
2207 <literal>retrieval</literal> defines specific combination of
2208 syntax, name and identifier supported by this retrieval service.
2211 The <literal>retrieval</literal> element may include any of the
2212 following attributes:
2214 <varlistentry><term><literal>syntax</literal> (REQUIRED)</term>
2217 Defines the record syntax. Possible values is any
2218 of the names defined in YAZ' OID database or a raw
2223 <varlistentry><term><literal>name</literal> (OPTIONAL)</term>
2226 Defines the name of the retrieval format. This can be
2227 any string. For SRU, the value, is equivalent to schema (short-hand);
2228 for Z39.50 it's equivalent to simple element set name.
2229 For YAZ 3.0.24 and later this name may be specified as a glob
2230 expression with operators
2231 <literal>*</literal> and <literal>?</literal>.
2235 <varlistentry><term><literal>identifier</literal> (OPTIONAL)</term>
2238 Defines the URI schema name of the retrieval format. This can be
2239 any string. For SRU, the value, is equivalent to URI schema.
2240 For Z39.50, there is no equivalent.
2247 The <literal>retrieval</literal> may include one
2248 <literal>backend</literal> element. If a <literal>backend</literal>
2249 element is given, it specifies how the records are retrieved by
2250 some backend and how the records are converted from the backend to
2254 The attributes, <literal>name</literal> and <literal>syntax</literal>
2255 may be specified for the <literal>backend</literal> element. These
2256 semantics of these attributes is equivalent to those for the
2257 <literal>retrieval</literal>. However, these values are passed to
2261 The <literal>backend</literal> element may includes one or more
2262 conversion instructions (as children elements). The supported
2265 <varlistentry><term><literal>marc</literal></term>
2268 The <literal>marc</literal> element specifies a conversion
2269 to - and from ISO2709 encoded MARC and
2270 <ulink url="&url.marcxml;">&acro.marcxml;</ulink>/MarcXchange.
2271 The following attributes may be specified:
2274 <varlistentry><term><literal>inputformat</literal> (REQUIRED)</term>
2277 Format of input. Supported values are
2278 <literal>marc</literal> (for ISO2709); and <literal>xml</literal>
2279 for MARCXML/MarcXchange.
2284 <varlistentry><term><literal>outputformat</literal> (REQUIRED)</term>
2287 Format of output. Supported values are
2288 <literal>line</literal> (MARC line format);
2289 <literal>marcxml</literal> (for MARCXML),
2290 <literal>marc</literal> (ISO2709),
2291 <literal>marcxhcange</literal> (for MarcXchange).
2296 <varlistentry><term><literal>inputcharset</literal> (OPTIONAL)</term>
2299 Encoding of input. For XML input formats, this need not
2300 be given, but for ISO2709 based inputformats, this should
2301 be set to the encoding used. For MARC21 records, a common
2302 inputcharset value would be <literal>marc-8</literal>.
2307 <varlistentry><term><literal>outputcharset</literal> (OPTIONAL)</term>
2310 Encoding of output. If outputformat is XML based, it is
2311 strongly recommened to use <literal>utf-8</literal>.
2320 <varlistentry><term><literal>xslt</literal></term>
2323 The <literal>xslt</literal> element specifies a conversion
2324 via &acro.xslt;. The following attributes may be specified:
2327 <varlistentry><term><literal>stylesheet</literal> (REQUIRED)</term>
2342 <sect2 id="tools.retrieval.examples">
2343 <title>Retrieval Facility Examples</title>
2344 <example id="tools.retrieval.marc21">
2345 <title>MARC21 backend</title>
2347 A typical way to use the retrieval facility is to enable XML
2348 for servers that only supports ISO2709 encoded MARC21 records.
2350 <programlisting><![CDATA[
2352 <retrieval syntax="usmarc" name="F"/>
2353 <retrieval syntax="usmarc" name="B"/>
2354 <retrieval syntax="xml" name="marcxml"
2355 identifier="info:srw/schema/1/marcxml-v1.1">
2356 <backend syntax="usmarc" name="F">
2357 <marc inputformat="marc" outputformat="marcxml"
2358 inputcharset="marc-8"/>
2361 <retrieval syntax="xml" name="dc">
2362 <backend syntax="usmarc" name="F">
2363 <marc inputformat="marc" outputformat="marcxml"
2364 inputcharset="marc-8"/>
2365 <xslt stylesheet="MARC21slim2DC.xsl"/>
2372 This means that our frontend supports:
2376 MARC21 F(ull) records.
2381 MARC21 B(rief) records.
2393 Dublin core records.
2400 <sect2 id="tools.retrieval.api">
2403 It should be easy to use the retrieval systems from applications. Refer
2405 <filename>yaz/retrieval.h</filename> and
2406 <filename>yaz/record_conv.h</filename>.
2412 <!-- Keep this comment at the end of the file
2417 sgml-minimize-attributes:nil
2418 sgml-always-quote-attributes:t
2421 sgml-parent-document: "yaz.xml"
2422 sgml-local-catalogs: nil
2423 sgml-namecase-general:t