1 <!-- $Id: tools.xml,v 1.65 2008-01-09 21:32:27 adam Exp $ -->
2 <chapter id="tools"><title>Supporting Tools</title>
5 In support of the service API - primarily the ASN module, which
6 provides the pro-grammatic interface to the Z39.50 APDUs, &yaz; contains
7 a collection of tools that support the development of applications.
10 <sect1 id="tools.query"><title>Query Syntax Parsers</title>
13 Since the type-1 (RPN) query structure has no direct, useful string
14 representation, every origin application needs to provide some form of
15 mapping from a local query notation or representation to a
16 <token>Z_RPNQuery</token> structure. Some programmers will prefer to
17 construct the query manually, perhaps using
18 <function>odr_malloc()</function> to simplify memory management.
19 The &yaz; distribution includes three separate, query-generating tools
20 that may be of use to you.
23 <sect2 id="PQF"><title>Prefix Query Format</title>
26 Since RPN or reverse polish notation is really just a fancy way of
27 describing a suffix notation format (operator follows operands), it
28 would seem that the confusion is total when we now introduce a prefix
29 notation for RPN. The reason is one of simple laziness - it's somewhat
30 simpler to interpret a prefix format, and this utility was designed
31 for maximum simplicity, to provide a baseline representation for use
32 in simple test applications and scripting environments (like Tcl). The
33 demonstration client included with YAZ uses the PQF.
38 The PQF have been adopted by other parties developing Z39.50
39 software. It is often referred to as Prefix Query Notation
44 The PQF is defined by the pquery module in the YAZ library.
45 There are two sets of function that have similar behavior. First
46 set operates on a PQF parser handle, second set doesn't. First set
47 set of functions are more flexible than the second set. Second set
48 is obsolete and is only provided to ensure backwards compatibility.
51 First set of functions all operate on a PQF parser handle:
54 #include <yaz/pquery.h>
56 YAZ_PQF_Parser yaz_pqf_create (void);
58 void yaz_pqf_destroy (YAZ_PQF_Parser p);
60 Z_RPNQuery *yaz_pqf_parse (YAZ_PQF_Parser p, ODR o, const char *qbuf);
62 Z_AttributesPlusTerm *yaz_pqf_scan (YAZ_PQF_Parser p, ODR o,
63 Odr_oid **attributeSetId, const char *qbuf);
66 int yaz_pqf_error (YAZ_PQF_Parser p, const char **msg, size_t *off);
69 A PQF parser is created and destructed by functions
70 <function>yaz_pqf_create</function> and
71 <function>yaz_pqf_destroy</function> respectively.
72 Function <function>yaz_pqf_parse</function> parses query given
73 by string <literal>qbuf</literal>. If parsing was successful,
74 a Z39.50 RPN Query is returned which is created using ODR stream
75 <literal>o</literal>. If parsing failed, a NULL pointer is
77 Function <function>yaz_pqf_scan</function> takes a scan query in
78 <literal>qbuf</literal>. If parsing was successful, the function
79 returns attributes plus term pointer and modifies
80 <literal>attributeSetId</literal> to hold attribute set for the
81 scan request - both allocated using ODR stream <literal>o</literal>.
82 If parsing failed, yaz_pqf_scan returns a NULL pointer.
83 Error information for bad queries can be obtained by a call to
84 <function>yaz_pqf_error</function> which returns an error code and
85 modifies <literal>*msg</literal> to point to an error description,
86 and modifies <literal>*off</literal> to the offset within last
87 query were parsing failed.
90 The second set of functions are declared as follows:
93 #include <yaz/pquery.h>
95 Z_RPNQuery *p_query_rpn (ODR o, oid_proto proto, const char *qbuf);
97 Z_AttributesPlusTerm *p_query_scan (ODR o, oid_proto proto,
98 Odr_oid **attributeSetP, const char *qbuf);
100 int p_query_attset (const char *arg);
103 The function <function>p_query_rpn()</function> takes as arguments an
104 &odr; stream (see section <link linkend="odr">The ODR Module</link>)
105 to provide a memory source (the structure created is released on
106 the next call to <function>odr_reset()</function> on the stream), a
107 protocol identifier (one of the constants <token>PROTO_Z3950</token> and
108 <token>PROTO_SR</token>), an attribute set reference, and
109 finally a null-terminated string holding the query string.
112 If the parse went well, <function>p_query_rpn()</function> returns a
113 pointer to a <literal>Z_RPNQuery</literal> structure which can be
114 placed directly into a <literal>Z_SearchRequest</literal>.
115 If parsing failed, due to syntax error, a NULL pointer is returned.
118 The <literal>p_query_attset</literal> specifies which attribute set
119 to use if the query doesn't specify one by the
120 <literal>@attrset</literal> operator.
121 The <literal>p_query_attset</literal> returns 0 if the argument is a
122 valid attribute set specifier; otherwise the function returns -1.
126 The grammar of the PQF is as follows:
130 query ::= top-set query-struct.
132 top-set ::= [ '@attrset' string ]
134 query-struct ::= attr-spec | simple | complex | '@term' term-type query
136 attr-spec ::= '@attr' [ string ] string query-struct
138 complex ::= operator query-struct query-struct.
140 operator ::= '@and' | '@or' | '@not' | '@prox' proximity.
142 simple ::= result-set | term.
144 result-set ::= '@set' string.
148 proximity ::= exclusion distance ordered relation which-code unit-code.
150 exclusion ::= '1' | '0' | 'void'.
152 distance ::= integer.
154 ordered ::= '1' | '0'.
156 relation ::= integer.
158 which-code ::= 'known' | 'private' | integer.
160 unit-code ::= integer.
162 term-type ::= 'general' | 'numeric' | 'string' | 'oid' | 'datetime' | 'null'.
166 You will note that the syntax above is a fairly faithful
167 representation of RPN, except for the Attribute, which has been
168 moved a step away from the term, allowing you to associate one or more
169 attributes with an entire query structure. The parser will
170 automatically apply the given attributes to each term as required.
174 The @attr operator is followed by an attribute specification
175 (<literal>attr-spec</literal> above). The specification consists
176 of an optional attribute set, an attribute type-value pair and
177 a sub-query. The attribute type-value pair is packed in one string:
178 an attribute type, an equals sign, and an attribute value, like this:
179 <literal>@attr 1=1003</literal>.
180 The type is always an integer but the value may be either an
181 integer or a string (if it doesn't start with a digit character).
182 A string attribute-value is encoded as a Type-1 ``complex''
183 attribute with the list of values containing the single string
184 specified, and including no semantic indicators.
188 Version 3 of the Z39.50 specification defines various encoding of terms.
189 Use <literal>@term </literal> <replaceable>type</replaceable>
190 <replaceable>string</replaceable>,
191 where type is one of: <literal>general</literal>,
192 <literal>numeric</literal> or <literal>string</literal>
193 (for InternationalString).
194 If no term type has been given, the <literal>general</literal> form
195 is used. This is the only encoding allowed in both versions 2 and 3
196 of the Z39.50 standard.
199 <sect3 id="PQF-prox">
200 <title>Using Proximity Operators with PQF</title>
203 This is an advanced topic, describing how to construct
204 queries that make very specific requirements on the
205 relative location of their operands.
206 You may wish to skip this section and go straight to
207 <link linkend="pqf-examples">the example PQF queries</link>.
212 Most Z39.50 servers do not support proximity searching, or
213 support only a small subset of the full functionality that
214 can be expressed using the PQF proximity operator. Be
215 aware that the ability to <emphasis>express</emphasis> a
216 query in PQF is no guarantee that any given server will
217 be able to <emphasis>execute</emphasis> it.
223 The proximity operator <literal>@prox</literal> is a special
224 and more restrictive version of the conjunction operator
225 <literal>@and</literal>. Its semantics are described in
226 section 3.7.2 (Proximity) of Z39.50 the standard itself, which
227 can be read on-line at
228 <ulink url="&url.z39.50.proximity;"/>
231 In PQF, the proximity operation is represented by a sequence
234 @prox <replaceable>exclusion</replaceable> <replaceable>distance</replaceable> <replaceable>ordered</replaceable> <replaceable>relation</replaceable> <replaceable>which-code</replaceable> <replaceable>unit-code</replaceable>
236 in which the meanings of the parameters are as described in in
237 the standard, and they can take the following values:
239 <listitem><formalpara><title>exclusion</title><para>
240 0 = false (i.e. the proximity condition specified by the
241 remaining parameters must be satisfied) or
242 1 = true (the proximity condition specified by the
243 remaining parameters must <emphasis>not</emphasis> be
245 </para></formalpara></listitem>
246 <listitem><formalpara><title>distance</title><para>
247 An integer specifying the difference between the locations
248 of the operands: e.g. two adjacent words would have
249 distance=1 since their locations differ by one unit.
250 </para></formalpara></listitem>
251 <listitem><formalpara><title>ordered</title><para>
252 1 = ordered (the operands must occur in the order the
253 query specifies them) or
254 0 = unordered (they may appear in either order).
255 </para></formalpara></listitem>
256 <listitem><formalpara><title>relation</title><para>
257 Recognised values are
261 4 (greaterThanOrEqual),
264 </para></formalpara></listitem>
265 <listitem><formalpara><title>which-code</title><para>
266 <literal>known</literal>
269 (the unit-code parameter is taken from the well-known list
270 of alternatives described in below) or
271 <literal>private</literal>
274 (the unit-code paramater has semantics specific to an
275 out-of-band agreement such as a profile).
276 </para></formalpara></listitem>
277 <listitem><formalpara><title>unit-code</title><para>
278 If the which-code parameter is <literal>known</literal>
279 then the recognised values are
291 If which-code is <literal>private</literal> then the
292 acceptable values are determined by the profile.
293 </para></formalpara></listitem>
295 (The numeric values of the relation and well-known unit-code
296 parameters are taken straight from
297 <ulink url="&url.z39.50.proximity.asn1;"
298 >the ASN.1</ulink> of the proximity structure in the standard.)
302 <sect3 id="pqf-examples"><title>PQF queries</title>
304 <example id="example.pqf.simple.terms">
305 <title>PQF queries using simple terms</title>
314 <example id="pqf.example.pqf.boolean.operators">
315 <title>PQF boolean operators</title>
318 @or "dylan" "zimmerman"
320 @and @or dylan zimmerman when
322 @and when @or dylan zimmerman
326 <example id="example.pqf.result.sets">
327 <title>PQF references to result sets</title>
332 @and @set seta @set setb
336 <example id="example.pqf.attributes">
337 <title>Attributes for terms</title>
342 @attr 1=4 @attr 4=1 "self portrait"
344 @attrset exp1 @attr 1=1 CategoryList
346 @attr gils 1=2008 Copenhagen
348 @attr 1=/book/title computer
352 <example id="example.pqf.proximity">
353 <title>PQF Proximity queries</title>
356 @prox 0 3 1 2 k 2 dylan zimmerman
359 Here the parameters 0, 3, 1, 2, k and 2 represent exclusion,
360 distance, ordered, relation, which-code and unit-code, in that
364 exclusion = 0: the proximity condition must hold
367 distance = 3: the terms must be three units apart
370 ordered = 1: they must occur in the order they are specified
373 relation = 2: lessThanOrEqual (to the distance of 3 units)
376 which-code is ``known'', so the standard unit-codes are used
382 So the whole proximity query means that the words
383 <literal>dylan</literal> and <literal>zimmerman</literal> must
384 both occur in the record, in that order, differing in position
385 by three or fewer words (i.e. with two or fewer words between
386 them.) The query would find ``Bob Dylan, aka. Robert
387 Zimmerman'', but not ``Bob Dylan, born as Robert Zimmerman''
388 since the distance in this case is four.
392 <example id="example.pqf.search.term.type">
393 <title>PQF specification of search term type</title>
396 @term string "a UTF-8 string, maybe?"
400 <example id="example.pqf.mixed.queries">
401 <title>PQF mixed queries</title>
404 @or @and bob dylan @set Result-1
406 @attr 4=1 @and @attr 1=1 "bob dylan" @attr 1=4 "slow train coming"
408 @and @attr 2=4 @attr gils 1=2038 -114 @attr 2=2 @attr gils 1=2039 -109
412 The last of these examples is a spatial search: in
413 <ulink url="http://www.gils.net/prof_v2.html#sec_7_4"
414 >the GILS attribute set</ulink>,
416 2038 indicates West Bounding Coordinate and
417 2030 indicates East Bounding Coordinate,
418 so the query is for areas extending from -114 degrees
419 to no more than -109 degrees.
426 <sect2 id="CCL"><title>CCL</title>
429 Not all users enjoy typing in prefix query structures and numerical
430 attribute values, even in a minimalistic test client. In the library
431 world, the more intuitive Common Command Language - CCL (ISO 8777)
432 has enjoyed some popularity - especially before the widespread
433 availability of graphical interfaces. It is still useful in
434 applications where you for some reason or other need to provide a
435 symbolic language for expressing boolean query structures.
438 <sect3 id="ccl.syntax">
439 <title>CCL Syntax</title>
442 The CCL parser obeys the following grammar for the FIND argument.
443 The syntax is annotated by in the lines prefixed by
444 <literal>--</literal>.
448 CCL-Find ::= CCL-Find Op Elements
451 Op ::= "and" | "or" | "not"
452 -- The above means that Elements are separated by boolean operators.
454 Elements ::= '(' CCL-Find ')'
457 | Qualifiers Relation Terms
458 | Qualifiers Relation '(' CCL-Find ')'
459 | Qualifiers '=' string '-' string
460 -- Elements is either a recursive definition, a result set reference, a
461 -- list of terms, qualifiers followed by terms, qualifiers followed
462 -- by a recursive definition or qualifiers in a range (lower - upper).
464 Set ::= 'set' = string
465 -- Reference to a result set
467 Terms ::= Terms Prox Term
469 -- Proximity of terms.
473 -- This basically means that a term may include a blank
475 Qualifiers ::= Qualifiers ',' string
477 -- Qualifiers is a list of strings separated by comma
479 Relation ::= '=' | '>=' | '<=' | '<>' | '>' | '<'
480 -- Relational operators. This really doesn't follow the ISO8777
484 -- Proximity operator
488 <example id="example.ccl.queries">
489 <title>CCL queries</title>
491 The following queries are all valid:
503 (dylan and bob) or set=1
507 Assuming that the qualifiers <literal>ti</literal>,
508 <literal>au</literal>
509 and <literal>date</literal> are defined we may use:
515 au=(bob dylan and slow train coming)
517 date>1980 and (ti=((self portrait)))
523 <sect3 id="ccl.qualifiers">
524 <title>CCL Qualifiers</title>
527 Qualifiers are used to direct the search to a particular searchable
528 index, such as title (ti) and author indexes (au). The CCL standard
529 itself doesn't specify a particular set of qualifiers, but it does
530 suggest a few short-hand notations. You can customize the CCL parser
531 to support a particular set of qualifiers to reflect the current target
532 profile. Traditionally, a qualifier would map to a particular
533 use-attribute within the BIB-1 attribute set. It is also
534 possible to set other attributes, such as the structure
539 A CCL profile is a set of predefined CCL qualifiers that may be
540 read from a file or set in the CCL API.
541 The YAZ client reads its CCL qualifiers from a file named
542 <filename>default.bib</filename>. There are four types of
543 lines in a CCL profile: qualifier specification,
544 qualifier alias, comments and directives.
546 <sect4 id="ccl.qualifier.specification">
547 <title>Qualifier specification</title>
549 A qualifier specification is of the form:
553 <replaceable>qualifier-name</replaceable>
554 [<replaceable>attributeset</replaceable><literal>,</literal>]<replaceable>type</replaceable><literal>=</literal><replaceable>val</replaceable>
555 [<replaceable>attributeset</replaceable><literal>,</literal>]<replaceable>type</replaceable><literal>=</literal><replaceable>val</replaceable> ...
559 where <replaceable>qualifier-name</replaceable> is the name of the
560 qualifier to be used (eg. <literal>ti</literal>),
561 <replaceable>type</replaceable> is attribute type in the attribute
562 set (Bib-1 is used if no attribute set is given) and
563 <replaceable>val</replaceable> is attribute value.
564 The <replaceable>type</replaceable> can be specified as an
565 integer or as it be specified either as a single-letter:
566 <literal>u</literal> for use,
567 <literal>r</literal> for relation,<literal>p</literal> for position,
568 <literal>s</literal> for structure,<literal>t</literal> for truncation
569 or <literal>c</literal> for completeness.
570 The attributes for the special qualifier name <literal>term</literal>
571 are used when no CCL qualifier is given in a query.
572 <table id="ccl.common.bib1.attributes">
573 <title>Common Bib-1 attributes</title>
575 <colspec colwidth="2*" colname="type"></colspec>
576 <colspec colwidth="9*" colname="description"></colspec>
580 <entry>Description</entry>
585 <entry><literal>u=</literal><replaceable>value</replaceable></entry>
587 Use attribute (1). Common use attributes are
588 1 Personal-name, 4 Title, 7 ISBN, 8 ISSN, 30 Date,
589 62 Subject, 1003 Author), 1016 Any. Specify value
595 <entry><literal>r=</literal><replaceable>value</replaceable></entry>
597 Relation attribute (2). Common values are
598 1 <, 2 <=, 3 =, 4 >=, 5 >, 6 <>,
599 100 phonetic, 101 stem, 102 relevance, 103 always matches.
604 <entry><literal>p=</literal><replaceable>value</replaceable></entry>
606 Position attribute (3). Values: 1 first in field, 2
607 first in any subfield, 3 any position in field.
612 <entry><literal>s=</literal><replaceable>value</replaceable></entry>
614 Structure attribute (4). Values: 1 phrase, 2 word,
615 3 key, 4 year, 5 date, 6 word list, 100 date (un),
616 101 name (norm), 102 name (un), 103 structure, 104 urx,
617 105 free-form-text, 106 document-text, 107 local-number,
618 108 string, 109 numeric string.
623 <entry><literal>t=</literal><replaceable>value</replaceable></entry>
625 Truncation attribute (5). Values: 1 right, 2 left,
626 3 left& right, 100 none, 101 process #, 102 regular-1,
627 103 regular-2, 104 CCL.
632 <entry><literal>c=</literal><replaceable>value</replaceable></entry>
634 Completeness attribute (6). Values: 1 incomplete subfield,
635 2 complete subfield, 3 complete field.
644 Refer to the complete
645 <ulink url="&url.z39.50.attset.bib1;">list of Bib-1 attributes</ulink>
648 It is also possible to specify non-numeric attribute values,
649 which are used in combination with certain types.
650 The special combinations are:
652 <table id="ccl.special.attribute.combos">
653 <title>Special attribute combos</title>
655 <colspec colwidth="2*" colname="name"></colspec>
656 <colspec colwidth="9*" colname="description"></colspec>
660 <entry>Description</entry>
665 <entry><literal>s=pw</literal></entry><entry>
666 The structure is set to either word or phrase depending
667 on the number of tokens in a term (phrase-word).
671 <entry><literal>s=al</literal></entry><entry>
672 Each token in the term is ANDed. (and-list).
673 This does not set the structure at all.
677 <row><entry><literal>s=ol</literal></entry><entry>
678 Each token in the term is ORed. (or-list).
679 This does not set the structure at all.
683 <row><entry><literal>r=o</literal></entry><entry>
684 Allows ranges and the operators greather-than, less-than, ...
686 This sets Bib-1 relation attribute accordingly (relation
687 ordered). A query construct is only treated as a range if
688 dash is used and that is surrounded by white-space. So
689 <literal>-1980</literal> is treated as term
690 <literal>"-1980"</literal> not <literal><= 1980</literal>.
691 If <literal>- 1980</literal> is used, however, that is
696 <row><entry><literal>r=r</literal></entry><entry>
697 Similar to <literal>r=o</literal> but assumes that terms
698 are non-negative (not prefixed with <literal>-</literal>).
699 Thus, a dash will always be treated as a range.
700 The construct <literal>1980-1990</literal> is
701 treated as a range with <literal>r=r</literal> but as a
702 single term <literal>"1980-1990"</literal> with
703 <literal>r=o</literal>. The special attribute
704 <literal>r=r</literal> is available in YAZ 2.0.24 or later.
708 <row><entry><literal>t=l</literal></entry><entry>
709 Allows term to be left-truncated.
710 If term is of the form <literal>?x</literal>, the resulting
711 Type-1 term is <literal>x</literal> and truncation is left.
715 <row><entry><literal>t=r</literal></entry><entry>
716 Allows term to be right-truncated.
717 If term is of the form <literal>x?</literal>, the resulting
718 Type-1 term is <literal>x</literal> and truncation is right.
722 <row><entry><literal>t=n</literal></entry><entry>
723 If term is does not include <literal>?</literal>, the
724 truncation attribute is set to none (100).
728 <row><entry><literal>t=b</literal></entry><entry>
729 Allows term to be both left&right truncated.
730 If term is of the form <literal>?x?</literal>, the
731 resulting term is <literal>x</literal> and trunctation is
732 set to both left&right.
739 <example id="example.ccl.profile"><title>CCL profile</title>
741 Consider the following definition:
752 <literal>ti</literal> and <literal>au</literal> both set
753 structure attribute to phrase (s=1).
754 <literal>ti</literal>
755 sets the use-attribute to 4. <literal>au</literal> sets the
757 When no qualifiers are used in the query the structure-attribute is
758 set to free-form-text (105) (rule for <literal>term</literal>).
759 The <literal>date</literal> sets the relation attribute to
760 the relation used in the CCL query and sets the use attribute
764 You can combine attributes. To Search for "ranked title" you
767 ti,ranked=knuth computer
769 which will set relation=ranked, use=title, structure=phrase.
776 is a valid query. But
784 <sect4 id="ccl.qualifier.alias">
785 <title>Qualifier alias</title>
787 A qualifier alias is of the form:
790 <replaceable>q</replaceable>
791 <replaceable>q1</replaceable> <replaceable>q2</replaceable> ..
794 which declares <replaceable>q</replaceable> to
795 be an alias for <replaceable>q1</replaceable>,
796 <replaceable>q2</replaceable>... such that the CCL
797 query <replaceable>q=x</replaceable> is equivalent to
798 <replaceable>q1=x or q2=x or ...</replaceable>.
802 <sect4 id="ccl.comments">
803 <title>Comments</title>
805 Lines with white space or lines that begin with
806 character <literal>#</literal> are treated as comments.
810 <sect4 id="ccl.directives">
811 <title>Directives</title>
813 Directive specifications takes the form
815 <para><literal>@</literal><replaceable>directive</replaceable> <replaceable>value</replaceable>
817 <table id="ccl.directives.table">
818 <title>CCL directives</title>
820 <colspec colwidth="2*" colname="name"></colspec>
821 <colspec colwidth="8*" colname="description"></colspec>
822 <colspec colwidth="1*" colname="default"></colspec>
826 <entry>Description</entry>
827 <entry>Default</entry>
832 <entry>truncation</entry>
833 <entry>Truncation character</entry>
834 <entry><literal>?</literal></entry>
838 <entry>Specifies how multiple fields are to be
839 combined. There are two modes: <literal>or</literal>:
840 multiple qualifier fields are ORed,
841 <literal>merge</literal>: attributes for the qualifier
842 fields are merged and assigned to one term.
844 <entry><literal>merge</literal></entry>
848 <entry>Specificies if CCL operatores and qualifiers should be
849 compared with case sensitivity or not. Specify 0 for
850 case sensitive; 1 for case insensitive.</entry>
851 <entry><literal>0</literal></entry>
856 <entry>Specifies token for CCL operator AND.</entry>
857 <entry><literal>and</literal></entry>
862 <entry>Specifies token for CCL operator OR.</entry>
863 <entry><literal>or</literal></entry>
868 <entry>Specifies token for CCL operator NOT.</entry>
869 <entry><literal>not</literal></entry>
874 <entry>Specifies token for CCL operator SET.</entry>
875 <entry><literal>set</literal></entry>
883 <title>CCL API</title>
885 All public definitions can be found in the header file
886 <filename>ccl.h</filename>. A profile identifier is of type
887 <literal>CCL_bibset</literal>. A profile must be created with the call
888 to the function <function>ccl_qual_mk</function> which returns a profile
889 handle of type <literal>CCL_bibset</literal>.
893 To read a file containing qualifier definitions the function
894 <function>ccl_qual_file</function> may be convenient. This function
895 takes an already opened <literal>FILE</literal> handle pointer as
896 argument along with a <literal>CCL_bibset</literal> handle.
900 To parse a simple string with a FIND query use the function
903 struct ccl_rpn_node *ccl_find_str (CCL_bibset bibset, const char *str,
904 int *error, int *pos);
907 which takes the CCL profile (<literal>bibset</literal>) and query
908 (<literal>str</literal>) as input. Upon successful completion the RPN
909 tree is returned. If an error occur, such as a syntax error, the integer
910 pointed to by <literal>error</literal> holds the error code and
911 <literal>pos</literal> holds the offset inside query string in which
916 An English representation of the error may be obtained by calling
917 the <literal>ccl_err_msg</literal> function. The error codes are
918 listed in <filename>ccl.h</filename>.
922 To convert the CCL RPN tree (type
923 <literal>struct ccl_rpn_node *</literal>)
924 to the Z_RPNQuery of YAZ the function <function>ccl_rpn_query</function>
925 must be used. This function which is part of YAZ is implemented in
926 <filename>yaz-ccl.c</filename>.
927 After calling this function the CCL RPN tree is probably no longer
928 needed. The <literal>ccl_rpn_delete</literal> destroys the CCL RPN tree.
932 A CCL profile may be destroyed by calling the
933 <function>ccl_qual_rm</function> function.
937 The token names for the CCL operators may be changed by setting the
938 globals (all type <literal>char *</literal>)
939 <literal>ccl_token_and</literal>, <literal>ccl_token_or</literal>,
940 <literal>ccl_token_not</literal> and <literal>ccl_token_set</literal>.
941 An operator may have aliases, i.e. there may be more than one name for
942 the operator. To do this, separate each alias with a space character.
946 <sect2 id="cql"><title>CQL</title>
948 <ulink url="&url.cql;">CQL</ulink>
949 - Common Query Language - was defined for the
950 <ulink url="&url.sru;">SRU</ulink> protocol.
951 In many ways CQL has a similar syntax to CCL.
952 The objective of CQL is different. Where CCL aims to be
953 an end-user language, CQL is <emphasis>the</emphasis> protocol
954 query language for SRU.
958 If you are new to CQL, read the
959 <ulink url="&url.cql.intro;">Gentle Introduction</ulink>.
963 The CQL parser in &yaz; provides the following:
967 It parses and validates a CQL query.
972 It generates a C structure that allows you to convert
973 a CQL query to some other query language, such as SQL.
978 The parser converts a valid CQL query to PQF, thus providing a
979 way to use CQL for both SRU servers and Z39.50 targets at the
985 The parser converts CQL to
986 <ulink url="&url.xcql;">XCQL</ulink>.
987 XCQL is an XML representation of CQL.
988 XCQL is part of the SRU specification. However, since SRU
989 supports CQL only, we don't expect XCQL to be widely used.
990 Furthermore, CQL has the advantage over XCQL that it is
996 <sect3 id="cql.parsing"><title>CQL parsing</title>
998 A CQL parser is represented by the <literal>CQL_parser</literal>
999 handle. Its contents should be considered &yaz; internal (private).
1001 #include <yaz/cql.h>
1003 typedef struct cql_parser *CQL_parser;
1005 CQL_parser cql_parser_create(void);
1006 void cql_parser_destroy(CQL_parser cp);
1008 A parser is created by <function>cql_parser_create</function> and
1009 is destroyed by <function>cql_parser_destroy</function>.
1012 To parse a CQL query string, the following function
1015 int cql_parser_string(CQL_parser cp, const char *str);
1017 A CQL query is parsed by the <function>cql_parser_string</function>
1018 which takes a query <parameter>str</parameter>.
1019 If the query was valid (no syntax errors), then zero is returned;
1020 otherwise -1 is returned to indicate a syntax error.
1024 int cql_parser_stream(CQL_parser cp,
1025 int (*getbyte)(void *client_data),
1026 void (*ungetbyte)(int b, void *client_data),
1029 int cql_parser_stdio(CQL_parser cp, FILE *f);
1031 The functions <function>cql_parser_stream</function> and
1032 <function>cql_parser_stdio</function> parses a CQL query
1033 - just like <function>cql_parser_string</function>.
1034 The only difference is that the CQL query can be
1035 fed to the parser in different ways.
1036 The <function>cql_parser_stream</function> uses a generic
1037 byte stream as input. The <function>cql_parser_stdio</function>
1038 uses a <literal>FILE</literal> handle which is opened for reading.
1042 <sect3 id="cql.tree"><title>CQL tree</title>
1044 The the query string is valid, the CQL parser
1045 generates a tree representing the structure of the
1050 struct cql_node *cql_parser_result(CQL_parser cp);
1052 <function>cql_parser_result</function> returns the
1053 a pointer to the root node of the resulting tree.
1056 Each node in a CQL tree is represented by a
1057 <literal>struct cql_node</literal>.
1058 It is defined as follows:
1060 #define CQL_NODE_ST 1
1061 #define CQL_NODE_BOOL 2
1071 struct cql_node *modifiers;
1075 struct cql_node *left;
1076 struct cql_node *right;
1077 struct cql_node *modifiers;
1082 There are two node types: search term (ST) and boolean (BOOL).
1083 A modifier is treated as a search term too.
1086 The search term node has five members:
1090 <literal>index</literal>: index for search term.
1091 If an index is unspecified for a search term,
1092 <literal>index</literal> will be NULL.
1097 <literal>index_uri</literal>: index URi for search term
1098 or NULL if none could be resolved for the index.
1103 <literal>term</literal>: the search term itself.
1108 <literal>relation</literal>: relation for search term.
1113 <literal>relation_uri</literal>: relation URI for search term.
1118 <literal>modifiers</literal>: relation modifiers for search
1119 term. The <literal>modifiers</literal> list itself of cql_nodes
1120 each of type <literal>ST</literal>.
1127 The boolean node represents both <literal>and</literal>,
1128 <literal>or</literal>, not as well as
1133 <literal>left</literal> and <literal>right</literal>: left
1134 - and right operand respectively.
1139 <literal>modifiers</literal>: proximity arguments.
1146 <sect3 id="cql.to.pqf"><title>CQL to PQF conversion</title>
1148 Conversion to PQF (and Z39.50 RPN) is tricky by the fact
1149 that the resulting RPN depends on the Z39.50 target
1150 capabilities (combinations of supported attributes).
1151 In addition, the CQL and SRU operates on index prefixes
1152 (URI or strings), whereas the RPN uses Object Identifiers
1156 The CQL library of &yaz; defines a <literal>cql_transform_t</literal>
1157 type. It represents a particular mapping between CQL and RPN.
1158 This handle is created and destroyed by the functions:
1160 cql_transform_t cql_transform_open_FILE (FILE *f);
1161 cql_transform_t cql_transform_open_fname(const char *fname);
1162 void cql_transform_close(cql_transform_t ct);
1164 The first two functions create a tranformation handle from
1165 either an already open FILE or from a filename respectively.
1168 The handle is destroyed by <function>cql_transform_close</function>
1169 in which case no further reference of the handle is allowed.
1172 When a <literal>cql_transform_t</literal> handle has been created
1173 you can convert to RPN.
1175 int cql_transform_buf(cql_transform_t ct,
1176 struct cql_node *cn, char *out, int max);
1178 This function converts the CQL tree <literal>cn</literal>
1179 using handle <literal>ct</literal>.
1180 For the resulting PQF, you supply a buffer <literal>out</literal>
1181 which must be able to hold at at least <literal>max</literal>
1185 If conversion failed, <function>cql_transform_buf</function>
1186 returns a non-zero SRU error code; otherwise zero is returned
1187 (conversion successful). The meanings of the numeric error
1188 codes are listed in the SRU specifications at
1189 <ulink url="&url.sru.diagnostics.list;"/>
1192 If conversion fails, more information can be obtained by calling
1194 int cql_transform_error(cql_transform_t ct, char **addinfop);
1196 This function returns the most recently returned numeric
1197 error-code and sets the string-pointer at
1198 <literal>*addinfop</literal> to point to a string containing
1199 additional information about the error that occurred: for
1200 example, if the error code is 15 (``Illegal or unsupported context
1201 set''), the additional information is the name of the requested
1202 context set that was not recognised.
1205 The SRU error-codes may be translated into brief human-readable
1206 error messages using
1208 const char *cql_strerror(int code);
1212 If you wish to be able to produce a PQF result in a different
1213 way, there are two alternatives.
1215 void cql_transform_pr(cql_transform_t ct,
1216 struct cql_node *cn,
1217 void (*pr)(const char *buf, void *client_data),
1220 int cql_transform_FILE(cql_transform_t ct,
1221 struct cql_node *cn, FILE *f);
1223 The former function produces output to a user-defined
1224 output stream. The latter writes the result to an already
1225 open <literal>FILE</literal>.
1228 <sect3 id="cql.to.rpn">
1229 <title>Specification of CQL to RPN mappings</title>
1231 The file supplied to functions
1232 <function>cql_transform_open_FILE</function>,
1233 <function>cql_transform_open_fname</function> follows
1234 a structure found in many Unix utilities.
1235 It consists of mapping specifications - one per line.
1236 Lines starting with <literal>#</literal> are ignored (comments).
1239 Each line is of the form
1241 <replaceable>CQL pattern</replaceable><literal> = </literal> <replaceable> RPN equivalent</replaceable>
1245 An RPN pattern is a simple attribute list. Each attribute pair
1248 [<replaceable>set</replaceable>] <replaceable>type</replaceable><literal>=</literal><replaceable>value</replaceable>
1250 The attribute <replaceable>set</replaceable> is optional.
1251 The <replaceable>type</replaceable> is the attribute type,
1252 <replaceable>value</replaceable> the attribute value.
1255 The character <literal>*</literal> (asterisk) has special meaning
1256 when used in the RPN pattern.
1257 Each occurrence of <literal>*</literal> is substituted with the
1258 CQL matching name (index, relation, qualifier etc).
1259 This facility can be used to copy a CQL name verbatim to the RPN result.
1262 The following CQL patterns are recognized:
1264 <varlistentry><term>
1265 <literal>index.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
1269 This pattern is invoked when a CQL index, such as
1270 dc.title is converted. <replaceable>set</replaceable>
1271 and <replaceable>name</replaceable> are the context set and index
1273 Typically, the RPN specifies an equivalent use attribute.
1276 For terms not bound by an index the pattern
1277 <literal>index.cql.serverChoice</literal> is used.
1278 Here, the prefix <literal>cql</literal> is defined as
1279 <literal>http://www.loc.gov/zing/cql/cql-indexes/v1.0/</literal>.
1280 If this pattern is not defined, the mapping will fail.
1284 <literal>index.</literal><replaceable>set</replaceable><literal>.*</literal>
1285 is used when no other index pattern is matched.
1289 <varlistentry><term>
1290 <literal>qualifier.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
1295 For backwards compatibility, this is recognised as a synonym of
1296 <literal>index.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
1300 <varlistentry><term>
1301 <literal>relation.</literal><replaceable>relation</replaceable>
1305 This pattern specifies how a CQL relation is mapped to RPN.
1306 <replaceable>pattern</replaceable> is name of relation
1307 operator. Since <literal>=</literal> is used as
1308 separator between CQL pattern and RPN, CQL relations
1309 including <literal>=</literal> cannot be
1310 used directly. To avoid a conflict, the names
1311 <literal>ge</literal>,
1312 <literal>eq</literal>,
1313 <literal>le</literal>,
1314 must be used for CQL operators, greater-than-or-equal,
1315 equal, less-than-or-equal respectively.
1316 The RPN pattern is supposed to include a relation attribute.
1319 For terms not bound by a relation, the pattern
1320 <literal>relation.scr</literal> is used. If the pattern
1321 is not defined, the mapping will fail.
1324 The special pattern, <literal>relation.*</literal> is used
1325 when no other relation pattern is matched.
1330 <varlistentry><term>
1331 <literal>relationModifier.</literal><replaceable>mod</replaceable>
1335 This pattern specifies how a CQL relation modifier is mapped to RPN.
1336 The RPN pattern is usually a relation attribute.
1341 <varlistentry><term>
1342 <literal>structure.</literal><replaceable>type</replaceable>
1346 This pattern specifies how a CQL structure is mapped to RPN.
1347 Note that this CQL pattern is somewhat to similar to
1348 CQL pattern <literal>relation</literal>.
1349 The <replaceable>type</replaceable> is a CQL relation.
1352 The pattern, <literal>structure.*</literal> is used
1353 when no other structure pattern is matched.
1354 Usually, the RPN equivalent specifies a structure attribute.
1359 <varlistentry><term>
1360 <literal>position.</literal><replaceable>type</replaceable>
1364 This pattern specifies how the anchor (position) of
1365 CQL is mapped to RPN.
1366 The <replaceable>type</replaceable> is one
1367 of <literal>first</literal>, <literal>any</literal>,
1368 <literal>last</literal>, <literal>firstAndLast</literal>.
1371 The pattern, <literal>position.*</literal> is used
1372 when no other position pattern is matched.
1377 <varlistentry><term>
1378 <literal>set.</literal><replaceable>prefix</replaceable>
1382 This specification defines a CQL context set for a given prefix.
1383 The value on the right hand side is the URI for the set -
1384 <emphasis>not</emphasis> RPN. All prefixes used in
1385 index patterns must be defined this way.
1390 <varlistentry><term>
1391 <literal>set</literal>
1395 This specification defines a default CQL context set for index names.
1396 The value on the right hand side is the URI for the set.
1403 <example id="example.cql.to.rpn.mapping">
1404 <title>CQL to RPN mapping file</title>
1406 This simple file defines two context sets, three indexes and three
1407 relations, a position pattern and a default structure.
1409 <programlisting><![CDATA[
1410 set.cql = http://www.loc.gov/zing/cql/context-sets/cql/v1.1/
1411 set.dc = http://www.loc.gov/zing/cql/dc-indexes/v1.0/
1413 index.cql.serverChoice = 1=1016
1414 index.dc.title = 1=4
1415 index.dc.subject = 1=21
1421 position.any = 3=3 6=1
1427 With the mappings above, the CQL query
1431 is converted to the PQF:
1433 @attr 1=1016 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "computer"
1435 by rules <literal>index.cql.serverChoice</literal>,
1436 <literal>relation.scr</literal>, <literal>structure.*</literal>,
1437 <literal>position.any</literal>.
1444 is rejected, since <literal>position.right</literal> is
1450 >my = "http://www.loc.gov/zing/cql/dc-indexes/v1.0/" my.title = x
1454 @attr 1=4 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "x"
1458 <example id="example.cql.to.rpn.string">
1459 <title>CQL to RPN string attributes</title>
1461 In this example we allow any index to be passed to RPN as
1464 <programlisting><![CDATA[
1465 # Identifiers for prefixes used in this file. (index.*)
1466 set.cql = info:srw/cql-context-set/1/cql-v1.1
1467 set.rpn = http://bogus/rpn
1468 set = http://bogus/rpn
1470 # The default index when none is specified by the query
1471 index.cql.serverChoice = 1=any
1480 The <literal>http://bogus/rpn</literal> context set is also the default
1481 so we can make queries such as
1485 which is converted to
1487 @attr 2=3 @attr 4=1 @attr 3=3 @attr 1=title "a"
1491 <example id="example.cql.to.rpn.bathprofile">
1492 <title>CQL to RPN using Bath Profile</title>
1494 The file <filename>etc/pqf.properties</filename> has mappings from
1495 the Bath Profile and Dublin Core to RPN.
1496 If YAZ is installed as a package it's usually located
1497 in <filename>/usr/share/yaz/etc</filename> and part of the
1498 development package, such as <literal>libyaz-dev</literal>.
1502 <sect3 id="cql.xcql"><title>CQL to XCQL conversion</title>
1504 Conversion from CQL to XCQL is trivial and does not
1505 require a mapping to be defined.
1506 There three functions to choose from depending on the
1507 way you wish to store the resulting output (XML buffer
1510 int cql_to_xml_buf(struct cql_node *cn, char *out, int max);
1511 void cql_to_xml(struct cql_node *cn,
1512 void (*pr)(const char *buf, void *client_data),
1514 void cql_to_xml_stdio(struct cql_node *cn, FILE *f);
1516 Function <function>cql_to_xml_buf</function> converts
1517 to XCQL and stores result in a user supplied buffer of a given
1521 <function>cql_to_xml</function> writes the result in
1522 a user defined output stream.
1523 <function>cql_to_xml_stdio</function> writes to a
1529 <sect1 id="tools.oid"><title>Object Identifiers</title>
1532 The basic YAZ representation of an OID is an array of integers,
1533 terminated with the value -1. This integer is of type
1534 <literal>Odr_oid</literal>.
1537 Fundamental OID operations and the type <literal>Odr_oid</literal>
1538 are defined in <filename>yaz/oid_util.h</filename>.
1541 An OID can either be declared as a automatic variable or it can
1542 allocated using the memory utilities or ODR/NMEM. It's
1543 guaranteed that an OID can fit in <literal>OID_SIZE</literal> integers.
1545 <example id="tools.oid.bib1.1"><title>Create OID on stack</title>
1547 We can create an OID for the Bib-1 attribute set with:
1549 Odr_oid bib1[OID_SIZE];
1561 And OID may also be filled from a string-based representation using
1562 dots (.). This is achieved by function
1564 int oid_dotstring_to_oid(const char *name, Odr_oid *oid);
1566 This functions returns 0 if name could be converted; -1 otherwise.
1568 <example id="tools.oid.bib1.2"><title>Using oid_oiddotstring_to_oid</title>
1570 We can fill the Bib-1 attribute set OID easier with:
1572 Odr_oid bib1[OID_SIZE];
1573 oid_oiddotstring_to_oid("1.2.840.10003.3.1", bib1);
1578 We can also allocate an OID dynamically on a ODR stream with:
1580 Odr_oid *odr_getoidbystr(ODR o, const char *str);
1582 This creates an OID from string-based representation using dots.
1583 This function take an &odr; stream as parameter. This stream is used to
1584 allocate memory for the data elements, which is released on a
1585 subsequent call to <function>odr_reset()</function> on that stream.
1588 <example id="tools.oid.bib1.3"><title>Using odr_getoidbystr</title>
1590 We can create a OID for the Bib-1 attribute set with:
1592 Odr_oid *bib1 = odr_getoidbystr(odr, "1.2.840.10003.3.1");
1600 char *oid_oid_to_dotstring(const Odr_oid *oid, char *oidbuf)
1602 does the reverse of <function>oid_oiddotstring_to_oid</function>. It
1603 converts an OID to the string-based representation using dots.
1604 The supplied char buffer <literal>oidbuf</literal> holds the resulting
1605 string and must be at least <literal>OID_STR_MAX</literal> in size.
1609 OIDs can be copied with <function>oid_oidcpy</function> which takes
1610 two OID lists as arguments. Alternativly, an OID copy can be allocated
1611 on a ODR stream with:
1613 Odr_oid *odr_oiddup(ODR odr, const Odr_oid *o);
1618 OIDs can be compared with <function>oid_oidcmp</function> which returns
1619 zero if the two OIDs provided are identical; non-zero otherwise.
1622 <sect2 id="tools.oid.database"><title>OID database</title>
1624 From YAZ version 3 and later, the oident system has been replaced
1625 by an OID database. OID database is a misnomer .. the old odient
1626 system was also a database.
1629 The OID database is really just a map between named Object Identifiers
1630 (string) and their OID raw equivalents. Most operations either
1631 convert from string to OID or other way around.
1634 Unfortunately, whenever we supply a string we must also specify the
1635 <emphasis>OID class</emphasis>. The class is necessary because some
1636 strings correspond to multiple OIDs. An example of such a string is
1637 <literal>Bib-1</literal> which may either be an attribute-set
1638 or a diagnostic-set.
1641 Applications using the YAZ database should include
1642 <filename>yaz/oid_db.h</filename>.
1645 A YAZ database handle is of type <literal>yaz_oid_db_t</literal>.
1646 Actually that's a pointer. You need not think deal with that.
1647 YAZ has a built-in database which can be considered "constant" for
1649 We can get hold that by using function <function>yaz_oid_std</function>.
1652 All functions with prefix <function>yaz_string_to_oid</function>
1653 converts from class + string to OID. We have variants of this
1654 operation due to different memory allocation strategies.
1657 All functions with prefix
1658 <function>yaz_oid_to_string</function> converts from OID to string
1662 <example id="tools.oid.bib1.4"><title>Create OID with YAZ DB</title>
1664 We can create an OID for the Bib-1 attribute set on the ODR stream
1668 yaz_string_to_oid_odr(yaz_oid_std(), CLASS_ATTSET, "Bib-1", odr);
1670 This is more complex than using <function>odr_getoidbystr</function>.
1671 You would only use <function>yaz_string_to_oid_odr</function> when the
1672 string (here Bib-1) is supplied by a user or configuration.
1677 <sect2 id="tools.oid.std"><title>Standard OIDs</title>
1680 All the object identifers in the standard OID database as returned
1681 by <function>yaz_oid_std</function> can referenced directly in a
1682 program as a constant OID.
1683 Each constant OID is prefixed with <literal>yaz_oid_</literal> -
1684 followed by OID class (lowercase) - then by OID name (normalized and
1688 See <xref linkend="list-oids"/> for list of all object identifiers
1690 These are declared in <filename>yaz/oid_std.h</filename> but are
1691 included by <filename>yaz/oid_db.h</filename> as well.
1694 <example id="tools.oid.bib1.5"><title>Use a built-in OID</title>
1696 We can allocate our own OID filled with the constant OID for
1699 Odr_oid *bib1 = odr_oiddup(o, yaz_oid_attset_bib1);
1705 <sect1 id="tools.nmem"><title>Nibble Memory</title>
1708 Sometimes when you need to allocate and construct a large,
1709 interconnected complex of structures, it can be a bit of a pain to
1710 release the associated memory again. For the structures describing the
1711 Z39.50 PDUs and related structures, it is convenient to use the
1712 memory-management system of the &odr; subsystem (see
1713 <xref linkend="odr.use"/>). However, in some circumstances
1714 where you might otherwise benefit from using a simple nibble memory
1715 management system, it may be impractical to use
1716 <function>odr_malloc()</function> and <function>odr_reset()</function>.
1717 For this purpose, the memory manager which also supports the &odr;
1718 streams is made available in the NMEM module. The external interface
1719 to this module is given in the <filename>nmem.h</filename> file.
1723 The following prototypes are given:
1727 NMEM nmem_create(void);
1728 void nmem_destroy(NMEM n);
1729 void *nmem_malloc(NMEM n, int size);
1730 void nmem_reset(NMEM n);
1731 int nmem_total(NMEM n);
1732 void nmem_init(void);
1733 void nmem_exit(void);
1737 The <function>nmem_create()</function> function returns a pointer to a
1738 memory control handle, which can be released again by
1739 <function>nmem_destroy()</function> when no longer needed.
1740 The function <function>nmem_malloc()</function> allocates a block of
1741 memory of the requested size. A call to <function>nmem_reset()</function>
1742 or <function>nmem_destroy()</function> will release all memory allocated
1743 on the handle since it was created (or since the last call to
1744 <function>nmem_reset()</function>. The function
1745 <function>nmem_total()</function> returns the number of bytes currently
1746 allocated on the handle.
1750 The nibble memory pool is shared amongst threads. POSIX
1751 mutex'es and WIN32 Critical sections are introduced to keep the
1752 module thread safe. Function <function>nmem_init()</function>
1753 initializes the nibble memory library and it is called automatically
1754 the first time the <literal>YAZ.DLL</literal> is loaded. &yaz; uses
1755 function <function>DllMain</function> to achieve this. You should
1756 <emphasis>not</emphasis> call <function>nmem_init</function> or
1757 <function>nmem_exit</function> unless you're absolute sure what
1758 you're doing. Note that in previous &yaz; versions you'd have to call
1759 <function>nmem_init</function> yourself.
1764 <sect1 id="tools.log"><title>Log</title>
1766 &yaz; has evolved a fairly complex log system which should be useful both
1767 for debugging &yaz; itself, debugging applications that use &yaz;, and for
1768 production use of those applications.
1771 The log functions are declared in header <filename>yaz/log.h</filename>
1772 and implemented in <filename>src/log.c</filename>.
1773 Due to name clash with syslog and some math utilities the logging
1774 interface has been modified as of YAZ 2.0.29. The obsolete interface
1775 is still available if in header file <filename>yaz/log.h</filename>.
1776 The key points of the interface are:
1779 void yaz_log(int level, const char *fmt, ...)
1781 void yaz_log_init(int level, const char *prefix, const char *name);
1782 void yaz_log_init_file(const char *fname);
1783 void yaz_log_init_level(int level);
1784 void yaz_log_init_prefix(const char *prefix);
1785 void yaz_log_time_format(const char *fmt);
1786 void yaz_log_init_max_size(int mx);
1788 int yaz_log_mask_str(const char *str);
1789 int yaz_log_module_level(const char *name);
1793 The reason for the whole log module is the <function>yaz_log</function>
1794 function. It takes a bitmask indicating the log levels, a
1795 <literal>printf</literal>-like format string, and a variable number of
1800 The <literal>log level</literal> is a bit mask, that says on which level(s)
1801 the log entry should be made, and optionally set some behaviour of the
1802 logging. In the most simple cases, it can be one of <literal>YLOG_FATAL,
1803 YLOG_DEBUG, YLOG_WARN, YLOG_LOG</literal>. Those can be combined with bits
1804 that modify the way the log entry is written:<literal>YLOG_ERRNO,
1805 YLOG_NOTIME, YLOG_FLUSH</literal>.
1806 Most of the rest of the bits are deprecated, and should not be used. Use
1807 the dynamic log levels instead.
1811 Applications that use &yaz;, should not use the LOG_LOG for ordinary
1812 messages, but should make use of the dynamic loglevel system. This consists
1813 of two parts, defining the loglevel and checking it.
1817 To define the log levels, the (main) program should pass a string to
1818 <function>yaz_log_mask_str</function> to define which log levels are to be
1819 logged. This string should be a comma-separated list of log level names,
1820 and can contain both hard-coded names and dynamic ones. The log level
1821 calculation starts with <literal>YLOG_DEFAULT_LEVEL</literal> and adds a bit
1822 for each word it meets, unless the word starts with a '-', in which case it
1823 clears the bit. If the string <literal>'none'</literal> is found,
1824 all bits are cleared. Typically this string comes from the command-line,
1825 often identified by <literal>-v</literal>. The
1826 <function>yaz_log_mask_str</function> returns a log level that should be
1827 passed to <function>yaz_log_init_level</function> for it to take effect.
1831 Each module should check what log bits it should be used, by calling
1832 <function>yaz_log_module_level</function> with a suitable name for the
1833 module. The name is cleared from a preceding path and an extension, if any,
1834 so it is quite possible to use <literal>__FILE__</literal> for it. If the
1835 name has been passed to <function>yaz_log_mask_str</function>, the routine
1836 returns a non-zero bitmask, which should then be used in consequent calls
1837 to yaz_log. (It can also be tested, so as to avoid unnecessary calls to
1838 yaz_log, in time-critical places, or when the log entry would take time
1843 Yaz uses the following dynamic log levels:
1844 <literal>server, session, request, requestdetail</literal> for the server
1846 <literal>zoom</literal> for the zoom client api.
1847 <literal>ztest</literal> for the simple test server.
1848 <literal>malloc, nmem, odr, eventl</literal> for internal debugging of yaz itself.
1849 Of course, any program using yaz is welcome to define as many new ones, as
1854 By default the log is written to stderr, but this can be changed by a call
1855 to <function>yaz_log_init_file</function> or
1856 <function>yaz_log_init</function>. If the log is directed to a file, the
1857 file size is checked at every write, and if it exceeds the limit given in
1858 <function>yaz_log_init_max_size</function>, the log is rotated. The
1859 rotation keeps one old version (with a <literal>.1</literal> appended to
1860 the name). The size defaults to 1GB. Setting it to zero will disable the
1865 A typical yaz-log looks like this
1866 13:23:14-23/11 yaz-ztest(1) [session] Starting session from tcp:127.0.0.1 (pid=30968)
1867 13:23:14-23/11 yaz-ztest(1) [request] Init from 'YAZ' (81) (ver 2.0.28) OK
1868 13:23:17-23/11 yaz-ztest(1) [request] Search Z: @attrset Bib-1 foo OK:7 hits
1869 13:23:22-23/11 yaz-ztest(1) [request] Present: [1] 2+2 OK 2 records returned
1870 13:24:13-23/11 yaz-ztest(1) [request] Close OK
1874 The log entries start with a time stamp. This can be omitted by setting the
1875 <literal>YLOG_NOTIME</literal> bit in the loglevel. This way automatic tests
1876 can be hoped to produce identical log files, that are easy to diff. The
1877 format of the time stamp can be set with
1878 <function>yaz_log_time_format</function>, which takes a format string just
1879 like <function>strftime</function>.
1883 Next in a log line comes the prefix, often the name of the program. For
1884 yaz-based servers, it can also contain the session number. Then
1885 comes one or more logbits in square brackets, depending on the logging
1886 level set by <function>yaz_log_init_level</function> and the loglevel
1887 passed to <function>yaz_log_init_level</function>. Finally comes the format
1888 string and additional values passed to <function>yaz_log</function>
1892 The log level <literal>YLOG_LOGLVL</literal>, enabled by the string
1893 <literal>loglevel</literal>, will log all the log-level affecting
1894 operations. This can come in handy if you need to know what other log
1895 levels would be useful. Grep the logfile for <literal>[loglevel]</literal>.
1899 The log system is almost independent of the rest of &yaz;, the only
1900 important dependence is of <filename>nmem</filename>, and that only for
1901 using the semaphore definition there.
1905 The dynamic log levels and log rotation were introduced in &yaz; 2.0.28. At
1906 the same time, the log bit names were changed from
1907 <literal>LOG_something</literal> to <literal>YLOG_something</literal>,
1908 to avoid collision with <filename>syslog.h</filename>.
1913 <sect1 id="marc"><title>MARC</title>
1916 YAZ provides a fast utility that decodes MARC records and
1917 encodes to a varity of output formats. The MARC records must
1918 be encoded in ISO2709.
1921 #include <yaz/marcdisp.h>
1923 /* create handler */
1924 yaz_marc_t yaz_marc_create(void);
1926 void yaz_marc_destroy(yaz_marc_t mt);
1928 /* set XML mode YAZ_MARC_LINE, YAZ_MARC_SIMPLEXML, ... */
1929 void yaz_marc_xml(yaz_marc_t mt, int xmlmode);
1930 #define YAZ_MARC_LINE 0
1931 #define YAZ_MARC_SIMPLEXML 1
1932 #define YAZ_MARC_OAIMARC 2
1933 #define YAZ_MARC_MARCXML 3
1934 #define YAZ_MARC_ISO2709 4
1935 #define YAZ_MARC_XCHANGE 5
1937 /* supply iconv handle for character set conversion .. */
1938 void yaz_marc_iconv(yaz_marc_t mt, yaz_iconv_t cd);
1940 /* set debug level, 0=none, 1=more, 2=even more, .. */
1941 void yaz_marc_debug(yaz_marc_t mt, int level);
1943 /* decode MARC in buf of size bsize. Returns >0 on success; <=0 on failure.
1944 On success, result in *result with size *rsize. */
1945 int yaz_marc_decode_buf (yaz_marc_t mt, const char *buf, int bsize,
1946 char **result, int *rsize);
1948 /* decode MARC in buf of size bsize. Returns >0 on success; <=0 on failure.
1949 On success, result in WRBUF */
1950 int yaz_marc_decode_wrbuf (yaz_marc_t mt, const char *buf,
1951 int bsize, WRBUF wrbuf);
1955 A MARC conversion handle must be created by using
1956 <function>yaz_marc_create</function> and destroyed
1957 by calling <function>yaz_marc_destroy</function>.
1960 All other function operate on a <literal>yaz_marc_t</literal> handle.
1961 The output is specified by a call to <function>yaz_marc_xml</function>.
1962 The <literal>xmlmode</literal> must be one of
1965 <term>YAZ_MARC_LINE</term>
1968 A simple line-by-line format suitable for display but not
1969 recommend for further (machine) processing.
1975 <term>YAZ_MARC_MARCXML</term>
1978 The resulting record is converted to MARCXML.
1984 <term>YAZ_MARC_ISO2709</term>
1987 The resulting record is converted to ISO2709 (MARC).
1994 The actual conversion functions are
1995 <function>yaz_marc_decode_buf</function> and
1996 <function>yaz_marc_decode_wrbuf</function> which decodes and encodes
1997 a MARC record. The former function operates on simple buffers, the
1998 stores the resulting record in a WRBUF handle (WRBUF is a simple string
2001 <example id="example.marc.display">
2002 <title>Display of MARC record</title>
2004 The followint program snippet illustrates how the MARC API may
2005 be used to convert a MARC record to the line-by-line format:
2006 <programlisting><![CDATA[
2007 void print_marc(const char *marc_buf, int marc_buf_size)
2009 char *result; /* for result buf */
2010 int result_len; /* for size of result */
2011 yaz_marc_t mt = yaz_marc_create();
2012 yaz_marc_xml(mt, YAZ_MARC_LINE);
2013 yaz_marc_decode_buf(mt, marc_buf, marc_buf_size,
2014 &result, &result_len);
2015 fwrite(result, result_len, 1, stdout);
2016 yaz_marc_destroy(mt); /* note that result is now freed... */
2024 <sect1 id="tools.retrieval">
2025 <title>Retrieval Facility</title>
2027 YAZ version 2.1.20 or later includes a Retrieval facility tool
2028 which allows a SRU/Z39.50 to describe itself and perform record
2029 conversions. The idea is the following:
2034 An SRU/Z39.50 client sends a retrieval request which includes
2035 a combination of the following parameters: syntax (format),
2036 schema (or element set name).
2042 The retrieval facility is invoked with parameters in a
2043 server/proxy. The retrieval facility matches the parameters a set of
2044 "supported" retrieval types.
2045 If there is no match, the retrieval signals an error
2046 (syntax and / or schema not supported).
2052 For a successful match, the backend is invoked with the same
2053 or altered retrieval parameters (syntax, schema). If
2054 a record is received from the backend, it is converted to the
2055 frontend name / syntax.
2061 The resulting record is sent back the client and tagged with
2062 the frontend syntax / schema.
2069 The Retrieval facility is driven by an XML configuration. The
2070 configuration is neither Z39.50 ZeeRex or SRU ZeeRex. But it
2071 should be easy to generate both of them from the XML configuration.
2072 (unfortunately the two versions
2073 of ZeeRex differ substantially in this regard).
2075 <sect2 id="tools.retrieval.format">
2076 <title>Retrieval XML format</title>
2078 All elements should be covered by namespace
2079 <literal>http://indexdata.com/yaz</literal> .
2080 The root element node must be <literal>retrievalinfo</literal>.
2083 The <literal>retrievalinfo</literal> must include one or
2084 more <literal>retrieval</literal> elements. Each
2085 <literal>retrieval</literal> defines specific combination of
2086 syntax, name and identifier supported by this retrieval service.
2089 The <literal>retrieval</literal> element may include any of the
2090 following attributes:
2092 <varlistentry><term><literal>syntax</literal> (REQUIRED)</term>
2095 Defines the record syntax. Possible values is any
2096 of the names defined in YAZ' OID database or a raw
2101 <varlistentry><term><literal>name</literal> (OPTIONAL)</term>
2104 Defines the name of the retrieval format. This can be
2105 any string. For SRU, the value, is equivalent to schema (short-hand);
2106 for Z39.50 it's equivalent to simple element set name.
2110 <varlistentry><term><literal>identifier</literal> (OPTIONAL)</term>
2113 Defines the URI schema name of the retrieval format. This can be
2114 any string. For SRU, the value, is equivalent to URI schema.
2115 For Z39.50, there is no equivalent.
2122 The <literal>retrieval</literal> may include one
2123 <literal>backend</literal> element. If a <literal>backend</literal>
2124 element is given, it specifies how the records are retrieved by
2125 some backend and how the records are converted from the backend to
2129 The attributes, <literal>name</literal> and <literal>syntax</literal>
2130 may be specified for the <literal>backend</literal> element. These
2131 semantics of these attributes is equivalent to those for the
2132 <literal>retrieval</literal>. However, these values are passed to
2136 The <literal>backend</literal> element may includes one or more
2137 conversion instructions (as children elements). The supported
2140 <varlistentry><term><literal>marc</literal></term>
2143 The <literal>marc</literal> element specifies a conversion
2144 to - and from ISO2709 encoded MARC and
2145 <ulink url="&url.marcxml;">&acro.marcxml;</ulink>/MarcXchange.
2146 The following attributes may be specified:
2149 <varlistentry><term><literal>inputformat</literal> (REQUIRED)</term>
2152 Format of input. Supported values are
2153 <literal>marc</literal> (for ISO2709); and <literal>xml</literal>
2154 for MARCXML/MarcXchange.
2159 <varlistentry><term><literal>outputformat</literal> (REQUIRED)</term>
2162 Format of output. Supported values are
2163 <literal>line</literal> (MARC line format);
2164 <literal>marcxml</literal> (for MARCXML),
2165 <literal>marc</literal> (ISO2709),
2166 <literal>marcxhcange</literal> (for MarcXchange).
2171 <varlistentry><term><literal>inputcharset</literal> (OPTIONAL)</term>
2174 Encoding of input. For XML input formats, this need not
2175 be given, but for ISO2709 based inputformats, this should
2176 be set to the encoding used. For MARC21 records, a common
2177 inputcharset value would be <literal>marc-8</literal>.
2182 <varlistentry><term><literal>outputcharset</literal> (OPTIONAL)</term>
2185 Encoding of output. If outputformat is XML based, it is
2186 strongly recommened to use <literal>utf-8</literal>.
2195 <varlistentry><term><literal>xslt</literal></term>
2198 The <literal>xslt</literal> element specifies a conversion
2199 via &acro.xslt;. The following attributes may be specified:
2202 <varlistentry><term><literal>stylesheet</literal> (REQUIRED)</term>
2217 <sect2 id="tools.retrieval.examples">
2218 <title>Retrieval Facility Examples</title>
2219 <example id="tools.retrieval.marc21">
2220 <title>MARC21 backend</title>
2222 A typical way to use the retrieval facility is to enable XML
2223 for servers that only supports ISO2709 encoded MARC21 records.
2225 <programlisting><![CDATA[
2227 <retrieval syntax="usmarc" name="F"/>
2228 <retrieval syntax="usmarc" name="B"/>
2229 <retrieval syntax="xml" name="marcxml"
2230 identifier="info:srw/schema/1/marcxml-v1.1">
2231 <backend syntax="usmarc" name="F">
2232 <marc inputformat="marc" outputformat="marcxml"
2233 inputcharset="marc-8"/>
2236 <retrieval syntax="xml" name="dc">
2237 <backend syntax="usmarc" name="F">
2238 <marc inputformat="marc" outputformat="marcxml"
2239 inputcharset="marc-8"/>
2240 <xslt stylesheet="MARC21slim2DC.xsl"/>
2247 This means that our frontend supports:
2251 MARC21 F(ull) records.
2256 MARC21 B(rief) records.
2268 Dublin core records.
2275 <sect2 id="tools.retrieval.api">
2278 It should be easy to use the retrieval systems from applications. Refer
2280 <filename>yaz/retrieval.h</filename> and
2281 <filename>yaz/record_conv.h</filename>.
2287 <!-- Keep this comment at the end of the file
2292 sgml-minimize-attributes:nil
2293 sgml-always-quote-attributes:t
2296 sgml-parent-document: "yaz.xml"
2297 sgml-local-catalogs: nil
2298 sgml-namecase-general:t