1 <!-- $Id: tools.xml,v 1.46 2005-04-26 19:51:31 adam Exp $ -->
2 <chapter id="tools"><title>Supporting Tools</title>
5 In support of the service API - primarily the ASN module, which
6 provides the pro-grammatic interface to the Z39.50 APDUs, &yaz; contains
7 a collection of tools that support the development of applications.
10 <sect1 id="tools.query"><title>Query Syntax Parsers</title>
13 Since the type-1 (RPN) query structure has no direct, useful string
14 representation, every origin application needs to provide some form of
15 mapping from a local query notation or representation to a
16 <token>Z_RPNQuery</token> structure. Some programmers will prefer to
17 construct the query manually, perhaps using
18 <function>odr_malloc()</function> to simplify memory management.
19 The &yaz; distribution includes three separate, query-generating tools
20 that may be of use to you.
23 <sect2 id="PQF"><title>Prefix Query Format</title>
26 Since RPN or reverse polish notation is really just a fancy way of
27 describing a suffix notation format (operator follows operands), it
28 would seem that the confusion is total when we now introduce a prefix
29 notation for RPN. The reason is one of simple laziness - it's somewhat
30 simpler to interpret a prefix format, and this utility was designed
31 for maximum simplicity, to provide a baseline representation for use
32 in simple test applications and scripting environments (like Tcl). The
33 demonstration client included with YAZ uses the PQF.
38 The PQF have been adopted by other parties developing Z39.50
39 software. It is often referred to as Prefix Query Notation
44 The PQF is defined by the pquery module in the YAZ library.
45 There are two sets of function that have similar behavior. First
46 set operates on a PQF parser handle, second set doesn't. First set
47 set of functions are more flexible than the second set. Second set
48 is obsolete and is only provided to ensure backwards compatibility.
51 First set of functions all operate on a PQF parser handle:
54 #include <yaz/pquery.h>
56 YAZ_PQF_Parser yaz_pqf_create (void);
58 void yaz_pqf_destroy (YAZ_PQF_Parser p);
60 Z_RPNQuery *yaz_pqf_parse (YAZ_PQF_Parser p, ODR o, const char *qbuf);
62 Z_AttributesPlusTerm *yaz_pqf_scan (YAZ_PQF_Parser p, ODR o,
63 Odr_oid **attributeSetId, const char *qbuf);
66 int yaz_pqf_error (YAZ_PQF_Parser p, const char **msg, size_t *off);
69 A PQF parser is created and destructed by functions
70 <function>yaz_pqf_create</function> and
71 <function>yaz_pqf_destroy</function> respectively.
72 Function <function>yaz_pqf_parse</function> parses query given
73 by string <literal>qbuf</literal>. If parsing was successful,
74 a Z39.50 RPN Query is returned which is created using ODR stream
75 <literal>o</literal>. If parsing failed, a NULL pointer is
77 Function <function>yaz_pqf_scan</function> takes a scan query in
78 <literal>qbuf</literal>. If parsing was successful, the function
79 returns attributes plus term pointer and modifies
80 <literal>attributeSetId</literal> to hold attribute set for the
81 scan request - both allocated using ODR stream <literal>o</literal>.
82 If parsing failed, yaz_pqf_scan returns a NULL pointer.
83 Error information for bad queries can be obtained by a call to
84 <function>yaz_pqf_error</function> which returns an error code and
85 modifies <literal>*msg</literal> to point to an error description,
86 and modifies <literal>*off</literal> to the offset within last
87 query were parsing failed.
90 The second set of functions are declared as follows:
93 #include <yaz/pquery.h>
95 Z_RPNQuery *p_query_rpn (ODR o, oid_proto proto, const char *qbuf);
97 Z_AttributesPlusTerm *p_query_scan (ODR o, oid_proto proto,
98 Odr_oid **attributeSetP, const char *qbuf);
100 int p_query_attset (const char *arg);
103 The function <function>p_query_rpn()</function> takes as arguments an
104 &odr; stream (see section <link linkend="odr">The ODR Module</link>)
105 to provide a memory source (the structure created is released on
106 the next call to <function>odr_reset()</function> on the stream), a
107 protocol identifier (one of the constants <token>PROTO_Z3950</token> and
108 <token>PROTO_SR</token>), an attribute set reference, and
109 finally a null-terminated string holding the query string.
112 If the parse went well, <function>p_query_rpn()</function> returns a
113 pointer to a <literal>Z_RPNQuery</literal> structure which can be
114 placed directly into a <literal>Z_SearchRequest</literal>.
115 If parsing failed, due to syntax error, a NULL pointer is returned.
118 The <literal>p_query_attset</literal> specifies which attribute set
119 to use if the query doesn't specify one by the
120 <literal>@attrset</literal> operator.
121 The <literal>p_query_attset</literal> returns 0 if the argument is a
122 valid attribute set specifier; otherwise the function returns -1.
126 The grammar of the PQF is as follows:
130 query ::= top-set query-struct.
132 top-set ::= [ '@attrset' string ]
134 query-struct ::= attr-spec | simple | complex | '@term' term-type query
136 attr-spec ::= '@attr' [ string ] string query-struct
138 complex ::= operator query-struct query-struct.
140 operator ::= '@and' | '@or' | '@not' | '@prox' proximity.
142 simple ::= result-set | term.
144 result-set ::= '@set' string.
148 proximity ::= exclusion distance ordered relation which-code unit-code.
150 exclusion ::= '1' | '0' | 'void'.
152 distance ::= integer.
154 ordered ::= '1' | '0'.
156 relation ::= integer.
158 which-code ::= 'known' | 'private' | integer.
160 unit-code ::= integer.
162 term-type ::= 'general' | 'numeric' | 'string' | 'oid' | 'datetime' | 'null'.
166 You will note that the syntax above is a fairly faithful
167 representation of RPN, except for the Attribute, which has been
168 moved a step away from the term, allowing you to associate one or more
169 attributes with an entire query structure. The parser will
170 automatically apply the given attributes to each term as required.
174 The @attr operator is followed by an attribute specification
175 (<literal>attr-spec</literal> above). The specification consists
176 of an optional attribute set, an attribute type-value pair and
177 a sub-query. The attribute type-value pair is packed in one string:
178 an attribute type, an equals sign, and an attribute value, like this:
179 <literal>@attr 1=1003</literal>.
180 The type is always an integer but the value may be either an
181 integer or a string (if it doesn't start with a digit character).
182 A string attribute-value is encoded as a Type-1 ``complex''
183 attribute with the list of values containing the single string
184 specified, and including no semantic indicators.
188 Version 3 of the Z39.50 specification defines various encoding of terms.
189 Use <literal>@term </literal> <replaceable>type</replaceable>
190 <replaceable>string</replaceable>,
191 where type is one of: <literal>general</literal>,
192 <literal>numeric</literal> or <literal>string</literal>
193 (for InternationalString).
194 If no term type has been given, the <literal>general</literal> form
195 is used. This is the only encoding allowed in both versions 2 and 3
196 of the Z39.50 standard.
199 <sect3 id="PQF-prox">
200 <title>Using Proximity Operators with PQF</title>
203 This is an advanced topic, describing how to construct
204 queries that make very specific requirements on the
205 relative location of their operands.
206 You may wish to skip this section and go straight to
207 <link linkend="pqf-examples">the example PQF queries</link>.
212 Most Z39.50 servers do not support proximity searching, or
213 support only a small subset of the full functionality that
214 can be expressed using the PQF proximity operator. Be
215 aware that the ability to <emphasis>express</emphasis> a
216 query in PQF is no guarantee that any given server will
217 be able to <emphasis>execute</emphasis> it.
223 The proximity operator <literal>@prox</literal> is a special
224 and more restrictive version of the conjunction operator
225 <literal>@and</literal>. Its semantics are described in
226 section 3.7.2 (Proximity) of Z39.50 the standard itself, which
227 can be read on-line at
228 <ulink url="http://lcweb.loc.gov/z3950/agency/markup/09.html"/>
231 In PQF, the proximity operation is represented by a sequence
234 @prox <replaceable>exclusion</replaceable> <replaceable>distance</replaceable> <replaceable>ordered</replaceable> <replaceable>relation</replaceable> <replaceable>which-code</replaceable> <replaceable>unit-code</replaceable>
236 in which the meanings of the parameters are as described in in
237 the standard, and they can take the following values:
239 <listitem><formalpara><title>exclusion</title><para>
240 0 = false (i.e. the proximity condition specified by the
241 remaining parameters must be satisfied) or
242 1 = true (the proximity condition specified by the
243 remaining parameters must <emphasis>not</emphasis> be
245 </para></formalpara></listitem>
246 <listitem><formalpara><title>distance</title><para>
247 An integer specifying the difference between the locations
248 of the operands: e.g. two adjacent words would have
249 distance=1 since their locations differ by one unit.
250 </para></formalpara></listitem>
251 <listitem><formalpara><title>ordered</title><para>
252 1 = ordered (the operands must occur in the order the
253 query specifies them) or
254 0 = unordered (they may appear in either order).
255 </para></formalpara></listitem>
256 <listitem><formalpara><title>relation</title><para>
257 Recognised values are
261 4 (greaterThanOrEqual),
264 </para></formalpara></listitem>
265 <listitem><formalpara><title>which-code</title><para>
266 <literal>known</literal>
269 (the unit-code parameter is taken from the well-known list
270 of alternatives described in below) or
271 <literal>private</literal>
274 (the unit-code paramater has semantics specific to an
275 out-of-band agreement such as a profile).
276 </para></formalpara></listitem>
277 <listitem><formalpara><title>unit-code</title><para>
278 If the which-code parameter is <literal>known</literal>
279 then the recognised values are
291 If which-code is <literal>private</literal> then the
292 acceptable values are determined by the profile.
293 </para></formalpara></listitem>
295 (The numeric values of the relation and well-known unit-code
296 parameters are taken straight from
297 <ulink url="http://lcweb.loc.gov/z3950/agency/asn1.html#ProximityOperator"
298 >the ASN.1</ulink> of the proximity structure in the standard.)
302 <sect3 id="pqf-examples"><title>PQF queries</title>
304 <example><title>PQF queries using simple terms</title>
313 <example><title>PQF boolean operators</title>
316 @or "dylan" "zimmerman"
318 @and @or dylan zimmerman when
320 @and when @or dylan zimmerman
324 <example><title>PQF references to result sets</title>
329 @and @set seta @set setb
333 <example><title>Attributes for terms</title>
338 @attr 1=4 @attr 4=1 "self portrait"
340 @attrset exp1 @attr 1=1 CategoryList
342 @attr gils 1=2008 Copenhagen
344 @attr 1=/book/title computer
348 <example><title>PQF Proximity queries</title>
351 @prox 0 3 1 2 k 2 dylan zimmerman
354 Here the parameters 0, 3, 1, 2, k and 2 represent exclusion,
355 distance, ordered, relation, which-code and unit-code, in that
359 exclusion = 0: the proximity condition must hold
362 distance = 3: the terms must be three units apart
365 ordered = 1: they must occur in the order they are specified
368 relation = 2: lessThanOrEqual (to the distance of 3 units)
371 which-code is ``known'', so the standard unit-codes are used
377 So the whole proximity query means that the words
378 <literal>dylan</literal> and <literal>zimmerman</literal> must
379 both occur in the record, in that order, differing in position
380 by three or fewer words (i.e. with two or fewer words between
381 them.) The query would find ``Bob Dylan, aka. Robert
382 Zimmerman'', but not ``Bob Dylan, born as Robert Zimmerman''
383 since the distance in this case is four.
387 <example><title>PQF specification of search term</title>
390 @term string "a UTF-8 string, maybe?"
394 <example><title>PQF mixed queries</title>
397 @or @and bob dylan @set Result-1
399 @attr 4=1 @and @attr 1=1 "bob dylan" @attr 1=4 "slow train coming"
401 @and @attr 2=4 @attr gils 1=2038 -114 @attr 2=2 @attr gils 1=2039 -109
405 The last of these examples is a spatial search: in
406 <ulink url="http://www.gils.net/prof_v2.html#sec_7_4"
407 >the GILS attribute set</ulink>,
409 2038 indicates West Bounding Coordinate and
410 2030 indicates East Bounding Coordinate,
411 so the query is for areas extending from -114 degrees
412 to no more than -109 degrees.
419 <sect2 id="CCL"><title>CCL</title>
422 Not all users enjoy typing in prefix query structures and numerical
423 attribute values, even in a minimalistic test client. In the library
424 world, the more intuitive Common Command Language - CCL (ISO 8777)
425 has enjoyed some popularity - especially before the widespread
426 availability of graphical interfaces. It is still useful in
427 applications where you for some reason or other need to provide a
428 symbolic language for expressing boolean query structures.
432 The EUROPAGATE research project working under the Libraries programme
433 of the European Commission's DG XIII has, amongst other useful tools,
434 implemented a general-purpose CCL parser which produces an output
435 structure that can be trivially converted to the internal RPN
436 representation of &yaz; (The <literal>Z_RPNQuery</literal> structure).
437 Since the CCL utility - along with the rest of the software
438 produced by EUROPAGATE - is made freely available on a liberal
439 license, it is included as a supplement to &yaz;.
442 <sect3><title>CCL Syntax</title>
445 The CCL parser obeys the following grammar for the FIND argument.
446 The syntax is annotated by in the lines prefixed by
447 <literal>‐‐</literal>.
451 CCL-Find ::= CCL-Find Op Elements
454 Op ::= "and" | "or" | "not"
455 -- The above means that Elements are separated by boolean operators.
457 Elements ::= '(' CCL-Find ')'
460 | Qualifiers Relation Terms
461 | Qualifiers Relation '(' CCL-Find ')'
462 | Qualifiers '=' string '-' string
463 -- Elements is either a recursive definition, a result set reference, a
464 -- list of terms, qualifiers followed by terms, qualifiers followed
465 -- by a recursive definition or qualifiers in a range (lower - upper).
467 Set ::= 'set' = string
468 -- Reference to a result set
470 Terms ::= Terms Prox Term
472 -- Proximity of terms.
476 -- This basically means that a term may include a blank
478 Qualifiers ::= Qualifiers ',' string
480 -- Qualifiers is a list of strings separated by comma
482 Relation ::= '=' | '>=' | '<=' | '<>' | '>' | '<'
483 -- Relational operators. This really doesn't follow the ISO8777
487 -- Proximity operator
491 <example><title>CCL queries</title>
493 The following queries are all valid:
505 (dylan and bob) or set=1
509 Assuming that the qualifiers <literal>ti</literal>,
510 <literal>au</literal>
511 and <literal>date</literal> are defined we may use:
517 au=(bob dylan and slow train coming)
519 date>1980 and (ti=((self portrait)))
525 <sect3><title>CCL Qualifiers</title>
528 Qualifiers are used to direct the search to a particular searchable
529 index, such as title (ti) and author indexes (au). The CCL standard
530 itself doesn't specify a particular set of qualifiers, but it does
531 suggest a few short-hand notations. You can customize the CCL parser
532 to support a particular set of qualifiers to reflect the current target
533 profile. Traditionally, a qualifier would map to a particular
534 use-attribute within the BIB-1 attribute set. It is also
535 possible to set other attributes, such as the structure
540 A CCL profile is a set of predefined CCL qualifiers that may be
541 read from a file or set in the CCL API.
542 The YAZ client reads its CCL qualifiers from a file named
543 <filename>default.bib</filename>. There are four types of
544 lines in a CCL profile: qualifier specification,
545 qualifier alias, comments and directives.
547 <sect4><title id="qualifier-specification">Qualifier specification</title>
549 A qualifier specification is of the form:
553 <replaceable>qualifier-name</replaceable>
554 [<replaceable>attributeset</replaceable><literal>,</literal>]<replaceable>type</replaceable><literal>=</literal><replaceable>val</replaceable>
555 [<replaceable>attributeset</replaceable><literal>,</literal>]<replaceable>type</replaceable><literal>=</literal><replaceable>val</replaceable> ...
559 where <replaceable>qualifier-name</replaceable> is the name of the
560 qualifier to be used (eg. <literal>ti</literal>),
561 <replaceable>type</replaceable> is attribute type in the attribute
562 set (Bib-1 is used if no attribute set is given) and
563 <replaceable>val</replaceable> is attribute value.
564 The <replaceable>type</replaceable> can be specified as an
565 integer or as it be specified either as a single-letter:
566 <literal>u</literal> for use,
567 <literal>r</literal> for relation,<literal>p</literal> for position,
568 <literal>s</literal> for structure,<literal>t</literal> for truncation
569 or <literal>c</literal> for completeness.
570 The attributes for the special qualifier name <literal>term</literal>
571 are used when no CCL qualifier is given in a query.
572 <table><title>Common Bib-1 attributes</title>
574 <colspec colwidth="2*" colname="type"></colspec>
575 <colspec colwidth="9*" colname="description"></colspec>
579 <entry>Description</entry>
584 <entry><literal>u=</literal><replaceable>value</replaceable></entry>
586 Use attribute (1). Common use attributes are
587 1 Personal-name, 4 Title, 7 ISBN, 8 ISSN, 30 Date,
588 62 Subject, 1003 Author), 1016 Any. Specify value
594 <entry><literal>r=</literal><replaceable>value</replaceable></entry>
596 Relation attribute (2). Common values are
597 1 <, 2 <=, 3 =, 4 >=, 5 >, 6 <>,
598 100 phonetic, 101 stem, 102 relevance, 103 always matches.
603 <entry><literal>p=</literal><replaceable>value</replaceable></entry>
605 Position attribute (3). Values: 1 first in field, 2
606 first in any subfield, 3 any position in field.
611 <entry><literal>s=</literal><replaceable>value</replaceable></entry>
613 Structure attribute (4). Values: 1 phrase, 2 word,
614 3 key, 4 year, 5 date, 6 word list, 100 date (un),
615 101 name (norm), 102 name (un), 103 structure, 104 urx,
616 105 free-form-text, 106 document-text, 107 local-number,
617 108 string, 109 numeric string.
622 <entry><literal>t=</literal><replaceable>value</replaceable></entry>
624 Truncation attribute (5). Values: 1 right, 2 left,
625 3 left& right, 100 none, 101 process #, 102 regular-1,
626 103 regular-2, 104 CCL.
631 <entry><literal>c=</literal><replaceable>value</replaceable></entry>
633 Completeness attribute (6). Values: 1 incomplete subfield,
634 2 complete subfield, 3 complete field.
643 The complete list of Bib-1 attributes can be found
644 <ulink url="http://lcweb.loc.gov/z3950/agency/defns/bib1.html">
649 It is also possible to specify non-numeric attribute values,
650 which are used in combination with certain types.
651 The special combinations are:
653 <table><title>Special attribute combos</title>
655 <colspec colwidth="2*" colname="name"></colspec>
656 <colspec colwidth="9*" colname="description"></colspec>
660 <entry>Description</entry>
665 <entry><literal>s=pw</literal></entry><entry>
666 The structure is set to either word or phrase depending
667 on the number of tokens in a term (phrase-word).
671 <entry><literal>s=al</literal></entry><entry>
672 Each token in the term is ANDed. (and-list).
673 This does not set the structure at all.
677 <row><entry><literal>s=ol</literal></entry><entry>
678 Each token in the term is ORed. (or-list).
679 This does not set the structure at all.
683 <row><entry><literal>r=o</literal></entry><entry>
684 Allows ranges and the operators greather-than, less-than, ...
686 This sets Bib-1 relation attribute accordingly (relation
687 ordered). A query construct is only treated as a range if
688 dash is used and that is surrounded by white-space. So
689 <literal>-1980</literal> is treated as term
690 <literal>"-1980"</literal> not <literal><= 1980</literal>.
691 If <literal>- 1980</literal> is used, however, that is
696 <row><entry><literal>r=r</literal></entry><entry>
697 Similar to <literal>r=o</literal> but assumes that terms
698 are non-negative (not prefixed with <literal>-</literal>).
699 Thus, a dash will always be treated as a range.
700 The construct <literal>1980-1990</literal> is
701 treated as a range with <literal>r=r</literal> but as a
702 single term <literal>"1980-1990"</literal> with
703 <literal>r=o</literal>. The special attribute
704 <literal>r=r</literal> is available in YAZ 2.0.24 or later.
708 <row><entry><literal>t=l</literal></entry><entry>
709 Allows term to be left-truncated.
710 If term is of the form <literal>?x</literal>, the resulting
711 Type-1 term is <literal>x</literal> and truncation is left.
715 <row><entry><literal>t=r</literal></entry><entry>
716 Allows term to be right-truncated.
717 If term is of the form <literal>x?</literal>, the resulting
718 Type-1 term is <literal>x</literal> and truncation is right.
722 <row><entry><literal>t=n</literal></entry><entry>
723 If term is does not include <literal>?</literal>, the
724 truncation attribute is set to none (100).
728 <row><entry><literal>t=b</literal></entry><entry>
729 Allows term to be both left&right truncated.
730 If term is of the form <literal>?x?</literal>, the
731 resulting term is <literal>x</literal> and trunctation is
732 set to both left&right.
739 <example><title>CCL profile</title>
741 Consider the following definition:
752 <literal>ti</literal> and <literal>au</literal> both set
753 structure attribute to phrase (s=1).
754 <literal>ti</literal>
755 sets the use-attribute to 4. <literal>au</literal> sets the
757 When no qualifiers are used in the query the structure-attribute is
758 set to free-form-text (105) (rule for <literal>term</literal>).
759 The <literal>date</literal> sets the relation attribute to
760 the relation used in the CCL query and sets the use attribute
764 You can combine attributes. To Search for "ranked title" you
767 ti,ranked=knuth computer
769 which will set relation=ranked, use=title, structure=phrase.
776 is a valid query. But
784 <sect4><title>Qualifier alias</title>
786 A qualifier alias is of the form:
789 <replaceable>q</replaceable>
790 <replaceable>q1</replaceable> <replaceable>q2</replaceable> ..
793 which declares <replaceable>q</replaceable> to
794 be an alias for <replaceable>q1</replaceable>,
795 <replaceable>q2</replaceable>... such that the CCL
796 query <replaceable>q=x</replaceable> is equivalent to
797 <replaceable>q1=x or q2=x or ...</replaceable>.
801 <sect4><title>Comments</title>
803 Lines with white space or lines that begin with
804 character <literal>#</literal> are treated as comments.
808 <sect4><title>Directives</title>
810 Directive specifications takes the form
812 <para><literal>@</literal><replaceable>directive</replaceable> <replaceable>value</replaceable>
814 <table><title>CCL directives</title>
816 <colspec colwidth="2*" colname="name"></colspec>
817 <colspec colwidth="8*" colname="description"></colspec>
818 <colspec colwidth="1*" colname="default"></colspec>
822 <entry>Description</entry>
823 <entry>Default</entry>
828 <entry>truncation</entry>
829 <entry>Truncation character</entry>
830 <entry><literal>?</literal></entry>
834 <entry>Specifies how multiple fields are to be
835 combined. There are two modes: <literal>or</literal>:
836 multiple qualifier fields are ORed,
837 <literal>merge</literal>: attributes for the qualifier
838 fields are merged and assigned to one term.
840 <entry><literal>merge</literal></entry>
844 <entry>Specificies if CCL operatores and qualifiers should be
845 compared with case sensitivity or not. Specify 0 for
846 case sensitive; 1 for case insensitive.</entry>
847 <entry><literal>0</literal></entry>
852 <entry>Specifies token for CCL operator AND.</entry>
853 <entry><literal>and</literal></entry>
858 <entry>Specifies token for CCL operator OR.</entry>
859 <entry><literal>or</literal></entry>
864 <entry>Specifies token for CCL operator NOT.</entry>
865 <entry><literal>not</literal></entry>
870 <entry>Specifies token for CCL operator SET.</entry>
871 <entry><literal>set</literal></entry>
878 <sect3><title>CCL API</title>
880 All public definitions can be found in the header file
881 <filename>ccl.h</filename>. A profile identifier is of type
882 <literal>CCL_bibset</literal>. A profile must be created with the call
883 to the function <function>ccl_qual_mk</function> which returns a profile
884 handle of type <literal>CCL_bibset</literal>.
888 To read a file containing qualifier definitions the function
889 <function>ccl_qual_file</function> may be convenient. This function
890 takes an already opened <literal>FILE</literal> handle pointer as
891 argument along with a <literal>CCL_bibset</literal> handle.
895 To parse a simple string with a FIND query use the function
898 struct ccl_rpn_node *ccl_find_str (CCL_bibset bibset, const char *str,
899 int *error, int *pos);
902 which takes the CCL profile (<literal>bibset</literal>) and query
903 (<literal>str</literal>) as input. Upon successful completion the RPN
904 tree is returned. If an error occur, such as a syntax error, the integer
905 pointed to by <literal>error</literal> holds the error code and
906 <literal>pos</literal> holds the offset inside query string in which
911 An English representation of the error may be obtained by calling
912 the <literal>ccl_err_msg</literal> function. The error codes are
913 listed in <filename>ccl.h</filename>.
917 To convert the CCL RPN tree (type
918 <literal>struct ccl_rpn_node *</literal>)
919 to the Z_RPNQuery of YAZ the function <function>ccl_rpn_query</function>
920 must be used. This function which is part of YAZ is implemented in
921 <filename>yaz-ccl.c</filename>.
922 After calling this function the CCL RPN tree is probably no longer
923 needed. The <literal>ccl_rpn_delete</literal> destroys the CCL RPN tree.
927 A CCL profile may be destroyed by calling the
928 <function>ccl_qual_rm</function> function.
932 The token names for the CCL operators may be changed by setting the
933 globals (all type <literal>char *</literal>)
934 <literal>ccl_token_and</literal>, <literal>ccl_token_or</literal>,
935 <literal>ccl_token_not</literal> and <literal>ccl_token_set</literal>.
936 An operator may have aliases, i.e. there may be more than one name for
937 the operator. To do this, separate each alias with a space character.
941 <sect2 id="tools.cql"><title>CQL</title>
943 <ulink url="http://www.loc.gov/z3950/agency/zing/cql/">CQL</ulink>
944 - Common Query Language - was defined for the
945 <ulink url="http://www.loc.gov/z3950/agency/zing/srw/">SRW</ulink>
947 In many ways CQL has a similar syntax to CCL.
948 The objective of CQL is different. Where CCL aims to be
949 an end-user language, CQL is <emphasis>the</emphasis> protocol
950 query language for SRW.
954 If you are new to CQL, read the
955 <ulink url="http://zing.z3950.org/cql/intro.html">Gentle
956 Introduction</ulink>.
960 The CQL parser in &yaz; provides the following:
964 It parses and validates a CQL query.
969 It generates a C structure that allows you to convert
970 a CQL query to some other query language, such as SQL.
975 The parser converts a valid CQL query to PQF, thus providing a
976 way to use CQL for both SRW/SRU servers and Z39.50 targets at the
982 The parser converts CQL to
983 <ulink url="http://www.loc.gov/z3950/agency/zing/cql/xcql.html">
985 XCQL is an XML representation of CQL.
986 XCQL is part of the SRW specification. However, since SRU
987 supports CQL only, we don't expect XCQL to be widely used.
988 Furthermore, CQL has the advantage over XCQL that it is
994 <sect3 id="tools.cql.parsing"><title>CQL parsing</title>
996 A CQL parser is represented by the <literal>CQL_parser</literal>
997 handle. Its contents should be considered &yaz; internal (private).
999 #include <yaz/cql.h>
1001 typedef struct cql_parser *CQL_parser;
1003 CQL_parser cql_parser_create(void);
1004 void cql_parser_destroy(CQL_parser cp);
1006 A parser is created by <function>cql_parser_create</function> and
1007 is destroyed by <function>cql_parser_destroy</function>.
1010 To parse a CQL query string, the following function
1013 int cql_parser_string(CQL_parser cp, const char *str);
1015 A CQL query is parsed by the <function>cql_parser_string</function>
1016 which takes a query <parameter>str</parameter>.
1017 If the query was valid (no syntax errors), then zero is returned;
1018 otherwise -1 is returned to indicate a syntax error.
1022 int cql_parser_stream(CQL_parser cp,
1023 int (*getbyte)(void *client_data),
1024 void (*ungetbyte)(int b, void *client_data),
1027 int cql_parser_stdio(CQL_parser cp, FILE *f);
1029 The functions <function>cql_parser_stream</function> and
1030 <function>cql_parser_stdio</function> parses a CQL query
1031 - just like <function>cql_parser_string</function>.
1032 The only difference is that the CQL query can be
1033 fed to the parser in different ways.
1034 The <function>cql_parser_stream</function> uses a generic
1035 byte stream as input. The <function>cql_parser_stdio</function>
1036 uses a <literal>FILE</literal> handle which is opened for reading.
1040 <sect3 id="tools.cql.tree"><title>CQL tree</title>
1042 The the query string is valid, the CQL parser
1043 generates a tree representing the structure of the
1048 struct cql_node *cql_parser_result(CQL_parser cp);
1050 <function>cql_parser_result</function> returns the
1051 a pointer to the root node of the resulting tree.
1054 Each node in a CQL tree is represented by a
1055 <literal>struct cql_node</literal>.
1056 It is defined as follows:
1058 #define CQL_NODE_ST 1
1059 #define CQL_NODE_BOOL 2
1069 struct cql_node *modifiers;
1073 struct cql_node *left;
1074 struct cql_node *right;
1075 struct cql_node *modifiers;
1080 There are two node types: search term (ST) and boolean (BOOL).
1081 A modifier is treated as a search term too.
1084 The search term node has five members:
1088 <literal>index</literal>: index for search term.
1089 If an index is unspecified for a search term,
1090 <literal>index</literal> will be NULL.
1095 <literal>index_uri</literal>: index URi for search term
1096 or NULL if none could be resolved for the index.
1101 <literal>term</literal>: the search term itself.
1106 <literal>relation</literal>: relation for search term.
1111 <literal>relation_uri</literal>: relation URI for search term.
1116 <literal>modifiers</literal>: relation modifiers for search
1117 term. The <literal>modifiers</literal> list itself of cql_nodes
1118 each of type <literal>ST</literal>.
1125 The boolean node represents both <literal>and</literal>,
1126 <literal>or</literal>, not as well as
1131 <literal>left</literal> and <literal>right</literal>: left
1132 - and right operand respectively.
1137 <literal>modifiers</literal>: proximity arguments.
1144 <sect3 id="tools.cql.pqf"><title>CQL to PQF conversion</title>
1146 Conversion to PQF (and Z39.50 RPN) is tricky by the fact
1147 that the resulting RPN depends on the Z39.50 target
1148 capabilities (combinations of supported attributes).
1149 In addition, the CQL and SRW operates on index prefixes
1150 (URI or strings), whereas the RPN uses Object Identifiers
1154 The CQL library of &yaz; defines a <literal>cql_transform_t</literal>
1155 type. It represents a particular mapping between CQL and RPN.
1156 This handle is created and destroyed by the functions:
1158 cql_transform_t cql_transform_open_FILE (FILE *f);
1159 cql_transform_t cql_transform_open_fname(const char *fname);
1160 void cql_transform_close(cql_transform_t ct);
1162 The first two functions create a tranformation handle from
1163 either an already open FILE or from a filename respectively.
1166 The handle is destroyed by <function>cql_transform_close</function>
1167 in which case no further reference of the handle is allowed.
1170 When a <literal>cql_transform_t</literal> handle has been created
1171 you can convert to RPN.
1173 int cql_transform_buf(cql_transform_t ct,
1174 struct cql_node *cn, char *out, int max);
1176 This function converts the CQL tree <literal>cn</literal>
1177 using handle <literal>ct</literal>.
1178 For the resulting PQF, you supply a buffer <literal>out</literal>
1179 which must be able to hold at at least <literal>max</literal>
1183 If conversion failed, <function>cql_transform_buf</function>
1184 returns a non-zero SRW error code; otherwise zero is returned
1185 (conversion successful). The meanings of the numeric error
1186 codes are listed in the SRW specifications at
1187 <ulink url="http://www.loc.gov/srw/diagnostic-list.html"/>
1190 If conversion fails, more information can be obtained by calling
1192 int cql_transform_error(cql_transform_t ct, char **addinfop);
1194 This function returns the most recently returned numeric
1195 error-code and sets the string-pointer at
1196 <literal>*addinfop</literal> to point to a string containing
1197 additional information about the error that occurred: for
1198 example, if the error code is 15 (``Illegal or unsupported context
1199 set''), the additional information is the name of the requested
1200 context set that was not recognised.
1203 The SRW error-codes may be translated into brief human-readable
1204 error messages using
1206 const char *cql_strerror(int code);
1210 If you wish to be able to produce a PQF result in a different
1211 way, there are two alternatives.
1213 void cql_transform_pr(cql_transform_t ct,
1214 struct cql_node *cn,
1215 void (*pr)(const char *buf, void *client_data),
1218 int cql_transform_FILE(cql_transform_t ct,
1219 struct cql_node *cn, FILE *f);
1221 The former function produces output to a user-defined
1222 output stream. The latter writes the result to an already
1223 open <literal>FILE</literal>.
1226 <sect3 id="tools.cql.map">
1227 <title>Specification of CQL to RPN mappings</title>
1229 The file supplied to functions
1230 <function>cql_transform_open_FILE</function>,
1231 <function>cql_transform_open_fname</function> follows
1232 a structure found in many Unix utilities.
1233 It consists of mapping specifications - one per line.
1234 Lines starting with <literal>#</literal> are ignored (comments).
1237 Each line is of the form
1239 <replaceable>CQL pattern</replaceable><literal> = </literal> <replaceable> RPN equivalent</replaceable>
1243 An RPN pattern is a simple attribute list. Each attribute pair
1246 [<replaceable>set</replaceable>] <replaceable>type</replaceable><literal>=</literal><replaceable>value</replaceable>
1248 The attribute <replaceable>set</replaceable> is optional.
1249 The <replaceable>type</replaceable> is the attribute type,
1250 <replaceable>value</replaceable> the attribute value.
1253 The following CQL patterns are recognized:
1255 <varlistentry><term>
1256 <literal>index.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
1260 This pattern is invoked when a CQL index, such as
1261 dc.title is converted. <replaceable>set</replaceable>
1262 and <replaceable>name</replaceable> are the context set and index
1264 Typically, the RPN specifies an equivalent use attribute.
1267 For terms not bound by an index the pattern
1268 <literal>index.cql.serverChoice</literal> is used.
1269 Here, the prefix <literal>cql</literal> is defined as
1270 <literal>http://www.loc.gov/zing/cql/cql-indexes/v1.0/</literal>.
1271 If this pattern is not defined, the mapping will fail.
1275 <varlistentry><term>
1276 <literal>qualifier.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
1281 For backwards compatibility, this is recognised as a synonym of
1282 <literal>index.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
1286 <varlistentry><term>
1287 <literal>relation.</literal><replaceable>relation</replaceable>
1291 This pattern specifies how a CQL relation is mapped to RPN.
1292 <replaceable>pattern</replaceable> is name of relation
1293 operator. Since <literal>=</literal> is used as
1294 separator between CQL pattern and RPN, CQL relations
1295 including <literal>=</literal> cannot be
1296 used directly. To avoid a conflict, the names
1297 <literal>ge</literal>,
1298 <literal>eq</literal>,
1299 <literal>le</literal>,
1300 must be used for CQL operators, greater-than-or-equal,
1301 equal, less-than-or-equal respectively.
1302 The RPN pattern is supposed to include a relation attribute.
1305 For terms not bound by a relation, the pattern
1306 <literal>relation.scr</literal> is used. If the pattern
1307 is not defined, the mapping will fail.
1310 The special pattern, <literal>relation.*</literal> is used
1311 when no other relation pattern is matched.
1316 <varlistentry><term>
1317 <literal>relationModifier.</literal><replaceable>mod</replaceable>
1321 This pattern specifies how a CQL relation modifier is mapped to RPN.
1322 The RPN pattern is usually a relation attribute.
1327 <varlistentry><term>
1328 <literal>structure.</literal><replaceable>type</replaceable>
1332 This pattern specifies how a CQL structure is mapped to RPN.
1333 Note that this CQL pattern is somewhat to similar to
1334 CQL pattern <literal>relation</literal>.
1335 The <replaceable>type</replaceable> is a CQL relation.
1338 The pattern, <literal>structure.*</literal> is used
1339 when no other structure pattern is matched.
1340 Usually, the RPN equivalent specifies a structure attribute.
1345 <varlistentry><term>
1346 <literal>position.</literal><replaceable>type</replaceable>
1350 This pattern specifies how the anchor (position) of
1351 CQL is mapped to RPN.
1352 The <replaceable>type</replaceable> is one
1353 of <literal>first</literal>, <literal>any</literal>,
1354 <literal>last</literal>, <literal>firstAndLast</literal>.
1357 The pattern, <literal>position.*</literal> is used
1358 when no other position pattern is matched.
1363 <varlistentry><term>
1364 <literal>set.</literal><replaceable>prefix</replaceable>
1368 This specification defines a CQL context set for a given prefix.
1369 The value on the right hand side is the URI for the set -
1370 <emphasis>not</emphasis> RPN. All prefixes used in
1371 index patterns must be defined this way.
1377 <example><title>CQL to RPN mapping file</title>
1379 This simple file defines two context sets, three indexes and three
1380 relations, a position pattern and a default structure.
1382 <programlisting><![CDATA[
1383 set.cql = http://www.loc.gov/zing/cql/context-sets/cql/v1.1/
1384 set.dc = http://www.loc.gov/zing/cql/dc-indexes/v1.0/
1386 index.cql.serverChoice = 1=1016
1387 index.dc.title = 1=4
1388 index.dc.subject = 1=21
1394 position.any = 3=3 6=1
1400 With the mappings above, the CQL query
1404 is converted to the PQF:
1406 @attr 1=1016 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "computer"
1408 by rules <literal>index.cql.serverChoice</literal>,
1409 <literal>relation.scr</literal>, <literal>structure.*</literal>,
1410 <literal>position.any</literal>.
1417 is rejected, since <literal>position.right</literal> is
1423 >my = "http://www.loc.gov/zing/cql/dc-indexes/v1.0/" my.title = x
1427 @attr 1=4 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "x"
1432 <sect3 id="tools.cql.xcql"><title>CQL to XCQL conversion</title>
1434 Conversion from CQL to XCQL is trivial and does not
1435 require a mapping to be defined.
1436 There three functions to choose from depending on the
1437 way you wish to store the resulting output (XML buffer
1440 int cql_to_xml_buf(struct cql_node *cn, char *out, int max);
1441 void cql_to_xml(struct cql_node *cn,
1442 void (*pr)(const char *buf, void *client_data),
1444 void cql_to_xml_stdio(struct cql_node *cn, FILE *f);
1446 Function <function>cql_to_xml_buf</function> converts
1447 to XCQL and stores result in a user supplied buffer of a given
1451 <function>cql_to_xml</function> writes the result in
1452 a user defined output stream.
1453 <function>cql_to_xml_stdio</function> writes to a
1459 <sect1 id="tools.oid"><title>Object Identifiers</title>
1462 The basic YAZ representation of an OID is an array of integers,
1463 terminated with the value -1. The &odr; module provides two
1464 utility-functions to create and copy this type of data elements:
1468 Odr_oid *odr_getoidbystr(ODR o, char *str);
1472 Creates an OID based on a string-based representation using dots (.)
1473 to separate elements in the OID.
1477 Odr_oid *odr_oiddup(ODR odr, Odr_oid *o);
1481 Creates a copy of the OID referenced by the <emphasis>o</emphasis>
1483 Both functions take an &odr; stream as parameter. This stream is used to
1484 allocate memory for the data elements, which is released on a
1485 subsequent call to <function>odr_reset()</function> on that stream.
1489 The OID module provides a higher-level representation of the
1490 family of object identifiers which describe the Z39.50 protocol and its
1491 related objects. The definition of the module interface is given in
1492 the <filename>oid.h</filename> file.
1496 The interface is mainly based on the <literal>oident</literal> structure.
1497 The definition of this structure looks like this:
1501 typedef struct oident
1506 int oidsuffix[OID_SIZE];
1512 The proto field takes one of the values
1521 Use <literal>PROTO_Z3950</literal> for Z39.50 Object Identifers,
1522 <literal>PROTO_GENERAL</literal> for other types (such as
1523 those associated with ILL).
1527 The oclass field takes one of the values
1549 corresponding to the OID classes defined by the Z39.50 standard.
1551 Finally, the value field takes one of the values
1609 again, corresponding to the specific OIDs defined by the standard.
1611 <ulink url="http://lcweb.loc.gov/z3950/agency/defns/oids.html">
1612 Registry of Z39.50 Object Identifiers</ulink> for the
1617 The desc field contains a brief, mnemonic name for the OID in question.
1625 struct oident *oid_getentbyoid(int *o);
1629 takes as argument an OID, and returns a pointer to a static area
1630 containing an <literal>oident</literal> structure. You typically use
1631 this function when you receive a PDU containing an OID, and you wish
1632 to branch out depending on the specific OID value.
1640 int *oid_ent_to_oid(struct oident *ent, int *dst);
1644 Takes as argument an <literal>oident</literal> structure - in which
1645 the <literal>proto</literal>, <literal>oclass</literal>/, and
1646 <literal>value</literal> fields are assumed to be set correctly -
1647 and returns a pointer to a the buffer as given by <literal>dst</literal>
1649 representation of the corresponding OID. The function returns
1650 NULL and the array dst is unchanged if a mapping couldn't place.
1651 The array <literal>dst</literal> should be at least of size
1652 <literal>OID_SIZE</literal>.
1656 The <function>oid_ent_to_oid()</function> function can be used whenever
1657 you need to prepare a PDU containing one or more OIDs. The separation of
1658 the <literal>protocol</literal> element from the remainder of the
1659 OID-description makes it simple to write applications that can
1660 communicate with either Z39.50 or OSI SR-based applications.
1668 oid_value oid_getvalbyname(const char *name);
1672 takes as argument a mnemonic OID name, and returns the
1673 <literal>/value</literal> field of the first entry in the database that
1674 contains the given name in its <literal>desc</literal> field.
1678 Three utility functions are provided for translating OIDs'
1679 symbolic names (e.g. <literal>Usmarc</literal> into OID structures
1680 (int arrays) and strings containing the OID in dotted notation
1681 (e.g. <literal>1.2.840.10003.9.5.1</literal>). They are:
1685 int *oid_name_to_oid(oid_class oclass, const char *name, int *oid);
1686 char *oid_to_dotstring(const int *oid, char *oidbuf);
1687 char *oid_name_to_dotstring(oid_class oclass, const char *name, char *oidbuf);
1691 <literal>oid_name_to_oid()</literal>
1692 translates the specified symbolic <literal>name</literal>,
1693 interpreted as being of class <literal>oclass</literal>. (The
1694 class must be specified as many symbolic names exist within
1695 multiple classes - for example, <literal>Zthes</literal> is the
1696 symbolic name of an attribute set, a schema and a tag-set.) The
1697 sequence of integers representing the OID is written into the
1698 area <literal>oid</literal> provided by the caller; it is the
1699 caller's responsibility to ensure that this area is large enough
1700 to contain the translated OID. As a convenience, the address of
1701 the buffer (i.e. the value of <literal>oid</literal>) is
1705 <literal>oid_to_dotstring()</literal>
1706 Translates the int-array <literal>oid</literal> into a dotted
1707 string which is written into the area <literal>oidbuf</literal>
1708 supplied by the caller; it is the caller's responsibility to
1709 ensure that this area is large enough. The address of the buffer
1713 <literal>oid_name_to_dotstring()</literal>
1714 combines the previous two functions to derive a dotted string
1715 representing the OID specified by <literal>oclass</literal> and
1716 <literal>name</literal>, writing it into the buffer passed as
1717 <literal>oidbuf</literal> and returning its address.
1721 Finally, the module provides the following utility functions, whose
1722 meaning should be obvious:
1726 void oid_oidcpy(int *t, int *s);
1727 void oid_oidcat(int *t, int *s);
1728 int oid_oidcmp(int *o1, int *o2);
1729 int oid_oidlen(int *o);
1734 The OID module has been criticized - and perhaps rightly so
1735 - for needlessly abstracting the
1736 representation of OIDs. Other toolkits use a simple
1737 string-representation of OIDs with good results. In practice, we have
1738 found the interface comfortable and quick to work with, and it is a
1739 simple matter (for what it's worth) to create applications compatible
1740 with both ISO SR and Z39.50. Finally, the use of the
1741 <literal>/oident</literal> database is by no means mandatory.
1742 You can easily create your own system for representing OIDs, as long
1743 as it is compatible with the low-level integer-array representation
1750 <sect1 id="tools.nmem"><title>Nibble Memory</title>
1753 Sometimes when you need to allocate and construct a large,
1754 interconnected complex of structures, it can be a bit of a pain to
1755 release the associated memory again. For the structures describing the
1756 Z39.50 PDUs and related structures, it is convenient to use the
1757 memory-management system of the &odr; subsystem (see
1758 <xref linkend="odr.use"/>). However, in some circumstances
1759 where you might otherwise benefit from using a simple nibble memory
1760 management system, it may be impractical to use
1761 <function>odr_malloc()</function> and <function>odr_reset()</function>.
1762 For this purpose, the memory manager which also supports the &odr;
1763 streams is made available in the NMEM module. The external interface
1764 to this module is given in the <filename>nmem.h</filename> file.
1768 The following prototypes are given:
1772 NMEM nmem_create(void);
1773 void nmem_destroy(NMEM n);
1774 void *nmem_malloc(NMEM n, int size);
1775 void nmem_reset(NMEM n);
1776 int nmem_total(NMEM n);
1777 void nmem_init(void);
1778 void nmem_exit(void);
1782 The <function>nmem_create()</function> function returns a pointer to a
1783 memory control handle, which can be released again by
1784 <function>nmem_destroy()</function> when no longer needed.
1785 The function <function>nmem_malloc()</function> allocates a block of
1786 memory of the requested size. A call to <function>nmem_reset()</function>
1787 or <function>nmem_destroy()</function> will release all memory allocated
1788 on the handle since it was created (or since the last call to
1789 <function>nmem_reset()</function>. The function
1790 <function>nmem_total()</function> returns the number of bytes currently
1791 allocated on the handle.
1795 The nibble memory pool is shared amongst threads. POSIX
1796 mutex'es and WIN32 Critical sections are introduced to keep the
1797 module thread safe. Function <function>nmem_init()</function>
1798 initializes the nibble memory library and it is called automatically
1799 the first time the <literal>YAZ.DLL</literal> is loaded. &yaz; uses
1800 function <function>DllMain</function> to achieve this. You should
1801 <emphasis>not</emphasis> call <function>nmem_init</function> or
1802 <function>nmem_exit</function> unless you're absolute sure what
1803 you're doing. Note that in previous &yaz; versions you'd have to call
1804 <function>nmem_init</function> yourself.
1809 <sect1 id="tools.log"><title>Log</title>
1811 &yaz; has evolved a fairly complex log system which should be useful both
1812 for debugging &yaz; itself, debugging applications that use &yaz;, and for
1813 production use of those applications.
1816 The log functions are declared in header <filename>yaz/log.h</filename>
1817 and implemented in <filename>src/log.c</filename>.
1818 Due to name clash with syslog and some math utilities the logging
1819 interface has been modified as of YAZ 2.0.29. The obsolete interface
1820 is still available if in header file <filename>yaz/log.h</filename>.
1821 The key points of the interface are:
1824 void yaz_log(int level, const char *fmt, ...)
1826 void yaz_log_init(int level, const char *prefix, const char *name);
1827 void yaz_log_init_file(const char *fname);
1828 void yaz_log_init_level(int level);
1829 void yaz_log_init_prefix(const char *prefix);
1830 void yaz_log_time_format(const char *fmt);
1831 void yaz_log_init_max_size(int mx);
1833 int yaz_log_mask_str(const char *str);
1834 int yaz_log_module_level(const char *name);
1838 The reason for the whole log module is the <function>yaz_log</function>
1839 function. It takes a bitmask indicating the log levels, a
1840 <literal>printf</literal>-like format string, and a variable number of
1845 The <literal>log level</literal> is a bit mask, that says on which level(s)
1846 the log entry should be made, and optionally set some behaviour of the
1847 logging. In the most simple cases, it can be one of <literal>YLOG_FATAL,
1848 YLOG_DEBUG, YLOG_WARN, YLOG_LOG</literal>. Those can be combined with bits
1849 that modify the way the log entry is written:<literal>YLOG_ERRNO,
1850 YLOG_NOTIME, YLOG_FLUSH</literal>.
1851 Most of the rest of the bits are deprecated, and should not be used. Use
1852 the dynamic log levels instead.
1856 Applications that use &yaz;, should not use the LOG_LOG for ordinary
1857 messages, but should make use of the dynamic loglevel system. This consists
1858 of two parts, defining the loglevel and checking it.
1862 To define the log levels, the (main) program should pass a string to
1863 <function>yaz_log_mask_str</function> to define which log levels are to be
1864 logged. This string should be a comma-separated list of log level names,
1865 and can contain both hard-coded names and dynamic ones. The log level
1866 calculation starts with <literal>YLOG_DEFAULT_LEVEL</literal> and adds a bit
1867 for each word it meets, unless the word starts with a '-', in which case it
1868 clears the bit. If the string <literal>'none'</literal> is found,
1869 all bits are cleared. Typically this string comes from the command-line,
1870 often identified by <literal>-v</literal>. The
1871 <function>yaz_log_mask_str</function> returns a log level that should be
1872 passed to <function>yaz_log_init_level</function> for it to take effect.
1876 Each module should check what log bits it should be used, by calling
1877 <function>yaz_log_module_level</function> with a suitable name for the
1878 module. The name is cleared from a preceding path and an extension, if any,
1879 so it is quite possible to use <literal>__FILE__</literal> for it. If the
1880 name has been passed to <function>yaz_log_mask_str</function>, the routine
1881 returns a non-zero bitmask, which should then be used in consequent calls
1882 to yaz_log. (It can also be tested, so as to avoid unnecessary calls to
1883 yaz_log, in time-critical places, or when the log entry would take time
1888 Yaz uses the following dynamic log levels:
1889 <literal>server, session, request, requestdetail</literal> for the server
1891 <literal>zoom</literal> for the zoom client api.
1892 <literal>ztest</literal> for the simple test server.
1893 <literal>malloc, nmem, odr, eventl</literal> for internal debugging of yaz itself.
1894 Of course, any program using yaz is welcome to define as many new ones, as
1899 By default the log is written to stderr, but this can be changed by a call
1900 to <function>yaz_log_init_file</function> or
1901 <function>yaz_log_init</function>. If the log is directed to a file, the
1902 file size is checked at every write, and if it exceeds the limit given in
1903 <function>yaz_log_init_max_size</function>, the log is rotated. The
1904 rotation keeps one old version (with a <literal>.1</literal> appended to
1905 the name). The size defaults to 1GB. Setting it to zero will disable the
1910 A typical yaz-log looks like this
1911 13:23:14-23/11 yaz-ztest(1) [session] Starting session from tcp:127.0.0.1 (pid=30968)
1912 13:23:14-23/11 yaz-ztest(1) [request] Init from 'YAZ' (81) (ver 2.0.28) OK
1913 13:23:17-23/11 yaz-ztest(1) [request] Search Z: @attrset Bib-1 foo OK:7 hits
1914 13:23:22-23/11 yaz-ztest(1) [request] Present: [1] 2+2 OK 2 records returned
1915 13:24:13-23/11 yaz-ztest(1) [request] Close OK
1919 The log entries start with a time stamp. This can be omitted by setting the
1920 <literal>YLOG_NOTIME</literal> bit in the loglevel. This way automatic tests
1921 can be hoped to produce identical log files, that are easy to diff. The
1922 format of the time stamp can be set with
1923 <function>yaz_log_time_format</function>, which takes a format string just
1924 like <function>strftime</function>.
1928 Next in a log line comes the prefix, often the name of the program. For
1929 yaz-based servers, it can also contain the session number. Then
1930 comes one or more logbits in square brackets, depending on the logging
1931 level set by <function>yaz_log_init_level</function> and the loglevel
1932 passed to <function>yaz_log_init_level</function>. Finally comes the format
1933 string and additional values passed to <function>yaz_log</function>
1937 The log level <literal>YLOG_LOGLVL</literal>, enabled by the string
1938 <literal>loglevel</literal>, will log all the log-level affecting
1939 operations. This can come in handy if you need to know what other log
1940 levels would be useful. Grep the logfile for <literal>[loglevel]</literal>.
1944 The log system is almost independent of the rest of &yaz;, the only
1945 important dependence is of <filename>nmem</filename>, and that only for
1946 using the semaphore definition there.
1950 The dynamic log levels and log rotation were introduced in &yaz; 2.0.28. At
1951 the same time, the log bit names were changed from
1952 <literal>LOG_something</literal> to <literal>YLOG_something</literal>,
1953 to avoid collision with <filename>syslog.h</filename>.
1958 <sect1 id="tools.marc"><title>MARC</title>
1961 YAZ provides a fast utility that decodes MARC records and
1962 encodes to a varity of output formats. The MARC records must
1963 be encoded in ISO2709.
1966 #include <yaz/marcdisp.h>
1968 /* create handler */
1969 yaz_marc_t yaz_marc_create(void);
1971 void yaz_marc_destroy(yaz_marc_t mt);
1973 /* set XML mode YAZ_MARC_LINE, YAZ_MARC_SIMPLEXML, ... */
1974 void yaz_marc_xml(yaz_marc_t mt, int xmlmode);
1975 #define YAZ_MARC_LINE 0
1976 #define YAZ_MARC_SIMPLEXML 1
1977 #define YAZ_MARC_OAIMARC 2
1978 #define YAZ_MARC_MARCXML 3
1979 #define YAZ_MARC_ISO2709 4
1980 #define YAZ_MARC_XCHANGE 5
1982 /* supply iconv handle for character set conversion .. */
1983 void yaz_marc_iconv(yaz_marc_t mt, yaz_iconv_t cd);
1985 /* set debug level, 0=none, 1=more, 2=even more, .. */
1986 void yaz_marc_debug(yaz_marc_t mt, int level);
1988 /* decode MARC in buf of size bsize. Returns >0 on success; <=0 on failure.
1989 On success, result in *result with size *rsize. */
1990 int yaz_marc_decode_buf (yaz_marc_t mt, const char *buf, int bsize,
1991 char **result, int *rsize);
1993 /* decode MARC in buf of size bsize. Returns >0 on success; <=0 on failure.
1994 On success, result in WRBUF */
1995 int yaz_marc_decode_wrbuf (yaz_marc_t mt, const char *buf,
1996 int bsize, WRBUF wrbuf);
2000 A MARC conversion handle must be created by using
2001 <function>yaz_marc_create</function> and destroyed
2002 by calling <function>yaz_marc_destroy</function>.
2005 All other function operate on a <literal>yaz_marc_t</literal> handle.
2006 The output is specified by a call to <function>yaz_marc_xml</function>.
2007 The <literal>xmlmode</literal> must be one of
2010 <term>YAZ_MARC_LINE</term>
2013 A simple line-by-line format suitable for display but not
2014 recommend for further (machine) processing.
2020 <term>YAZ_MARC_MARXML</term>
2023 The resulting record is converted to MARCXML.
2029 <term>YAZ_MARC_ISO2709</term>
2032 The resulting record is converted to ISO2709 (MARC).
2039 The actual conversion functions are
2040 <function>yaz_marc_decode_buf</function> and
2041 <function>yaz_marc_decode_wrbuf</function> which decodes and encodes
2042 a MARC record. The former function operates on simple buffers, the
2043 stores the resulting record in a WRBUF handle (WRBUF is a simple string
2047 <title>Display of MARC record</title>
2049 The followint program snippet illustrates how the MARC API may
2050 be used to convert a MARC record to the line-by-line format:
2051 <programlisting><![CDATA[
2052 void print_marc(const char *marc_buf, int marc_buf_size)
2054 char *result; /* for result buf */
2055 int result_len; /* for size of result */
2056 yaz_marc_t mt = yaz_marc_create();
2057 yaz_marc_xml(mt, YAZ_MARC_LINE);
2058 yaz_marc_decode_buf(mt, marc_buf, marc_buf_size,
2059 &result, &result_len);
2060 fwrite(result, result_len, 1, stdout);
2061 yaz_marc_destroy(mt); /* note that result is now freed... */
2071 <!-- Keep this comment at the end of the file
2076 sgml-minimize-attributes:nil
2077 sgml-always-quote-attributes:t
2080 sgml-parent-document: "yaz.xml"
2081 sgml-local-catalogs: nil
2082 sgml-namecase-general:t