1 <!-- $Id: tools.xml,v 1.25 2003-06-19 23:05:29 adam Exp $ -->
2 <chapter id="tools"><title>Supporting Tools</title>
5 In support of the service API - primarily the ASN module, which
6 provides the pro-grammatic interface to the Z39.50 APDUs, &yaz; contains
7 a collection of tools that support the development of applications.
10 <sect1 id="tools.query"><title>Query Syntax Parsers</title>
13 Since the type-1 (RPN) query structure has no direct, useful string
14 representation, every origin application needs to provide some form of
15 mapping from a local query notation or representation to a
16 <token>Z_RPNQuery</token> structure. Some programmers will prefer to
17 construct the query manually, perhaps using
18 <function>odr_malloc()</function> to simplify memory management.
19 The &yaz; distribution includes two separate, query-generating tools
20 that may be of use to you.
23 <sect2 id="PQF"><title>Prefix Query Format</title>
26 Since RPN or reverse polish notation is really just a fancy way of
27 describing a suffix notation format (operator follows operands), it
28 would seem that the confusion is total when we now introduce a prefix
29 notation for RPN. The reason is one of simple laziness - it's somewhat
30 simpler to interpret a prefix format, and this utility was designed
31 for maximum simplicity, to provide a baseline representation for use
32 in simple test applications and scripting environments (like Tcl). The
33 demonstration client included with YAZ uses the PQF.
38 The PQF have been adopted by other parties developing Z39.50
39 software. It is often referred to as Prefix Query Notation
44 The PQF is defined by the pquery module in the YAZ library.
45 There are two sets of function that have similar behavior. First
46 set operates on a PQF parser handle, second set doesn't. First set
47 set of functions are more flexible than the second set. Second set
48 is obsolete and is only provided to ensure backwards compatibility.
51 First set of functions all operate on a PQF parser handle:
54 #include <yaz/pquery.h>
56 YAZ_PQF_Parser yaz_pqf_create (void);
58 void yaz_pqf_destroy (YAZ_PQF_Parser p);
60 Z_RPNQuery *yaz_pqf_parse (YAZ_PQF_Parser p, ODR o, const char *qbuf);
62 Z_AttributesPlusTerm *yaz_pqf_scan (YAZ_PQF_Parser p, ODR o,
63 Odr_oid **attributeSetId, const char *qbuf);
66 int yaz_pqf_error (YAZ_PQF_Parser p, const char **msg, size_t *off);
69 A PQF parser is created and destructed by functions
70 <function>yaz_pqf_create</function> and
71 <function>yaz_pqf_destroy</function> respectively.
72 Function <function>yaz_pqf_parse</function> parses query given
73 by string <literal>qbuf</literal>. If parsing was successful,
74 a Z39.50 RPN Query is returned which is created using ODR stream
75 <literal>o</literal>. If parsing failed, a NULL pointer is
77 Function <function>yaz_pqf_scan</function> takes a scan query in
78 <literal>qbuf</literal>. If parsing was successful, the function
79 returns attributes plus term pointer and modifies
80 <literal>attributeSetId</literal> to hold attribute set for the
81 scan request - both allocated using ODR stream <literal>o</literal>.
82 If parsing failed, yaz_pqf_scan returns a NULL pointer.
83 Error information for bad queries can be obtained by a call to
84 <function>yaz_pqf_error</function> which returns an error code and
85 modifies <literal>*msg</literal> to point to an error description,
86 and modifies <literal>*off</literal> to the offset within last
87 query were parsing failed.
90 The second set of functions are declared as follows:
93 #include <yaz/pquery.h>
95 Z_RPNQuery *p_query_rpn (ODR o, oid_proto proto, const char *qbuf);
97 Z_AttributesPlusTerm *p_query_scan (ODR o, oid_proto proto,
98 Odr_oid **attributeSetP, const char *qbuf);
100 int p_query_attset (const char *arg);
103 The function <function>p_query_rpn()</function> takes as arguments an
104 &odr; stream (see section <link linkend="odr">The ODR Module</link>)
105 to provide a memory source (the structure created is released on
106 the next call to <function>odr_reset()</function> on the stream), a
107 protocol identifier (one of the constants <token>PROTO_Z3950</token> and
108 <token>PROTO_SR</token>), an attribute set reference, and
109 finally a null-terminated string holding the query string.
112 If the parse went well, <function>p_query_rpn()</function> returns a
113 pointer to a <literal>Z_RPNQuery</literal> structure which can be
114 placed directly into a <literal>Z_SearchRequest</literal>.
115 If parsing failed, due to syntax error, a NULL pointer is returned.
118 The <literal>p_query_attset</literal> specifies which attribute set
119 to use if the query doesn't specify one by the
120 <literal>@attrset</literal> operator.
121 The <literal>p_query_attset</literal> returns 0 if the argument is a
122 valid attribute set specifier; otherwise the function returns -1.
126 The grammar of the PQF is as follows:
130 query ::= top-set query-struct.
132 top-set ::= [ '@attrset' string ]
134 query-struct ::= attr-spec | simple | complex | '@term' term-type
136 attr-spec ::= '@attr' [ string ] string query-struct
138 complex ::= operator query-struct query-struct.
140 operator ::= '@and' | '@or' | '@not' | '@prox' proximity.
142 simple ::= result-set | term.
144 result-set ::= '@set' string.
148 proximity ::= exclusion distance ordered relation which-code unit-code.
150 exclusion ::= '1' | '0' | 'void'.
152 distance ::= integer.
154 ordered ::= '1' | '0'.
156 relation ::= integer.
158 which-code ::= 'known' | 'private' | integer.
160 unit-code ::= integer.
162 term-type ::= 'general' | 'numeric' | 'string' | 'oid' | 'datetime' | 'null'.
166 You will note that the syntax above is a fairly faithful
167 representation of RPN, except for the Attribute, which has been
168 moved a step away from the term, allowing you to associate one or more
169 attributes with an entire query structure. The parser will
170 automatically apply the given attributes to each term as required.
174 The @attr operator is followed by an attribute specification
175 (<literal>attr-spec</literal> above). The specification consists
176 of optional an attribute set, an attribute type-value pair and
177 a sub query. The attribute type-value pair is packed in one string:
178 an attribute type, a dash, followed by an attribute value.
179 The type is always an integer but the value may be either an
180 integer or a string (if it doesn't start with a digit character).
184 Version 3 of the Z39.50 specification defines various encoding of terms.
185 Use <literal>@term </literal> <replaceable>type</replaceable>
186 <replaceable>string</replaceable>,
187 where type is one of: <literal>general</literal>,
188 <literal>numeric</literal> or <literal>string</literal>
189 (for InternationalString).
190 If no term type has been given, the <literal>general</literal> form
191 is used. This is the only encoding allowed in both versions 2 and 3
192 of the Z39.50 standard.
195 <sect3 id="PQF-prox">
196 <title>Using Proximity Operators with PQF</title>
199 This is an advanced topic, describing how to construct
200 queries that make very specific requirements on the
201 relative location of their operands.
202 You may wish to skip this section and go straight to
203 <link linkend="pqf-examples">the example PQF queries</link>.
208 Most Z39.50 servers do not support proximity searching, or
209 support only a small subset of the full functionality that
210 can be expressed using the PQF proximity operator. Be
211 aware that the ability to <emphasis>express</emphasis> a
212 query in PQF is no guarantee that any given server will
213 be able to <emphasis>execute</emphasis> it.
219 The proximity operator <literal>@prox</literal> is a special
220 and more restrictive version of the conjunction operator
221 <literal>@and</literal>. Its semantics are described in
222 section 3.7.2 (Proximity) of Z39.50 the standard itself, which
223 can be read on-line at
224 <ulink url="http://lcweb.loc.gov/z3950/agency/markup/09.html"/>
227 In PQF, the proximity operation is represented by a sequence
230 @prox <replaceable>exclusion</replaceable> <replaceable>distance</replaceable> <replaceable>ordered</replaceable> <replaceable>relation</replaceable> <replaceable>which-code</replaceable> <replaceable>unit-code</replaceable>
232 in which the meanings of the parameters are as described in in
233 the standard, and they can take the following values:
235 <listitem><formalpara><title>exclusion</title><para>
236 0 = false (i.e. the proximity condition specified by the
237 remaining parameters must be satisfied) or
238 1 = true (the proximity condition specified by the
239 remaining parameters must <emphasis>not</emphasis> be
241 </para></formalpara></listitem>
242 <listitem><formalpara><title>distance</title><para>
243 An integer specifying the difference between the locations
244 of the operands: e.g. two adjacent words would have
245 distance=1 since their locations differ by one unit.
246 </para></formalpara></listitem>
247 <listitem><formalpara><title>ordered</title><para>
248 1 = ordered (the operands must occur in the order the
249 query specifies them) or
250 0 = unordered (they may appear in either order).
251 </para></formalpara></listitem>
252 <listitem><formalpara><title>relation</title><para>
253 Recognised values are
257 4 (greaterThanOrEqual),
260 </para></formalpara></listitem>
261 <listitem><formalpara><title>which-code</title><para>
262 <literal>known</literal>
265 (the unit-code parameter is taken from the well-known list
266 of alternatives described in below) or
267 <literal>private</literal>
270 (the unit-code paramater has semantics specific to an
271 out-of-band agreement such as a profile).
272 </para></formalpara></listitem>
273 <listitem><formalpara><title>unit-code</title><para>
274 If the which-code parameter is <literal>known</literal>
275 then the recognised values are
287 If which-code is <literal>private</literal> then the
288 acceptable values are determined by the profile.
289 </para></formalpara></listitem>
291 (The numeric values of the relation and well-known unit-code
292 parameters are taken straight from
293 <ulink url="http://lcweb.loc.gov/z3950/agency/asn1.html#ProximityOperator"
294 >the ASN.1</ulink> of the proximity structure in the standard.)
298 <sect3 id="pqf-examples"><title>PQF queries</title>
300 <para>Queries using simple terms.
306 <para>Boolean operators.
308 @or "dylan" "zimmerman"
309 @and @or dylan zimmerman when
310 @and when @or dylan zimmerman
314 Reference to result sets.
321 Attributes for terms.
324 @attr 1=4 @attr 4=1 "self portrait"
325 @attr exp1 @attr 1=1 CategoryList
326 @attr gils 1=2008 Copenhagen
327 @attr 1=/book/title computer
333 @prox 0 3 1 2 k 2 dylan zimmerman
336 Here the parameters 0, 3, 1, 2, k and 2 represent exclusion,
337 distance, ordered, relation, which-code and unit-code, in that
341 exclusion = 0: the proximity condition must hold
344 distance = 3: the terms must be three units apart
347 ordered = 1: they must occur in the order they are specified
350 relation = 2: lessThanOrEqual (to the distance of 3 units)
353 which-code is ``known'', so the standard unit-codes are used
359 So the whole proximity query means that the words
360 <literal>dylan</literal> and <literal>zimmerman</literal> must
361 both occur in the record, in that order, differing in position
362 by three or fewer words (i.e. with two or fewer words between
363 them.) The query would find ``Bob Dylan, aka. Robert
364 Zimmerman'', but not ``Bob Dylan, born as Robert Zimmerman''
365 since the distance in this case is four.
369 Specifying term type.
371 @term string "a UTF-8 string, maybe?"
376 @or @and bob dylan @set Result-1
378 @attr 4=1 @and @attr 1=1 "bob dylan" @attr 1=4 "slow train coming"
380 @and @attr 2=4 @attr gils 1=2038 -114 @attr 2=2 @attr gils 1=2039 -109
384 The last of these examples is a spatial search: in
385 <ulink url="http://www.gils.net/prof_v2.html#sec_7_4"
386 >the GILS attribute set</ulink>,
388 2038 indicates West Bounding Coordinate and
389 2030 indicates East Bounding Coordinate,
390 so the query is for areas extending from -114 degrees
391 to no more than -109 degrees.
397 <sect2 id="CCL"><title>CCL</title>
400 Not all users enjoy typing in prefix query structures and numerical
401 attribute values, even in a minimalistic test client. In the library
402 world, the more intuitive Common Command Language - CCL (ISO 8777)
403 has enjoyed some popularity - especially before the widespread
404 availability of graphical interfaces. It is still useful in
405 applications where you for some reason or other need to provide a
406 symbolic language for expressing boolean query structures.
410 The <ulink url="http://europagate.dtv.dk/">EUROPAGATE</ulink>
411 research project working under the Libraries programme
412 of the European Commission's DG XIII has, amongst other useful tools,
413 implemented a general-purpose CCL parser which produces an output
414 structure that can be trivially converted to the internal RPN
415 representation of &yaz; (The <literal>Z_RPNQuery</literal> structure).
416 Since the CCL utility - along with the rest of the software
417 produced by EUROPAGATE - is made freely available on a liberal
418 license, it is included as a supplement to &yaz;.
421 <sect3><title>CCL Syntax</title>
424 The CCL parser obeys the following grammar for the FIND argument.
425 The syntax is annotated by in the lines prefixed by
426 <literal>‐‐</literal>.
430 CCL-Find ::= CCL-Find Op Elements
433 Op ::= "and" | "or" | "not"
434 -- The above means that Elements are separated by boolean operators.
436 Elements ::= '(' CCL-Find ')'
439 | Qualifiers Relation Terms
440 | Qualifiers Relation '(' CCL-Find ')'
441 | Qualifiers '=' string '-' string
442 -- Elements is either a recursive definition, a result set reference, a
443 -- list of terms, qualifiers followed by terms, qualifiers followed
444 -- by a recursive definition or qualifiers in a range (lower - upper).
446 Set ::= 'set' = string
447 -- Reference to a result set
449 Terms ::= Terms Prox Term
451 -- Proximity of terms.
455 -- This basically means that a term may include a blank
457 Qualifiers ::= Qualifiers ',' string
459 -- Qualifiers is a list of strings separated by comma
461 Relation ::= '=' | '>=' | '<=' | '<>' | '>' | '<'
462 -- Relational operators. This really doesn't follow the ISO8777
466 -- Proximity operator
470 <example><title>CCL queries</title>
472 The following queries are all valid:
484 (dylan and bob) or set=1
488 Assuming that the qualifiers <literal>ti</literal>,
489 <literal>au</literal>
490 and <literal>date</literal> are defined we may use:
496 au=(bob dylan and slow train coming)
498 date>1980 and (ti=((self portrait)))
504 <sect3><title>CCL Qualifiers</title>
507 Qualifiers are used to direct the search to a particular searchable
508 index, such as title (ti) and author indexes (au). The CCL standard
509 itself doesn't specify a particular set of qualifiers, but it does
510 suggest a few short-hand notations. You can customize the CCL parser
511 to support a particular set of qualifiers to reflect the current target
512 profile. Traditionally, a qualifier would map to a particular
513 use-attribute within the BIB-1 attribute set. It is also
514 possible to set other attributes, such as the structure
519 A CCL profile is a set of predefined CCL qualifiers that may be
520 read from a file or set in the CCL API.
521 The YAZ client reads its CCL qualifiers from a file named
522 <filename>default.bib</filename>. There are four types of
523 lines in a CCL profile: qualifier specification,
524 qualifier alias, comments and directives.
526 <sect4><title id="qualifier-specification">Qualifier specification</title>
528 A qualifier specification is of the form:
532 <replaceable>qualifier-name</replaceable>
533 [<replaceable>attributeset</replaceable><literal>,</literal>]<replaceable>type</replaceable><literal>=</literal><replaceable>val</replaceable>
534 [<replaceable>attributeset</replaceable><literal>,</literal>]<replaceable>type</replaceable><literal>=</literal><replaceable>val</replaceable> ...
538 where <replaceable>qualifier-name</replaceable> is the name of the
539 qualifier to be used (eg. <literal>ti</literal>),
540 <replaceable>type</replaceable> is attribute type in the attribute
541 set (Bib-1 is used if no attribute set is given) and
542 <replaceable>val</replaceable> is attribute value.
543 The <replaceable>type</replaceable> can be specified as an
544 integer or as it be specified either as a single-letter:
545 <literal>u</literal> for use,
546 <literal>r</literal> for relation,<literal>p</literal> for position,
547 <literal>s</literal> for structure,<literal>t</literal> for truncation
548 or <literal>c</literal> for completeness.
549 The attributes for the special qualifier name <literal>term</literal>
550 are used when no CCL qualifier is given in a query.
553 The attribute value <replaceable>val</replaceable> may be
554 specified as in integer. It is also possible to specify
555 non-numeric values, however, which are used in combination with
556 certain types. The special combinations are:
558 <varlistentry><term><literal>s=pw</literal></term>
560 The structure is set to either word or phrase depending
561 on the number of tokens in a term (phrase-word).
566 <varlistentry><term><literal>s=al</literal></term>
568 Each token in the term is ANDed. (and-list).
569 This does not set the structure at all.
574 <varlistentry><term><literal>s=ol</literal></term>
576 Each token in the term is ORed. (or-list).
577 This does not set the structure at all.
582 <varlistentry><term><literal>r=o</literal></term>
584 Allows operators greather-than, less-than, ... equals and
585 sets relation attribute accordingly (relation ordered).
590 <varlistentry><term><literal>t=l</literal></term>
592 Allows term to be left-truncated.
593 If term is of the form <literal>?x</literal>, the resulting
594 Type-1 term is <literal>x</literal> and truncation is left.
599 <varlistentry><term><literal>t=r</literal></term>
601 Allows term to be right-truncated.
602 If term is of the form <literal>x?</literal>, the resulting
603 Type-1 term is <literal>x</literal> and truncation is right.
608 <varlistentry><term><literal>t=n</literal></term>
610 If term is does not include <literal>?</literal>, the
611 truncation attribute is set to none (100).
616 <varlistentry><term><literal>t=b</literal></term>
618 Allows term to be both left&right truncated.
619 If term is of the form <literal>?x?</literal>, the
620 resulting term is <literal>x</literal> and trunctation is
621 set to both left&right.
628 <example><title>CCL profile</title>
630 Consider the following definition:
640 Three qualifiers are defined, <literal>ti</literal>,
641 <literal>au</literal> and <literal>ranked</literal>.
642 <literal>ti</literal> and <literal>au</literal> both set
643 structure attribute to phrase (s=1).
644 <literal>ti</literal>
645 sets the use-attribute to 4. <literal>au</literal> sets the
647 When no qualifiers are used in the query the structure-attribute is
648 set to free-form-text (105).
651 You can combine attributes. To Search for "ranked title" you
654 ti,ranked=knuth computer
656 which will use "relation is ranked", "use is title", "structure is
661 <sect4><title>Qualifier alias</title>
663 A qualifier alias is of the form:
666 <replaceable>q</replaceable>
667 <replaceable>q1</replaceable> <replaceable>q2</replaceable> ..
670 which declares <replaceable>q</replaceable> to
671 be an alias for <replaceable>q1</replaceable>,
672 <replaceable>q2</replaceable>... such that the CCL
673 query <replaceable>q=x</replaceable> is equivalent to
674 <replaceable>q1=x or w2=x or ...</replaceable>.
678 <sect4><title>Comments</title>
680 Lines with white space or lines that begin with
681 character <literal>#</literal> are treated as comments.
685 <sect4><title>Directives</title>
687 Directive specifications takes the form
689 <para><literal>@</literal><replaceable>directive</replaceable> <replaceable>value</replaceable>
691 <table><title>CCL directives</title>
693 <colspec colwidth="3*" colname="name"></colspec>
694 <colspec colwidth="7*" colname="description"></colspec>
695 <colspec colwidth="2*" colname="default"></colspec>
699 <entry>Description</entry>
700 <entry>Default</entry>
705 <entry>truncation</entry>
706 <entry>Truncation character</entry>
707 <entry><literal>?</literal></entry>
711 <entry>Specifies how multiple fields are to be
712 combined. There are two modes: <literal>or</literal>:
713 multiple qualifier fields are ORed,
714 <literal>merge</literal>: attributes for the qualifier
715 fields are merged and assigned to one term.
717 <entry><literal>merge</literal></entry>
721 <entry>Specificies if CCL operatores and qualifiers should be
722 compared with case sensitivity or not. Specify 0 for
723 case sensitive; 1 for case insensitive.</entry>
724 <entry><literal>0</literal></entry>
729 <entry>Specifies token for CCL operator AND.</entry>
730 <entry><literal>and</literal></entry>
735 <entry>Specifies token for CCL operator OR.</entry>
736 <entry><literal>or</literal></entry>
741 <entry>Specifies token for CCL operator NOT.</entry>
742 <entry><literal>not</literal></entry>
747 <entry>Specifies token for CCL operator SET.</entry>
748 <entry><literal>set</literal></entry>
755 <sect3><title>CCL API</title>
757 All public definitions can be found in the header file
758 <filename>ccl.h</filename>. A profile identifier is of type
759 <literal>CCL_bibset</literal>. A profile must be created with the call
760 to the function <function>ccl_qual_mk</function> which returns a profile
761 handle of type <literal>CCL_bibset</literal>.
765 To read a file containing qualifier definitions the function
766 <function>ccl_qual_file</function> may be convenient. This function
767 takes an already opened <literal>FILE</literal> handle pointer as
768 argument along with a <literal>CCL_bibset</literal> handle.
772 To parse a simple string with a FIND query use the function
775 struct ccl_rpn_node *ccl_find_str (CCL_bibset bibset, const char *str,
776 int *error, int *pos);
779 which takes the CCL profile (<literal>bibset</literal>) and query
780 (<literal>str</literal>) as input. Upon successful completion the RPN
781 tree is returned. If an error occur, such as a syntax error, the integer
782 pointed to by <literal>error</literal> holds the error code and
783 <literal>pos</literal> holds the offset inside query string in which
788 An English representation of the error may be obtained by calling
789 the <literal>ccl_err_msg</literal> function. The error codes are
790 listed in <filename>ccl.h</filename>.
794 To convert the CCL RPN tree (type
795 <literal>struct ccl_rpn_node *</literal>)
796 to the Z_RPNQuery of YAZ the function <function>ccl_rpn_query</function>
797 must be used. This function which is part of YAZ is implemented in
798 <filename>yaz-ccl.c</filename>.
799 After calling this function the CCL RPN tree is probably no longer
800 needed. The <literal>ccl_rpn_delete</literal> destroys the CCL RPN tree.
804 A CCL profile may be destroyed by calling the
805 <function>ccl_qual_rm</function> function.
809 The token names for the CCL operators may be changed by setting the
810 globals (all type <literal>char *</literal>)
811 <literal>ccl_token_and</literal>, <literal>ccl_token_or</literal>,
812 <literal>ccl_token_not</literal> and <literal>ccl_token_set</literal>.
813 An operator may have aliases, i.e. there may be more than one name for
814 the operator. To do this, separate each alias with a space character.
818 <sect2 id="tools.cql"><title>CQL</title>
820 <ulink url="http://www.loc.gov/z3950/agency/zing/cql/">CQL</ulink>
821 - Common Query Language - was defined for the
822 <ulink url="http://www.loc.gov/z3950/agency/zing/srw/">SRW</ulink>
824 In many ways CQL has a similar syntax to CCL.
825 The objective of CQL is different. Where CCL aims to be
826 an end-user language, CQL is <emphasis>the</emphasis> protocol
827 query language for SRW.
831 If you are new to CQL, read the
832 <ulink url="http://zing.z3950.org/cql/intro.html">Gentle
833 Introduction</ulink>.
837 The CQL parser in &yaz; provides the following:
841 It parses and validates a CQL query.
846 It generates a C structure that allows you to convert
847 a CQL query to some other query language, such as SQL.
852 The parser converts a valid CQL query to PQF, thus providing a
853 way to use CQL for both SRW/SRU servers and Z39.50 targets at the
859 The parser converts CQL to
860 <ulink url="http://www.loc.gov/z3950/agency/zing/cql/xcql.html">
862 XCQL is an XML representation of CQL.
863 XCQL is part of the SRW specification. However, since SRU
864 supports CQL only, we don't expect XCQL to be widely used.
865 Furthermore, CQL has the advantage over XCQL that it is
871 <sect3 id="tools.cql.parsing"><title>CQL parsing</title>
873 A CQL parser is represented by the <literal>CQL_parser</literal>
874 handle. Its contents should be considered &yaz; internal (private).
876 #include <yaz/cql.h>
878 typedef struct cql_parser *CQL_parser;
880 CQL_parser cql_parser_create(void);
881 void cql_parser_destroy(CQL_parser cp);
883 A parser is created by <function>cql_parser_create</function> and
884 is destroyed by <function>cql_parser_destroy</function>.
887 To parse a CQL query string, the following function
890 int cql_parser_string(CQL_parser cp, const char *str);
892 A CQL query is parsed by the <function>cql_parser_string</function>
893 which takes a query <parameter>str</parameter>.
894 If the query was valid (no syntax errors), then zero is returned;
895 otherwise -1 is returned to indicate a syntax error.
899 int cql_parser_stream(CQL_parser cp,
900 int (*getbyte)(void *client_data),
901 void (*ungetbyte)(int b, void *client_data),
904 int cql_parser_stdio(CQL_parser cp, FILE *f);
906 The functions <function>cql_parser_stream</function> and
907 <function>cql_parser_stdio</function> parses a CQL query
908 - just like <function>cql_parser_string</function>.
909 The only difference is that the CQL query can be
910 fed to the parser in different ways.
911 The <function>cql_parser_stream</function> uses a generic
912 byte stream as input. The <function>cql_parser_stdio</function>
913 uses a <literal>FILE</literal> handle which is opened for reading.
917 <sect3 id="tools.cql.tree"><title>CQL tree</title>
919 The the query string is valid, the CQL parser
920 generates a tree representing the structure of the
925 struct cql_node *cql_parser_result(CQL_parser cp);
927 <function>cql_parser_result</function> returns the
928 a pointer to the root node of the resulting tree.
931 Each node in a CQL tree is represented by a
932 <literal>struct cql_node</literal>.
933 It is defined as follows:
935 #define CQL_NODE_ST 1
936 #define CQL_NODE_BOOL 2
937 #define CQL_NODE_MOD 3
945 struct cql_node *modifiers;
946 struct cql_node *prefixes;
950 struct cql_node *left;
951 struct cql_node *right;
952 struct cql_node *modifiers;
953 struct cql_node *prefixes;
958 struct cql_node *next;
963 There are three kinds of nodes, search term (ST), boolean (BOOL),
967 The search term node has five members:
971 <literal>index</literal>: index for search term.
972 If an index is unspecified for a search term,
973 <literal>index</literal> will be NULL.
978 <literal>term</literal>: the search term itself.
983 <literal>relation</literal>: relation for search term.
988 <literal>modifiers</literal>: relation modifiers for search
989 term. The <literal>modifiers</literal> is a simple linked
990 list (NULL for last entry). Each relation modifier node
991 is of type <literal>MOD</literal>.
996 <literal>prefixes</literal>: index prefixes for search
997 term. The <literal>prefixes</literal> is a simple linked
998 list (NULL for last entry). Each prefix node
999 is of type <literal>MOD</literal>.
1006 The boolean node represents both <literal>and</literal>,
1007 <literal>or</literal>, not as well as
1012 <literal>left</literal> and <literal>right</literal>: left
1013 - and right operand respectively.
1018 <literal>modifiers</literal>: proximity arguments.
1023 <literal>prefixes</literal>: index prefixes.
1024 The <literal>prefixes</literal> is a simple linked
1025 list (NULL for last entry). Each prefix node
1026 is of type <literal>MOD</literal>.
1033 The modifier node is a "utility" node used for name-value pairs,
1034 such as prefixes, proximity arguements, etc.
1038 <literal>name</literal> name of mod node.
1043 <literal>value</literal> value of mod node.
1048 <literal>next</literal>: pointer to next node which is
1049 always a mod node (NULL for last entry).
1056 <sect3 id="tools.cql.pqf"><title>CQL to PQF conversion</title>
1058 Conversion to PQF (and Z39.50 RPN) is tricky by the fact
1059 that the resulting RPN depends on the Z39.50 target
1060 capabilities (combinations of supported attributes).
1061 In addition, the CQL and SRW operates on index prefixes
1062 (URI or strings), whereas the RPN uses Object Identifiers
1066 The CQL library of &yaz; defines a <literal>cql_transform_t</literal>
1067 type. It represents a particular mapping between CQL and RPN.
1068 This handle is created and destroyed by the functions:
1070 cql_transform_t cql_transform_open_FILE (FILE *f);
1071 cql_transform_t cql_transform_open_fname(const char *fname);
1072 void cql_transform_close(cql_transform_t ct);
1074 The first two functions create a tranformation handle from
1075 either an already open FILE or from a filename respectively.
1078 The handle is destroyed by <function>cql_transform_close</function>
1079 in which case no further reference of the handle is allowed.
1082 When a <literal>cql_transform_t</literal> handle has been created
1083 you can convert to RPN.
1085 int cql_transform_buf(cql_transform_t ct,
1086 struct cql_node *cn, char *out, int max);
1088 This function converts the CQL tree <literal>cn</literal>
1089 using handle <literal>ct</literal>.
1090 For the resulting PQF, you supply a buffer <literal>out</literal>
1091 which must be able to hold at at least <literal>max</literal>
1095 If conversion failed, <function>cql_transform_buf</function>
1096 returns a non-zero SRW error code; otherwise zero is returned
1097 (conversion successful). The meanings of the numeric error
1098 codes are listed in the SRW specifications at
1099 <ulink url="http://www.loc.gov/srw/diagnostic-list.html"/>
1102 If conversion fails, more information can be obtained by calling
1104 int cql_transform_error(cql_transform_t ct, char **addinfop);
1106 This function returns the most recently returned numeric
1107 error-code and sets the string-pointer at
1108 <literal>*addinfop</literal> to point to a string containing
1109 additional information about the error that occurred: for
1110 example, if the error code is 15 (``Illegal or unsupported index
1111 set''), the additional information is the name of the requested
1112 index set that was not recognised.
1115 The SRW error-codes may be translated into brief human-readable
1116 error messages using
1118 const char *cql_strerror(int code);
1122 If you wish to be able to produce a PQF result in a different
1123 way, there are two alternatives.
1125 void cql_transform_pr(cql_transform_t ct,
1126 struct cql_node *cn,
1127 void (*pr)(const char *buf, void *client_data),
1130 int cql_transform_FILE(cql_transform_t ct,
1131 struct cql_node *cn, FILE *f);
1133 The former function produces output to a user-defined
1134 output stream. The latter writes the result to an already
1135 open <literal>FILE</literal>.
1138 <sect3 id="tools.cql.map">
1139 <title>Specification of CQL to RPN mapping</title>
1141 The file supplied to functions
1142 <function>cql_transform_open_FILE</function>,
1143 <function>cql_transform_open_fname</function> follows
1144 a structure found in many Unix utilities.
1145 It consists of mapping specifications - one per line.
1146 Lines starting with <literal>#</literal> are ignored (comments).
1149 Each line is of the form
1151 <replaceable>CQL pattern</replaceable><literal> = </literal> <replaceable> RPN equivalent</replaceable>
1155 An RPN pattern is a simple attribute list. Each attribute pair
1158 [<replaceable>set</replaceable>] <replaceable>type</replaceable><literal>=</literal><replaceable>value</replaceable>
1160 The attribute <replaceable>set</replaceable> is optional.
1161 The <replaceable>type</replaceable> is the attribute type,
1162 <replaceable>value</replaceable> the attribute value.
1165 The following CQL patterns are recognized:
1167 <varlistentry><term>
1168 <literal>qualifier.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
1172 This pattern is invoked when a CQL qualifier, such as
1173 dc.title is converted. <replaceable>set</replaceable>
1174 and <replaceable>name</replaceable> is the index set and qualifier
1176 Typically, the RPN specifies an equivalent use attribute.
1179 For terms not bound by a qualifier the pattern
1180 <literal>qualifier.srw.serverChoice</literal> is used.
1181 Here, the prefix <literal>srw</literal> is defined as
1182 <literal>http://www.loc.gov/zing/cql/srw-indexes/v1.0/</literal>.
1183 If this pattern is not defined, the mapping will fail.
1187 <varlistentry><term>
1188 <literal>relation.</literal><replaceable>relation</replaceable>
1192 This pattern specifies how a CQL relation is mapped to RPN.
1193 <replaceable>pattern</replaceable> is name of relation
1194 operator. Since <literal>=</literal> is used as
1195 separator between CQL pattern and RPN, CQL relations
1196 including <literal>=</literal> cannot be
1197 used directly. To avoid a conflict, the names
1198 <literal>ge</literal>,
1199 <literal>eq</literal>,
1200 <literal>le</literal>,
1201 must be used for CQL operators, greater-than-or-equal,
1202 equal, less-than-or-equal respectively.
1203 The RPN pattern is supposed to include a relation attribute.
1206 For terms not bound by a relation, the pattern
1207 <literal>relation.scr</literal> is used. If the pattern
1208 is not defined, the mapping will fail.
1211 The special pattern, <literal>relation.*</literal> is used
1212 when no other relation pattern is matched.
1217 <varlistentry><term>
1218 <literal>relationModifier.</literal><replaceable>mod</replaceable>
1222 This pattern specifies how a CQL relation modifier is mapped to RPN.
1223 The RPN pattern is usually a relation attribute.
1228 <varlistentry><term>
1229 <literal>structure.</literal><replaceable>type</replaceable>
1233 This pattern specifies how a CQL structure is mapped to RPN.
1234 Note that this CQL pattern is somewhat to similar to
1235 CQL pattern <literal>relation</literal>.
1236 The <replaceable>type</replaceable> is a CQL relation.
1239 The pattern, <literal>structure.*</literal> is used
1240 when no other structure pattern is matched.
1241 Usually, the RPN equivalent specifies a structure attribute.
1246 <varlistentry><term>
1247 <literal>position.</literal><replaceable>type</replaceable>
1251 This pattern specifies how the anchor (position) of
1252 CQL is mapped to RPN.
1253 The <replaceable>type</replaceable> is one
1254 of <literal>first</literal>, <literal>any</literal>,
1255 <literal>last</literal>, <literal>firstAndLast</literal>.
1258 The pattern, <literal>position.*</literal> is used
1259 when no other position pattern is matched.
1264 <varlistentry><term>
1265 <literal>set.</literal><replaceable>prefix</replaceable>
1269 This specification defines a CQL index set for a given prefix.
1270 The value on the right hand side is the URI for the set -
1271 <emphasis>not</emphasis> RPN. All prefixes used in
1272 qualifier patterns must be defined this way.
1278 <example><title>CQL to RPN mapping file</title>
1280 This simple file defines two index sets, three qualifiers and three
1281 relations, a position pattern and a default structure.
1283 <programlisting><![CDATA[
1284 set.srw = http://www.loc.gov/zing/cql/srw-indexes/v1.0/
1285 set.dc = http://www.loc.gov/zing/cql/dc-indexes/v1.0/
1287 qualifier.srw.serverChoice = 1=1016
1288 qualifier.dc.title = 1=4
1289 qualifier.dc.subject = 1=21
1295 position.any = 3=3 6=1
1301 With the mappings above, the CQL query
1305 is converted to the PQF:
1307 @attr 1=1016 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "computer"
1309 by rules <literal>qualifier.srw.serverChoice</literal>,
1310 <literal>relation.scr</literal>, <literal>structure.*</literal>,
1311 <literal>position.any</literal>.
1318 is rejected, since <literal>position.right</literal> is
1324 >my = "http://www.loc.gov/zing/cql/dc-indexes/v1.0/" my.title = x
1328 @attr 1=4 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "x"
1333 <sect3 id="tools.cql.xcql"><title>CQL to XCQL conversion</title>
1335 Conversion from CQL to XCQL is trivial and does not
1336 require a mapping to be defined.
1337 There three functions to choose from depending on the
1338 way you wish to store the resulting output (XML buffer
1341 int cql_to_xml_buf(struct cql_node *cn, char *out, int max);
1342 void cql_to_xml(struct cql_node *cn,
1343 void (*pr)(const char *buf, void *client_data),
1345 void cql_to_xml_stdio(struct cql_node *cn, FILE *f);
1347 Function <function>cql_to_xml_buf</function> converts
1348 to XCQL and stores result in a user supplied buffer of a given
1352 <function>cql_to_xml</function> writes the result in
1353 a user defined output stream.
1354 <function>cql_to_xml_stdio</function> writes to a
1360 <sect1 id="tools.oid"><title>Object Identifiers</title>
1363 The basic YAZ representation of an OID is an array of integers,
1364 terminated with the value -1. The &odr; module provides two
1365 utility-functions to create and copy this type of data elements:
1369 Odr_oid *odr_getoidbystr(ODR o, char *str);
1373 Creates an OID based on a string-based representation using dots (.)
1374 to separate elements in the OID.
1378 Odr_oid *odr_oiddup(ODR odr, Odr_oid *o);
1382 Creates a copy of the OID referenced by the <emphasis>o</emphasis>
1384 Both functions take an &odr; stream as parameter. This stream is used to
1385 allocate memory for the data elements, which is released on a
1386 subsequent call to <function>odr_reset()</function> on that stream.
1390 The OID module provides a higher-level representation of the
1391 family of object identifiers which describe the Z39.50 protocol and its
1392 related objects. The definition of the module interface is given in
1393 the <filename>oid.h</filename> file.
1397 The interface is mainly based on the <literal>oident</literal> structure.
1398 The definition of this structure looks like this:
1402 typedef struct oident
1407 int oidsuffix[OID_SIZE];
1413 The proto field takes one of the values
1422 If you don't care about talking to SR-based implementations (few
1423 exist, and they may become fewer still if and when the ISO SR and ANSI
1424 Z39.50 documents are merged into a single standard), you can ignore
1425 this field on incoming packages, and always set it to PROTO_Z3950
1426 for outgoing packages.
1430 The oclass field takes one of the values
1452 corresponding to the OID classes defined by the Z39.50 standard.
1454 Finally, the value field takes one of the values
1512 again, corresponding to the specific OIDs defined by the standard.
1516 The desc field contains a brief, mnemonic name for the OID in question.
1524 struct oident *oid_getentbyoid(int *o);
1528 takes as argument an OID, and returns a pointer to a static area
1529 containing an <literal>oident</literal> structure. You typically use
1530 this function when you receive a PDU containing an OID, and you wish
1531 to branch out depending on the specific OID value.
1539 int *oid_ent_to_oid(struct oident *ent, int *dst);
1543 Takes as argument an <literal>oident</literal> structure - in which
1544 the <literal>proto</literal>, <literal>oclass</literal>/, and
1545 <literal>value</literal> fields are assumed to be set correctly -
1546 and returns a pointer to a the buffer as given by <literal>dst</literal>
1548 representation of the corresponding OID. The function returns
1549 NULL and the array dst is unchanged if a mapping couldn't place.
1550 The array <literal>dst</literal> should be at least of size
1551 <literal>OID_SIZE</literal>.
1555 The <function>oid_ent_to_oid()</function> function can be used whenever
1556 you need to prepare a PDU containing one or more OIDs. The separation of
1557 the <literal>protocol</literal> element from the remainder of the
1558 OID-description makes it simple to write applications that can
1559 communicate with either Z39.50 or OSI SR-based applications.
1567 oid_value oid_getvalbyname(const char *name);
1571 takes as argument a mnemonic OID name, and returns the
1572 <literal>/value</literal> field of the first entry in the database that
1573 contains the given name in its <literal>desc</literal> field.
1577 Finally, the module provides the following utility functions, whose
1578 meaning should be obvious:
1582 void oid_oidcpy(int *t, int *s);
1583 void oid_oidcat(int *t, int *s);
1584 int oid_oidcmp(int *o1, int *o2);
1585 int oid_oidlen(int *o);
1590 The OID module has been criticized - and perhaps rightly so
1591 - for needlessly abstracting the
1592 representation of OIDs. Other toolkits use a simple
1593 string-representation of OIDs with good results. In practice, we have
1594 found the interface comfortable and quick to work with, and it is a
1595 simple matter (for what it's worth) to create applications compatible
1596 with both ISO SR and Z39.50. Finally, the use of the
1597 <literal>/oident</literal> database is by no means mandatory.
1598 You can easily create your own system for representing OIDs, as long
1599 as it is compatible with the low-level integer-array representation
1606 <sect1 id="tools.nmem"><title>Nibble Memory</title>
1609 Sometimes when you need to allocate and construct a large,
1610 interconnected complex of structures, it can be a bit of a pain to
1611 release the associated memory again. For the structures describing the
1612 Z39.50 PDUs and related structures, it is convenient to use the
1613 memory-management system of the &odr; subsystem (see
1614 <link linkend="odr-use">Using ODR</link>). However, in some circumstances
1615 where you might otherwise benefit from using a simple nibble memory
1616 management system, it may be impractical to use
1617 <function>odr_malloc()</function> and <function>odr_reset()</function>.
1618 For this purpose, the memory manager which also supports the &odr;
1619 streams is made available in the NMEM module. The external interface
1620 to this module is given in the <filename>nmem.h</filename> file.
1624 The following prototypes are given:
1628 NMEM nmem_create(void);
1629 void nmem_destroy(NMEM n);
1630 void *nmem_malloc(NMEM n, int size);
1631 void nmem_reset(NMEM n);
1632 int nmem_total(NMEM n);
1633 void nmem_init(void);
1634 void nmem_exit(void);
1638 The <function>nmem_create()</function> function returns a pointer to a
1639 memory control handle, which can be released again by
1640 <function>nmem_destroy()</function> when no longer needed.
1641 The function <function>nmem_malloc()</function> allocates a block of
1642 memory of the requested size. A call to <function>nmem_reset()</function>
1643 or <function>nmem_destroy()</function> will release all memory allocated
1644 on the handle since it was created (or since the last call to
1645 <function>nmem_reset()</function>. The function
1646 <function>nmem_total()</function> returns the number of bytes currently
1647 allocated on the handle.
1651 The nibble memory pool is shared amongst threads. POSIX
1652 mutex'es and WIN32 Critical sections are introduced to keep the
1653 module thread safe. Function <function>nmem_init()</function>
1654 initializes the nibble memory library and it is called automatically
1655 the first time the <literal>YAZ.DLL</literal> is loaded. &yaz; uses
1656 function <function>DllMain</function> to achieve this. You should
1657 <emphasis>not</emphasis> call <function>nmem_init</function> or
1658 <function>nmem_exit</function> unless you're absolute sure what
1659 you're doing. Note that in previous &yaz; versions you'd have to call
1660 <function>nmem_init</function> yourself.
1666 <!-- Keep this comment at the end of the file
1671 sgml-minimize-attributes:nil
1672 sgml-always-quote-attributes:t
1675 sgml-parent-document: "yaz.xml"
1676 sgml-local-catalogs: nil
1677 sgml-namecase-general:t