1 <?xml version="1.0" standalone="no"?>
2 <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
3 "http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd"
5 <!ENTITY % local SYSTEM "local.ent">
7 <!ENTITY % entities SYSTEM "entities.ent">
9 <!ENTITY % idcommon SYSTEM "common/common.ent">
14 <title>YAZ User's Guide and Reference</title>
16 <author><firstname>Sebastian</firstname><surname>Hammer</surname></author>
17 <author><firstname>Adam</firstname><surname>Dickmeiss</surname></author>
18 <author><firstname>Mike</firstname><surname>Taylor</surname></author>
19 <author><firstname>Heikki</firstname><surname>Levanto</surname></author>
20 <author><firstname>Dennis</firstname><surname>Schafroth</surname></author>
22 <releaseinfo>&version;</releaseinfo>
24 <year>©right-year;</year>
25 <holder>Index Data</holder>
29 This document is the programmer's guide and reference to the &yaz;
30 package version &version;. &yaz; is a compact toolkit that provides
31 access to the Z39.50 and SRU/Solr protocols, as well as a set of
32 higher-level tools for implementing the server and client
34 The documentation can be used on its own, or as a reference when
35 looking at the example applications provided with the package.
40 <imagedata fileref="common/id.png" format="PNG"/>
43 <imagedata fileref="common/id.eps" format="EPS"/>
48 <chapter id="introduction">
49 <title>Introduction</title>
51 &yaz; is a C/C++ library for information retrieval applications
52 using the Z39.50/SRU/Solr protocols for information retrieval.
60 <ulink url="&url.z39.50;">Z39.50</ulink> version 3 support.
61 Amendments and Z39.50-2002 revision is supported.
67 <ulink url="&url.sru;">SRU GET/POST/SOAP</ulink>
68 version 1.1, 1.2 and 2.0 (over HTTP and HTTPS).
73 Includes BER encoders/decoders for the
74 <ulink url="&url.ill;">ISO ILL</ulink>
81 <ulink url="&url.solr;">Apache Solr</ulink> Web Service version 1.4.x
87 Supports the following transports: BER over TCP/IP
88 (<ulink url="&url.ber.over.tcpip;">RFC1729</ulink>),
89 BER over unix local socket, and
90 <ulink url="&url.http.1.1;">HTTP 1.1</ulink>.
95 Secure Socket Layer support using
96 <ulink url="&url.gnutls;">GnuTLS</ulink>.
97 If enabled, &yaz; uses HTTPS transport (for SOAP) or
98 "Secure BER" (for Z39.50).
104 <ulink url="&url.zoom;">ZOOM</ulink> C API implementing
105 Z39.50, SRU and Solr Web Service.
110 The &yaz; library offers a set of useful utilities
111 related to the protocols, such as MARC (ISO2709) parser,
112 CCL (ISO8777) parser,
113 <ulink url="&url.cql;">CQL</ulink>
114 parser, memory management routines, character set conversion.
119 Portable code. &yaz; compiles out-of-the box on most Unixes and
120 on Windows using Microsoft Visual C++.
125 Fast operation. The C based BER encoders/decoders as well
126 as the server component of &yaz; is very fast.
131 Liberal license that allows for commercial use of &yaz;.
137 <sect1 id="introduction.reading">
138 <title>Reading this Manual</title>
140 Most implementors only need to read a fraction of the
141 material in this manual, so a quick walkthrough of the chapters
147 <xref linkend="installation"/> contains installation
148 instructions for &yaz;. You don't need to read this
149 if you expect to download &yaz; binaries.
150 However, the chapter contains information about how
151 to make <emphasis>your</emphasis> application link
157 <xref linkend="zoom"/> describes the ZOOM API of &yaz;.
158 This is definitely worth reading if you wish to develop a Z39.50/SRU
164 <xref linkend="server"/> describes the generic frontend server
165 and explains how to develop server Z39.50/SRU applications for &yaz;.
166 Obviously worth reading if you're to develop a server.
171 <xref linkend="yaz-client"/> describes how to use the &yaz; Z39.50
172 client. If you're a developer and wish to test your server
173 or a server from another party, you might find this chapter
179 <xref linkend="asn"/> documents the most commonly used Z39.50
180 C data structures offered by the &yaz; API. Client
181 developers using ZOOM and non-Z39.50 implementors may skip this.
186 <xref linkend="soap"/> describes how SRU and SOAP is used
187 in &yaz;. Only if you're developing SRU applications
188 this section is a must.
193 <xref linkend="tools"/> contains sections for the various
194 tools offered by &yaz;. Scan through the material quickly
195 and see what's relevant to you! SRU implementors
196 might find the <link linkend="cql">CQL</link> section
202 <xref linkend="odr"/> goes through the details of the
203 ODR module which is the work horse that encodes and decodes
204 BER packages. Implementors using ZOOM only, do <emphasis>not</emphasis>
206 Most other Z39.50 implementors only need to read the first two
207 sections (<xref linkend="odr.introduction"/> and
208 <xref linkend="odr.use"/>).
213 <xref linkend="comstack"/> describes the network layer module
214 COMSTACK. Implementors using ZOOM or the generic frontend server
215 may skip this. Others, presumably, handling client/server
216 communication on their own should read this.
221 <sect1 id="introduction.api">
222 <title>The API</title>
224 The <ulink url="&url.yaz;">&yaz;</ulink>
225 toolkit offers several different levels of access to the
226 <ulink url="&url.z39.50;">ISO23950/Z39.50</ulink>,
227 <ulink url="&url.ill;">ILL</ulink> and
228 <ulink url="&url.sru;">SRU</ulink>
230 The level that you need to use depends on your requirements, and
231 the role (server or client) that you want to implement.
232 If you're developing a client application you should consider the
233 <link linkend="zoom">ZOOM</link> API.
234 It is, by far, the easiest way to develop clients in C.
235 Server implementers should consider the
236 <link linkend="server">generic frontend server</link>.
237 None of those high-level APIs support the whole protocol, but
238 they do include most facilities used in existing Z39.50 applications.
241 If you're using 'exotic' functionality (meaning anything not included in
242 the high-level APIs), developing non-standard extensions to Z39.50 or
243 you're going to develop an ILL application you'll have to learn the lower
247 The YAZ toolkit modules are shown in figure <xref linkend="yaz.layer"/>.
249 <figure id="yaz.layer">
250 <title>YAZ layers</title>
253 <imagedata fileref="apilayer.png" format="PNG"/>
256 <imagedata fileref="apilayer.eps" format="EPS"/>
261 There are four layers.
264 <para>A client or server application (or both).
265 This layer includes ZOOM and the generic frontend server.
270 The second layer provides a C represenation of the
271 protocol units (packages) for Z39.50 ASN.1, ILL ASN.1,
277 The third layer encodes and decodes protocol data units to
278 simple packages (buffer with certain length). The &odr; module
279 encodes and decodes BER whereas the HTTP modules encodes and
280 decodes HTTP ruquests/responses.
285 The lowest layer is &comstack; which exchanges the encoded packages
286 with a peer process over a network.
292 The &asn; module represents the ASN.1 definition of
293 the Z39.50 protocol. It establishes a set of type and
294 structure definitions, with one structure for each of the top-level
295 PDUs, and one structure or type for each of the contained ASN.1 types.
296 For primitive types, or other types that are defined by the ASN.1
297 standard itself (such as the EXTERNAL type), the C representation is
298 provided by the &odr; (Open Data Representation) subsystem.
301 &odr; is a basic mechanism for representing an
302 ASN.1 type in the C programming language, and for implementing BER
303 encoders and decoders for values of that type. The types defined in
304 the &asn; module generally have the prefix <literal>Z_</literal>, and
305 a suffix corresponding to the name of the type in the ASN.1
306 specification of the protocol (generally Z39.50-1995). In the case of
307 base types (those originating in the ASN.1 standard itself), the prefix
308 <literal>Odr_</literal> is sometimes seen. Either way, look for
309 the actual definition in either <filename>z-core.h</filename> (for the types
310 from the protocol), <filename>odr.h</filename> (for the primitive ASN.1
312 The &asn; library also provides functions (which are, in turn,
313 defined using &odr; primitives) for encoding and decoding data values.
314 Their general form is
316 <funcprototype><funcdef>int <function>z_<replaceable>xxx</replaceable></function></funcdef>
317 <paramdef>ODR <parameter>o</parameter></paramdef>
318 <paramdef>Z_<replaceable>xxx</replaceable> **<parameter>p</parameter></paramdef>
319 <paramdef>int <parameter>optional</parameter></paramdef>
320 <paramdef>const char *<parameter>name</parameter></paramdef>
323 (note the lower-case "z" in the function name)
327 If you are using the premade definitions of the &asn; module, and you
328 are not adding a new protocol of your own, the only parts of &odr; that you
329 need to worry about are documented in
330 <xref linkend="odr.use"/>.
334 When you have created a BER-encoded buffer, you can use the &comstack;
335 subsystem to transmit (or receive) data over the network. The &comstack;
336 module provides simple functions for establishing a connection
337 (passively or actively, depending on the role of your application),
338 and for exchanging BER-encoded PDUs over that connection. When you
339 create a connection endpoint, you need to specify what transport to
340 use (TCP/IP, SSL or UNIX sockets).
341 For the remainder of the connection's lifetime, you don't have
342 to worry about the underlying transport protocol at all - the &comstack;
343 will ensure that the correct mechanism is used.
346 We call the combined interfaces to &odr;, &asn;, and &comstack; the service
347 level API. It's the API that most closely models the Z39.50
348 service/protocol definition, and it provides unlimited access to all
349 fields and facilities of the protocol definitions.
352 The reason that the &yaz; service-level API is a conglomerate of the
353 APIs from three different submodules is twofold. First, we wanted to allow
354 the user a choice of different options for each major task. For instance,
355 if you don't like the protocol API provided by &odr;/&asn;, you
356 can use SNACC or BERUtils instead, and still have the benefits of the
357 transparent transport approach of the &comstack; module. Secondly,
358 we realize that you may have to fit the toolkit into an existing
359 event-processing structure, in a way that is incompatible with
360 the &comstack; interface or some other part of &yaz;.
364 <chapter id="installation">
365 <title>Compilation and Installation</title>
366 <sect1 id="installation-introduction">
367 <title>Introduction</title>
369 The latest version of the software will generally be found at:
372 <ulink url="&url.yaz.download;"/>
375 We have tried our best to keep the software portable, and on many
376 platforms, you should be able to compile everything with little or
380 The software is regularly tested on
381 <ulink url="&url.debian;">Debian GNU/Linux</ulink>,
382 <ulink url="&url.centos;">CentOS</ulink>,
383 <ulink url="&url.ubuntu;">Ubuntu Linux</ulink>,
384 <ulink url="&url.freebsd;">FreeBSD (i386)</ulink>,
385 <ulink url="&url.macosx;">MAC OSX</ulink>,
386 <ulink url="&url.solaris;">Solaris</ulink>,
387 Windows 7, Windows XP.
390 Some versions have be known to work on HP/UX,
391 DEC Unix, <ulink url="&url.netbsd;">NetBSD</ulink>,
392 <ulink url="&url.openbsd;">OpenBSD</ulink>,
394 Data General DG/UX (with some CFLAGS tinkering),
395 SGI/IRIX, DDE Supermax, Apple Macintosh (using the Codewarrior programming
396 environment and the GUSI socket libraries),
400 If you move the software to other platforms, we'd be grateful if you'd
401 let us know about it. If you run into difficulties, we will try to help
402 if we can, and if you solve the problems, we would be happy to include
403 your fixes in the next release. So far, we have mostly avoided
404 <literal>#ifdefs</literal> for individual platforms, and we'd
405 like to keep it that way as far as it makes sense.
408 We maintain a mailing-list for the purpose of announcing new releases and
409 bug-fixes, as well as general discussion. Subscribe by
411 <ulink url="&url.yaz.mailinglist;">here</ulink>.
412 General questions and problems can be directed at
413 <ulink url="&url.yaz.mail;"/>, or the address given at the top of
417 <sect1 id="installation.unix"><title>UNIX</title>
420 <ulink url="&url.debian;">Debian GNU/Linux</ulink> (i386 and amd64),
421 <ulink url="&url.ubuntu;">Ubuntu</ulink> (i386 and amd64)
423 <ulink url="&url.centos;">CentOS</ulink> (amd64 only) packages for &yaz;.
424 You should be able to create packages for other CPUs by building
425 them from the source package.
428 YAZ is also part of several packages repositories. Some of them are
433 Solaris CSW: <ulink url="http://www.opencsw.org/packages/yaz/"/>
438 Solaris: <ulink url="http://unixpackages.com"/>
443 FreeBSD: <ulink url="http://www.freshports.org/net/yaz"/>
448 Debian: <ulink url="http://packages.debian.org/search?keywords=yaz"/>
453 Ubuntu: <ulink url="https://launchpad.net/ubuntu/+source/yaz"/>
459 <ulink url="http://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc/net/yaz/README.html"/>
463 <sect2 id="installation.source.unix">
464 <title>Compiling from source on Unix</title>
466 Note that if your system doesn't have a native ANSI C compiler, you may
467 have to acquire one separately. We recommend
468 <ulink url="&url.gcc;">GCC</ulink>.
471 If you wish to use character set conversion facilities in &yaz; or if you
472 are compiling &yaz; for use with Zebra it is a good idea to ensure that
473 the iconv library is installed. Some Unixes today already have it
475 <ulink url="&url.libiconv;">GNU libiconv</ulink>.
478 YAZ 3.0.16 and later includes a wrapper for the
479 <ulink url="&url.icu;">ICU</ulink>
480 (International Components for Unicode).
481 In order to use this, the developer version of the ICU library
482 must be available. ICU support is recommended for applications
483 such as Pazpar2 and Zebra.
486 The <ulink url="&url.libxslt;">libxslt</ulink>,
487 <ulink url="&url.libxml2;">libxml2</ulink> librararies are required
488 if &yaz; is to support SRU/Solr.
489 These libraries are very portable and should compile out-of-the
490 box on virtually all Unix platforms. It is available in binary
491 forms for Linux and others.
495 <ulink url="&url.autoconf;">Autoconf</ulink>,
496 <ulink url="&url.automake;">Automake</ulink> and
497 <ulink url="&url.libtool;">Libtool</ulink>
498 are used to generate Makefiles and configure &yaz; for the system.
499 You do <emphasis>not</emphasis> need these tools unless you're using the
500 Git version of &yaz;.
503 The CQL parser for &yaz; is built using
504 GNU <ulink url="&url.bison;">Bison</ulink>.
505 This tool is only needed if you're using the Git version of &yaz;.
508 &yaz; includes a tiny ASN.1 compiler. This compiler is
509 written in <ulink url="&url.tcl;">Tcl</ulink>.
510 But as for Bison you do not need it unless you're using Git
511 version of &yaz; or you're using the compiler to build your own codecs
515 Generally it should be sufficient to run configure without options,
522 The configure script attempts to use use the C compiler specified by
523 the <literal>CC</literal> environment variable. If not set, GNU C will be
524 used if it is available. The <literal>CFLAGS</literal> environment
525 variable holds options to be passed to the C compiler. If you're using
526 Bourne-compatible shell you may pass something like this to use a
527 particular C compiler with optimization enabled:
530 CC=/opt/ccs/bin/cc CFLAGS=-O ./configure
533 To customize &yaz;, the configure script also accepts a set of options.
534 The most important are:
538 <literal>--prefix</literal>=<replaceable>prefix</replaceable>
541 <para>Specifies installation prefix for &yaz;. This is
542 only needed if you run <literal>make install</literal> later to
543 perform a "system" installation. The prefix is
544 <literal>/usr/local</literal> if not specified.
550 <literal>--enable-tcpd</literal>
553 <para>The front end server will be built using Wietse's
554 <ulink url="&url.tcpwrapper;">TCP wrapper library</ulink>.
555 It allows you to allow/deny clients depending on IP number.
556 The TCP wrapper library is often used in GNU/Linux and
560 <refentrytitle>hosts_access</refentrytitle>
561 <manvolnum>5</manvolnum>
565 <refentrytitle>tcpd</refentrytitle>
566 <manvolnum>8</manvolnum>
573 <literal>--enable-threads</literal>
576 <para>&yaz; will be built using POSIX threads.
577 Specifically, <constant>_REENTRANT</constant> will be defined during
584 <literal>--disable-shared</literal>
587 <para>The make process will not create shared
588 libraries (also known as shared objects <filename>.so</filename>).
589 By default, shared libraries are created -
590 equivalent to <literal>--enable-shared</literal>.
596 <literal>--disable-shared</literal>
599 <para>The make process will not create
600 static libraries (<filename>.a</filename>).
601 By default, static libraries are created -
602 equivalent to <literal>--enable-static</literal>.
608 <literal>--with-iconv</literal>[=<replaceable>prefix</replaceable>]
611 <para>Compile &yaz; with iconv library in directory
612 <replaceable>prefix</replaceable>. By default configure will
613 search for iconv on the system. Use this option if it
614 doesn't find iconv. Alternatively,
615 <literal>--without-iconv</literal>, can be used to force &yaz;
622 <literal>--with-xslt</literal>[=<replaceable>prefix</replaceable>]
625 <para>Compile &yaz; with
626 <ulink url="&url.libxslt;">libxslt</ulink> in directory
627 <replaceable>prefix</replaceable>.
628 Use this option if you want XSLT and XML support.
629 By default, configure will
630 search for libxslt on the system. Use this option if
631 libxslt is not found automatically. Alternatively,
632 <literal>--without-xslt</literal>, can be used to force &yaz;
639 <literal>--with-xml2</literal>[=<replaceable>prefix</replaceable>]
642 <para>Compile &yaz; with
643 <ulink url="&url.libxml2;">libxml2</ulink> in directory
644 <replaceable>prefix</replaceable>.
645 Use this option if you want &yaz; to use XML and support SRU/Solr.
646 By default, configure will
647 search for libxml2 on the system. Use this option if
648 libxml2 is not found automatically. Alternatively,
649 <literal>--without-xml2</literal>, can be used to force &yaz;
653 Note that option <literal>--with-xslt</literal>
654 also enables libxml2.
660 <literal>--with-gnutls</literal>[=<replaceable>prefix</replaceable>]
663 <para>&yaz; will be linked with the GNU TLS libraries and
664 an SSL COMSTACK will be provided. By default configure enables
665 SSL support for YAZ if the GNU TLS development libraries are found
672 <literal>--with-icu</literal>[=<replaceable>prefix</replaceable>]
675 <para>&yaz; will be linked the
676 <ulink url="&url.icu;">ICU</ulink> library in the prefix if given.
677 If prefix is not given, the libraries exposed by the script
678 <application>icu-config</application> will be used if found.
685 <literal>--with-memcached</literal>
688 <para>&yaz; will be linked with
689 <ulink url="&url.libmemcached;">libMemcached</ulink> to allow
690 for result-set caching for ZOOM.
691 The prefix can not be given.
692 Note that 0.40 of libmemcached is required.
698 <literal>--with-redis</literal>
701 <para>&yaz; will be linked with the hiredis C library
702 to allow for result-set caching for ZOOM on a
703 <ulink url="&url.redis;">redis</ulink> server.
704 The prefix can not be given.
712 When configured, build the software by typing:
718 The following files are generated by the make process:
721 <term><filename>src/libyaz.la</filename></term>
723 Main &yaz; library. This is no ordinary library. It's
725 By default, &yaz; creates a static library in
726 <filename>lib/.libs/libyaz.a</filename>.
730 <term><filename>src/libyaz_server.la</filename></term>
732 Generic Frontend server. This is an add-on for libyaz.la.
733 Code in this library uses POSIX threads functions - if POSIX
734 threads are available on the platform.
738 <term><filename>src/libyaz_icu.la</filename></term>
740 Functions that wrap the ICU library.
744 <term><filename>ztest/yaz-ztest</filename></term>
745 <listitem><para>Test Z39.50 server.
749 <term><filename>client/yaz-client</filename></term>
750 <listitem><para>Z39.50 client for testing the protocol.
751 See chapter <link linkend="yaz-client">
752 YAZ client</link> for more information.
756 <term><filename>util/yaz-config</filename></term>
757 <listitem><para>A Bourne-shell script, generated by configure, that
758 specifies how external applications should compile - and link with
763 <term><filename>util/yaz-asncomp</filename></term>
764 <listitem><para>The ASN.1 compiler for &yaz;. Requires the
765 Tcl Shell, <application>tclsh</application>, in
766 <literal>PATH</literal> to operate.
770 <term><filename>util/yaz-iconv</filename></term>
771 <listitem><para>This program converts data in one character set to
772 another. This command exercises the YAZ character set
777 <term><filename>util/yaz-marcdump</filename></term>
778 <listitem><para>This program parses ISO2709 encoded MARC records
779 and prints them in line-format or XML.
783 <term><filename>util/yaz-icu</filename></term>
784 <listitem><para>This program exposes the ICU wrapper library if that
785 is enabled for YAZ. Only if ICU is available this program is
790 <term><filename>util/yaz-url</filename></term>
791 <listitem><para>This program is a simple HTTP page fetcher ala
796 <term><filename>zoom/zoomsh</filename></term>
798 A simple shell implemented on top of the
799 <link linkend="zoom">ZOOM</link> functions.
800 The shell is a command line application that allows you to enter
801 simple commands to perform ZOOM operations.
805 <term><filename>zoom/zoomtst1</filename>,
806 <filename>zoom/zoomtst2</filename>, ..</term>
808 Several small applications that demonstrates the ZOOM API.
814 If you wish to install &yaz; in system directories
815 <filename>/usr/local/bin</filename>,
816 <filename>/usr/local/lib</filename> .. etc, you can type:
822 You probably need to have root access in order to perform this.
823 You must specify the <literal>--prefix</literal> option for configure if
824 you wish to install &yaz; in other directories than the default
825 <filename>/usr/local/</filename>.
828 If you wish to perform an un-installation of &yaz;, use:
834 This will only work if you haven't reconfigured &yaz; (and therefore
835 changed installation prefix). Note that uninstall will not
836 remove directories created by make install, e.g.
837 <filename>/usr/local/include/yaz</filename>.
840 <sect2 id="installation-linking-yaz-unix">
841 <title>How to make apps using YAZ on UNIX</title>
843 This section describes how to compile - and link your own
844 applications using the &yaz; toolkit.
845 If you're used to Makefiles this shouldn't be hard. As for
846 other libraries you have used before, you need to set a proper include
847 path for your C/C++ compiler and specify the location of
848 &yaz; libraries. You can do it by hand, but generally we suggest
849 you use the <filename>yaz-config</filename> that is generated
850 by <filename>configure</filename>. This is especially
851 important if you're using the threaded version of &yaz; which
852 require you to pass more options to your linker/compiler.
855 The <filename>yaz-config</filename> script accepts command line
856 options that makes the <filename>yaz-config</filename> script print
857 options that you should use in your make process.
858 The most important ones are:
859 <literal>--cflags</literal>, <literal>--libs</literal>
860 which prints C compiler flags, and linker flags respectively.
863 A small and complete <literal>Makefile</literal> for a C
864 application consisting of one source file,
865 <filename>myprog.c</filename>, may look like this:
867 YAZCONFIG=/usr/local/bin/yaz-config
868 CFLAGS=`$(YAZCONFIG) --cflags`
869 LIBS=`$(YAZCONFIG) --libs`
871 $(CC) $(CFLAGS) -o myprog myprog.o $(LIBS)
875 The CFLAGS variable consists of a C compiler directive that will set
876 the include path to the <emphasis>parent</emphasis> directory
877 of <filename>yaz</filename>. That is, if &yaz; header files were
878 installed in <filename>/usr/local/include/yaz</filename>,
879 then include path is set to <filename>/usr/local/include</filename>.
880 Therefore, in your applications you should use
882 #include <yaz/proto.h>
884 and <emphasis>not</emphasis>
886 #include <proto.h>
890 For Libtool users, the <filename>yaz-config</filename> script provides
891 a different variant of option <literal>--libs</literal>, called
892 <literal>--lalibs</literal> that returns the name of the
893 Libtool archive(s) for &yaz; rather than the ordinary ones.
896 For applications using the threaded version of &yaz;,
897 specify <literal>threads</literal> after the
898 other options. When <literal>threads</literal> is given,
899 more flags and linker flags will be printed by
900 <filename>yaz-config</filename>. If our previous example was
901 using threads, you'd have to modify the lines that set
902 <literal>CFLAGS</literal> and <literal>LIBS</literal> as
905 CFLAGS=`$(YAZCONFIG) --cflags threads`
906 LIBS=`$(YAZCONFIG) --libs threads`
908 There is no need specify POSIX thread libraries in your Makefile.
909 The <literal>LIBS</literal> variable includes that as well.
913 <sect1 id="installation.win32">
914 <title>Windows</title>
915 <para>The easiest way to install YAZ on Windows is by downloading
917 <ulink url="&url.yaz.download.win32;">here</ulink>.
918 The installer comes with source too - in case you wish to
919 compile YAZ with different compiler options, etc.
922 <sect2 id="installation.win32.source">
923 <title>Compiling from Source on Windows</title>
925 &yaz; is shipped with "makefiles" for the NMAKE tool that comes
926 with <ulink url="&url.vstudio;">
927 Microsoft Visual Studio</ulink>. It has been tested with
928 Microsoft Visual Studio 2015.
931 Start a command prompt and switch the sub directory
932 <filename>WIN</filename> where the file <filename>makefile</filename>
933 is located. Customize the installation by editing the
934 <filename>makefile</filename> file (for example by using notepad).
935 The following summarizes the most important settings in that file:
938 <term><literal>DEBUG</literal></term>
940 If set to 1, the software is
941 compiled with debugging libraries (code generation is
942 multi-threaded debug DLL).
943 If set to 0, the software is compiled with release libraries
944 (code generation is multi-threaded DLL).
948 <term><literal>HAVE_TCL</literal>, <literal>TCL</literal></term>
950 If <literal>HAVE_TCL</literal> is set to 1, nmake will
951 use the ASN.1 compiler (<ulink url="&url.tcl;">Tcl</ulink> based).
952 You must set <literal>TCL</literal> to the full path of the Tcl
953 interpreter. A Windows version of Tcl is part of
954 <ulink url="&url.gitwindows;">Git for Windows</ulink>.
957 If you do not have Tcl installed, set
958 <literal>HAVE_TCL</literal> to 0.
962 <term><literal>HAVE_BISON</literal>,
963 <literal>BISON</literal></term>
965 If GNU Bison is present, you might set <literal>HAVE_BISON</literal>
966 to 1 and specify the Bison executable in <literal>BISON</literal>.
967 Bison is only required if you use the Git version of
968 YAZ or if you modify the grammar for CQL
969 (<filename>cql.y</filename>).
972 A Windows version of GNU Bison is part of
973 <ulink url="&url.gitwindows;">Git for Windows</ulink>.
977 <term><literal>HAVE_ICONV</literal>,
978 <literal>ICONV_DIR</literal></term>
980 If <literal>HAVE_ICONV</literal> is set to 1, YAZ is compiled
981 with iconv support. In this configuration, set
982 <literal>ICONV_DIR</literal> to the iconv source directory.
986 <term><literal>HAVE_LIBXML2</literal>,
987 <literal>LIBXML2_DIR</literal></term>
990 If <literal>HAVE_LIBXML2</literal> is set to 1, YAZ is compiled
991 with SRU support. In this configuration, set
992 <literal>LIBXML2_DIR</literal> to the
993 <ulink url="&url.libxml2;">libxml2</ulink> source directory.
996 You can get pre-compiled Libxml2+Libxslt DLLs and headers from
997 <ulink url="&url.libxml2.download.windows;">here</ulink>.
998 Should you with to compile those libraries yourself, refer to
999 to <xref linkend="installation.windows.libxml2"/>
1004 <term><literal>HAVE_LIBXSLT</literal>,
1005 <literal>LIBXSLT_DIR</literal></term>
1008 If <literal>HAVE_LIBXSLT</literal> is set to 1, YAZ is compiled
1009 with XSLT support. In this configuration, set
1010 <literal>LIBXSLT_DIR</literal> to the
1011 <ulink url="&url.libxslt;">libxslt</ulink> source directory.
1015 libxslt depends on libxml2.
1021 <term><literal>HAVE_ICU</literal>,
1022 <literal>ICU_DIR</literal></term>
1025 If <literal>HAVE_ICU</literal> is set to 1, YAZ is compiled
1026 with <ulink url="&url.icu;">ICU</ulink> support.
1027 In this configuration, set
1028 <literal>ICU_DIR</literal> to the
1029 <ulink url="&url.icu;">ICU</ulink> source directory.
1036 When satisfied with the settings in the makefile, type
1043 If the <filename>nmake</filename> command is not found on your system
1044 you probably haven't defined the environment variables required to
1045 use that tool. To fix that, find and run the batch file
1046 <filename>vcvars32.bat</filename>. You need to run it from within
1047 the command prompt or set the environment variables "globally";
1048 otherwise it doesn't work.
1052 If you wish to recompile &yaz; - for example if you modify
1053 settings in the <filename>makefile</filename> you can delete
1054 object files, etc by running.
1060 The following files are generated upon successful compilation:
1063 <term><filename>bin/yaz&soversion;.dll</filename> /
1064 <filename>bin/yaz&soversion;d.dll</filename></term>
1066 &yaz; Release/Debug DLL.
1070 <term><filename>lib/yaz&soversion;.lib</filename> /
1071 <filename>lib/yaz&soversion;d.lib</filename></term>
1073 Import library for <filename>yaz&soversion;.dll</filename> /
1074 <filename>yaz&soversion;d.dll</filename>.
1078 <term><filename>bin/yaz_cond&soversion;.dll</filename> /
1079 <filename>bin/yaz_cond&soversion;d.dll</filename></term>
1081 Release/Debug DLL for condition variable utilities (condvar.c).
1085 <term><filename>lib/yaz_cond&soversion;.lib</filename> /
1086 <filename>lib/yaz_cond&soversion;d.lib</filename></term>
1088 Import library for <filename>yaz_cond&soversion;.dll</filename> /
1089 <filename>yaz_cond&soversion;d.dll</filename>.
1093 <term><filename>bin/yaz_icu&soversion;.dll</filename> /
1094 <filename>bin/yaz_icu&soversion;d.dll</filename></term>
1096 Release/Debug DLL for the ICU wrapper utility.
1097 Only build if HAVE_ICU is 1.
1101 <term><filename>lib/yaz_icu&soversion;.lib</filename> /
1102 <filename>lib/yaz_icu&soversion;d.lib</filename></term>
1104 Import library for <filename>yaz_icu&soversion;.dll</filename> /
1105 <filename>yaz_icu&soversion;d.dll</filename>.
1109 <term><filename>bin/yaz-ztest.exe</filename></term>
1111 Z39.50 multi-threaded test/example server. It's a WIN32
1112 console application.
1116 <term><filename>bin/yaz-client.exe</filename></term>
1118 &yaz; Z39.50 client application. It's a WIN32 console application.
1119 See chapter <link linkend="yaz-client">YAZ client</link> for more
1124 <term><filename>bin/yaz-icu.exe</filename></term>
1125 <listitem><para>This program exposes the ICU wrapper library if that
1126 is enabled for YAZ. Only if ICU is available this program is
1131 <term><filename>bin/zoomsh.exe</filename></term>
1133 Simple console application implemented on top of the
1134 <link linkend="zoom">ZOOM</link> functions.
1135 The application is a command line shell that allows you to enter
1136 simple commands to perform ZOOM operations.
1140 <term><filename>bin/zoomtst1.exe</filename>,
1141 <filename>bin/zoomtst2.exe</filename>, ..</term>
1143 Several small applications that demonstrate the ZOOM API.
1150 <sect2 id="installation-linking-yaz-win32">
1151 <title>How to make apps using YAZ on Windows</title>
1153 This section will go though the process of linking your Windows
1154 applications with &yaz;.
1157 Some people are confused by the fact that we use the nmake
1158 tool to build &yaz;. They think they have to do that too - in order
1159 to make their Windows applications work with &yaz;. The good news is that
1160 you don't have to. You can use the integrated environment of
1161 Visual Studio if desired for your own application.
1164 When setting up a project or Makefile you have to set the following:
1167 <term>include path</term>
1169 Set it to the <filename>include</filename> directory of &yaz;.
1173 <term>import library <filename>yaz&soversion;.lib</filename></term>
1175 You must link with this library. It's located in the
1176 sub directory <filename>lib</filename> of &yaz;.
1177 If you want to link with the debug version of &yaz;, you must
1178 link against <filename>yaz&soversion;d.lib</filename> instead.
1182 <term>dynamic link library
1183 <filename>yaz&soversion;.dll</filename>
1186 This DLL must be in your execution path when you invoke
1187 your application. Specifically, you should distribute this
1188 DLL with your application.
1195 <sect2 id="installation.windows.libxml2">
1196 <title>Compiling Libxml2 and Libxslt on windows</title>
1198 Download libxml2 and Libxslt source and unpack it.
1199 In the example below we install Libxml2 2.9.2 and Libxslt 1.1.28
1200 for 32-bit, so we use the destination directories
1201 libxml2.2.9.2.win32 and libxslt-1.1.28.win32 to reflect both
1202 version and architecture.
1205 cscript configure.js prefix=c:\libxml2-2.9.2.win32 iconv=no
1212 There's an error in <filename>configure.js</filename> for Libxml2 2.9.2.
1213 Line 17 should be assigned to <filename>configure.ac</filename>
1214 rather than <filename>configure.in</filename>.
1218 For Libxslt it is similar. We must ensure that compilation of
1219 Libxslt links against the already installed libxml2.
1222 cscript configure.js prefix=c:\libxslt-1.1.28.win32 iconv=no \
1223 lib=c:\libxml2-2.9.2.win32\lib \
1224 include=c:\libxml2-2.9.2.win32\include\libxml2
1234 ### Still to document:
1235 ZOOM_connection_errcode(c)
1236 ZOOM_connection_errmsg(c)
1237 ZOOM_connection_addinfo(c)
1238 ZOOM_connection_addinfo(c)
1239 ZOOM_connection_diagset(c);
1240 ZOOM_connection_save_apdu_wrbuf
1241 ZOOM_diag_str(error)
1242 ZOOM_resultset_record_immediate(s, pos)
1243 ZOOM_resultset_cache_reset(r)
1244 ZOOM_options_set_callback(opt, function, handle)
1245 ZOOM_options_create_with_parent2(parent1, parent2)
1246 ZOOM_options_getl(opt, name, len)
1247 ZOOM_options_setl(opt, name, value, len)
1248 ZOOM_options_get_bool(opt, name, defa)
1249 ZOOM_options_get_int(opt, name, defa)
1250 ZOOM_options_set_int(opt, name, value)
1255 &zoom; is an acronym for 'Z39.50 Object-Orientation Model' and is
1256 an initiative started by Mike Taylor (Mike is from the UK, which
1257 explains the peculiar name of the model). The goal of &zoom; is to
1258 provide a common Z39.50 client API not bound to a particular
1259 programming language or toolkit.
1262 From YAZ version 2.1.12, <ulink url="&url.sru;">SRU</ulink> is supported.
1263 You can make SRU ZOOM connections by specifying scheme
1264 <literal>http://</literal> for the hostname for a connection.
1265 The dialect of SRU used is specified by the value of the
1266 connection's <literal>sru</literal> option, which may be SRU over
1267 HTTP GET (<literal>get</literal>),
1268 SRU over HTTP POST (<literal>post</literal>), (SRU over
1269 SOAP) (<literal>soap</literal>) or <literal>solr</literal>
1270 (<ulink url="&url.solr;">Solr</ulink> Web Service).
1271 Using the facility for embedding options in target strings, a
1272 connection can be forced to use SRU rather the SRW (the default) by
1273 prefixing the target string with <literal>sru=get,</literal>, like this:
1274 <literal>sru=get,http://sru.miketaylor.org.uk:80/sru.pl</literal>
1277 <ulink url="&url.solr;">Solr</ulink> protocol support was added to
1278 YAZ in version 4.1.0, as a dialect of a SRU protocol, since both are
1279 HTTP based protocols.
1282 The lack of a simple Z39.50 client API for &yaz; has become more
1283 and more apparent over time. So when the first &zoom; specification
1285 an implementation for &yaz; was quickly developed. For the first time, it is
1286 now as easy (or easier!) to develop clients as it is to develop
1287 servers with &yaz;. This
1288 chapter describes the &zoom; C binding. Before going further, please
1289 reconsider whether C is the right programming language for the job.
1290 There are other language bindings available for &yaz;, and still
1292 are in active development. See the
1293 <ulink url="&url.zoom;">ZOOM web-site</ulink> for
1297 In order to fully understand this chapter you should read and
1298 try the example programs <literal>zoomtst1.c</literal>,
1299 <literal>zoomtst2.c</literal>, .. in the <literal>zoom</literal>
1303 The C language misses features found in object oriented languages
1304 such as C++, Java, etc. For example, you'll have to manually,
1305 destroy all objects you create, even though you may think of them as
1306 temporary. Most objects have a <literal>_create</literal> - and a
1307 <literal>_destroy</literal> variant.
1308 All objects are in fact pointers to internal stuff, but you don't see
1309 that because of typedefs. All destroy methods should gracefully ignore a
1310 <literal>NULL</literal> pointer.
1313 In each of the sections below you'll find a sub section called
1314 protocol behavior, that describes how the API maps to the Z39.50
1317 <sect1 id="zoom-connections">
1318 <title>Connections</title>
1319 <para>The Connection object is a session with a target.
1322 #include <yaz/zoom.h>
1324 ZOOM_connection ZOOM_connection_new(const char *host, int portnum);
1326 ZOOM_connection ZOOM_connection_create(ZOOM_options options);
1328 void ZOOM_connection_connect(ZOOM_connection c, const char *host,
1330 void ZOOM_connection_destroy(ZOOM_connection c);
1333 Connection objects are created with either function
1334 <function>ZOOM_connection_new</function> or
1335 <function>ZOOM_connection_create</function>.
1336 The former creates and automatically attempts to establish a network
1337 connection with the target. The latter doesn't establish
1338 a connection immediately, thus allowing you to specify options
1339 before establishing network connection using the function
1340 <function>ZOOM_connection_connect</function>.
1341 If the port number, <literal>portnum</literal>, is zero, the
1342 <literal>host</literal> is consulted for a port specification.
1343 If no port is given, 210 is used. A colon denotes the beginning of
1344 a port number in the host string. If the host string includes a
1345 slash, the following part specifies a database for the connection.
1348 You can prefix the host with a scheme followed by colon. The
1349 default scheme is <literal>tcp</literal> (Z39.50 protocol).
1350 The scheme <literal>http</literal> selects SRU/get over HTTP by default,
1351 but can overridden to use SRU/post, SRW, and the Solr protocol.
1354 You can prefix the scheme-qualified host-string with one or more
1356 <literal><parameter>key</parameter>=<parameter>value</parameter></literal>
1357 sequences, each of which represents an option to be set into the
1358 connection structure <emphasis>before</emphasis> the
1359 protocol-level connection is forged and the initialization
1360 handshake takes place. This facility can be used to provide
1361 authentication credentials, as in host-strings such as:
1362 <literal>user=admin,password=halfAm4n,tcp:localhost:8017/db</literal>
1365 Connection objects should be destroyed using the function
1366 <function>ZOOM_connection_destroy</function>.
1369 void ZOOM_connection_option_set(ZOOM_connection c,
1370 const char *key, const char *val);
1372 void ZOOM_connection_option_setl(ZOOM_connection c,
1374 const char *val, int len);
1376 const char *ZOOM_connection_option_get(ZOOM_connection c,
1378 const char *ZOOM_connection_option_getl(ZOOM_connection c,
1383 The functions <function>ZOOM_connection_option_set</function> and
1384 <function>ZOOM_connection_option_setl</function> allows you to
1385 set an option given by <parameter>key</parameter> to the value
1386 <parameter>value</parameter> for the connection.
1387 For <function>ZOOM_connection_option_set</function>, the
1388 value is assumed to be a 0-terminated string. Function
1389 <function>ZOOM_connection_option_setl</function> specifies a
1390 value of a certain size (len).
1393 Functions <function>ZOOM_connection_option_get</function> and
1394 <function>ZOOM_connection_option_getl</function> returns
1395 the value for an option given by <parameter>key</parameter>.
1397 <table id="zoom-connection-options" frame="top">
1398 <title>ZOOM Connection Options</title>
1400 <colspec colwidth="4*" colname="name"></colspec>
1401 <colspec colwidth="7*" colname="description"></colspec>
1402 <colspec colwidth="3*" colname="default"></colspec>
1405 <entry>Option</entry>
1406 <entry>Description</entry>
1407 <entry>Default</entry>
1412 implementationName</entry><entry>Name of Your client
1413 </entry><entry>none</entry></row>
1415 user</entry><entry>Authentication user name
1416 </entry><entry>none</entry></row>
1418 group</entry><entry>Authentication group name
1419 </entry><entry>none</entry></row>
1421 password</entry><entry>Authentication password.
1422 </entry><entry>none</entry></row>
1424 authenticationMode</entry><entry>How authentication is encoded.
1425 </entry><entry>basic</entry></row>
1427 host</entry><entry>Target host. This setting is "read-only".
1428 It's automatically set internally when connecting to a target.
1429 </entry><entry>none</entry></row>
1431 proxy</entry><entry>Proxy host. If set, the logical host
1432 is encoded in the otherInfo area of the Z39.50 Init PDU
1433 with OID 1.2.840.10003.10.1000.81.1.
1434 </entry><entry>none</entry></row>
1436 clientIP</entry><entry>Client IP. If set, is
1437 encoded in the otherInfo area of a Z39.50 PDU with OID
1438 1.2.840.10003.10.1000.81.3. Holds the original IP addreses
1439 of a client. Is used if ZOOM is used in a gateway of some sort.
1440 </entry><entry>none</entry></row>
1442 async</entry><entry>If true (1) the connection operates in
1443 asynchronous operation which means that all calls are non-blocking
1445 <link linkend="zoom.events"><function>ZOOM_event</function></link>.
1446 </entry><entry>0</entry></row>
1448 maximumRecordSize</entry><entry> Maximum size of single record.
1449 </entry><entry>1 MB</entry></row>
1451 preferredMessageSize</entry><entry> Maximum size of multiple records.
1452 </entry><entry>1 MB</entry></row>
1454 lang</entry><entry> Language for negotiation.
1455 </entry><entry>none</entry></row>
1457 charset</entry><entry> Character set for negotiation.
1458 </entry><entry>none</entry></row>
1460 serverImplementationId</entry><entry>
1461 Implementation ID of server. (The old targetImplementationId
1462 option is also supported for the benefit of old applications.)
1463 </entry><entry>none</entry></row>
1465 targetImplementationName</entry><entry>
1466 Implementation Name of server. (The old
1467 targetImplementationName option is also supported for the
1468 benefit of old applications.)
1469 </entry><entry>none</entry></row>
1471 serverImplementationVersion</entry><entry>
1472 Implementation Version of server. (the old
1473 targetImplementationVersion option is also supported for the
1474 benefit of old applications.)
1475 </entry><entry>none</entry></row>
1477 databaseName</entry><entry>One or more database names
1478 separated by character plus (<literal>+</literal>), which is to
1479 be used by subsequent search requests on this Connection.
1480 </entry><entry>Default</entry></row>
1482 piggyback</entry><entry>True (1) if piggyback should be
1483 used in searches; false (0) if not.
1484 </entry><entry>1</entry></row>
1486 smallSetUpperBound</entry><entry>If hits is less than or equal to this
1487 value, then target will return all records using small element set name
1488 </entry><entry>0</entry></row>
1490 largeSetLowerBound</entry><entry>If hits is greater than this
1491 value, the target will return no records.
1492 </entry><entry>1</entry></row>
1494 mediumSetPresentNumber</entry><entry>This value represents
1495 the number of records to be returned as part of a search when
1496 hits is less than or equal to large set lower bound and if hits
1497 is greater than small set upper bound.
1498 </entry><entry>0</entry></row>
1500 smallSetElementSetName</entry><entry>
1501 The element set name to be used for small result sets.
1502 </entry><entry>none</entry></row>
1504 mediumSetElementSetName</entry><entry>
1505 The element set name to be used for medium-sized result sets.
1506 </entry><entry>none</entry></row>
1508 init_opt_search, init_opt_present, init_opt_delSet, etc.</entry><entry>
1509 After a successful Init, these options may be interrogated to
1510 discover whether the server claims to support the specified
1512 </entry><entry>none</entry></row>
1514 <entry>sru</entry><entry>
1515 SRU/Solr transport type. Must be either <literal>soap</literal>,
1516 <literal>get</literal>, <literal>post</literal>, or
1517 <literal>solr</literal>.
1518 </entry><entry>soap</entry></row>
1520 sru_version</entry><entry>
1521 SRU/SRW version. Should be <literal>1.1</literal>, or
1522 <literal>1.2</literal>. This is, prior to connect, the version
1523 to offer (highest version). And following connect (in fact
1524 first operation), holds the negotiated version with the server
1525 (same or lower version).
1526 </entry><entry>1.2</entry></row>
1527 <row id="zoom.extraArgs.option"><entry>
1528 extraArgs</entry><entry>
1529 Extra arguments for SRU/Solr URLs. The value must be
1530 URL encoded already.
1531 </entry><entry></entry></row>
1532 <row id="zoom.facets.option"><entry>
1533 facets</entry><entry>
1534 Requested or recommended facets may be given before a search is sent.
1535 The value of this setting is described in <xref linkend="facets"/>
1536 For inspection of the facets returned, refer to the functions
1537 described in <xref linkend="zoom.facets"/>.
1538 </entry><entry>none</entry></row>
1540 apdulog</entry><entry>
1541 If set to a true value such as "1", a log of low-level
1542 protocol packets is emitted on standard error stream. This
1543 can be very useful for debugging.
1544 </entry><entry>0</entry></row>
1546 saveAPDU</entry><entry>
1547 If set to a true value such as "1", a log of low-level
1548 protocol packets is saved. The log can be retrieved by reading
1549 option APDU. Setting saveAPDU always has the side effect of
1550 resetting the currently saved log. This setting is
1551 <emphasis>write-only</emphasis>. If read, NULL will be returned.
1552 It is only recognized in
1553 <function>ZOOM_connection_option_set</function>.
1554 </entry><entry>0</entry></row>
1557 Returns the log of protocol packets. Will be empty if logging
1558 is not enabled (see saveAPDU above). This setting is
1559 <emphasis>read-only</emphasis>. It is only recognized if used
1560 in call to <function>ZOOM_connection_option_get</function> or
1561 <function>ZOOM_connection_option_getl</function>.
1562 </entry><entry></entry></row>
1564 memcached</entry><entry>
1565 If given and non-empty,
1566 <ulink url="&url.libmemcached;">libMemcached</ulink>
1567 will be configured for the connection.
1568 This option is inspected by ZOOM when a connection is established.
1569 If the <literal>memcached</literal> option is given
1570 and YAZ is compiled without libMemcached support, an internal
1571 diagnostic (10018) will be thrown.
1572 libMemcached support is available for YAZ 5.0.13 or later. If this
1573 option is supplied for an earlier version of YAZ, it is
1574 <emphasis>ignored</emphasis>.
1575 The value of this option is a list options - each is of the
1576 form <literal>--name=value</literal>.
1577 Option <literal>--server=</literal>host[:port] specifies a memcached
1578 server. It may be repeated for multiple memcached servers.
1579 Option <literal>--expire=</literal>seconds sets expiry time in seconds
1580 for how long result sets are to be cached.
1581 </entry><entry>none</entry></row>
1583 redis</entry><entry>
1584 If given and non-empty,
1585 a <ulink url="&url.redis;">redis</ulink> context will be created
1587 This option is inspected by ZOOM when a connection is established.
1588 If the <literal>redis</literal> option is given
1589 and YAZ is compiled without redis support, an internal
1590 diagnostic (10018) will be thrown.
1591 redis support is available for YAZ 5.2.0 or later. If this
1592 option is supplied for an earlier version of YAZ, it is
1593 <emphasis>ignored</emphasis>.
1594 The value of this option is a set of options, similar to that
1595 of the memcached setting. At this stage only --server=host[:port]
1596 and --expire=seconds are supported.
1597 </entry><entry>none</entry></row>
1602 If either option <literal>lang</literal> or <literal>charset</literal>
1604 <ulink url="&url.z39.50.charneg;">
1605 Character Set and Language Negotiation</ulink> is in effect.
1608 int ZOOM_connection_error(ZOOM_connection c, const char **cp,
1609 const char **addinfo);
1610 int ZOOM_connection_error_x(ZOOM_connection c, const char **cp,
1611 const char **addinfo, const char **dset);
1614 Function <function>ZOOM_connection_error</function> checks for
1615 errors for the last operation(s) performed. The function returns
1616 zero if no errors occurred; non-zero otherwise indicating the error.
1617 Pointers <parameter>cp</parameter> and <parameter>addinfo</parameter>
1618 holds messages for the error and additional-info if passed as
1619 non-<literal>NULL</literal>. Function
1620 <function>ZOOM_connection_error_x</function> is an extended version
1621 of <function>ZOOM_connection_error</function> that is capable of
1622 returning name of diagnostic set in <parameter>dset</parameter>.
1624 <sect2 id="zoom-connection-z39.50">
1625 <title>Z39.50 Protocol behavior</title>
1627 The calls <function>ZOOM_connection_new</function> and
1628 <function>ZOOM_connection_connect</function> establishes a TCP/IP
1629 connection and sends an Initialize Request to the target if
1630 possible. In addition, the calls wait for an Initialize Response
1631 from the target and the result is inspected (OK or rejected).
1634 If <literal>proxy</literal> is set then the client will establish
1635 a TCP/IP connection with the peer as specified by the
1636 <literal>proxy</literal> host and the hostname as part of the
1637 connect calls will be set as part of the Initialize Request.
1638 The proxy server will then "forward" the PDU's transparently
1639 to the target behind the proxy.
1642 For the authentication parameters, if option <literal>user</literal>
1643 is set and both options <literal>group</literal> and
1644 <literal>pass</literal> are unset, then Open style
1645 authentication is used (Version 2/3) in which case the username
1646 is usually followed by a slash, then by a password.
1647 If either <literal>group</literal>
1648 or <literal>pass</literal> is set then idPass authentication
1649 (Version 3 only) is used. If none of the options are set, no
1650 authentication parameters are set as part of the Initialize Request
1654 When option <literal>async</literal> is 1, it really means that
1655 all network operations are postponed (and queued) until the
1656 function <literal>ZOOM_event</literal> is invoked. When doing so
1657 it doesn't make sense to check for errors after
1658 <literal>ZOOM_connection_new</literal> is called since that
1659 operation "connecting - and init" is still incomplete and the
1660 API cannot tell the outcome (yet).
1663 <sect2 id="zoom.sru.init.behavior">
1664 <title>SRU/Solr Protocol behavior</title>
1666 The HTTP based protocols (SRU, SRW, Solr) do not feature an
1667 Inititialize Request, so the connection phase merely establishes a
1668 TCP/IP connection with the HTTP server.
1670 <para>Most of the ZOOM connection options do not
1671 affect SRU/Solr and they are ignored. However, future versions
1672 of &yaz; might honor <literal>implementationName</literal> and
1673 put that as part of User-Agent header for HTTP requests.
1676 The <literal>charset</literal> is used in the Content-Type header
1680 Setting <literal>authentcationMode</literal> specifies how
1681 authentication parameters are encoded for HTTP. The default is
1682 "<literal>basic</literal>" where <literal>user</literal> and
1683 <literal>password</literal> are encoded by using HTTP basic
1687 If <literal>authentcationMode</literal> is "<literal>url</literal>", then
1688 user and password are encoded in the URL by parameters
1689 <literal>x-username</literal> and <literal>x-password</literal> as
1690 given by the SRU standard.
1694 <sect1 id="zoom.query">
1695 <title>Queries</title>
1697 Query objects represents queries.
1700 ZOOM_query ZOOM_query_create(void);
1702 void ZOOM_query_destroy(ZOOM_query q);
1704 int ZOOM_query_prefix(ZOOM_query q, const char *str);
1706 int ZOOM_query_cql(ZOOM_query s, const char *str);
1708 int ZOOM_query_sortby(ZOOM_query q, const char *criteria);
1710 int ZOOM_query_sortby2(ZOOM_query q, const char *strategy,
1711 const char *criteria);
1714 Create query objects using <function>ZOOM_query_create</function>
1715 and destroy them by calling <function>ZOOM_query_destroy</function>.
1716 RPN-queries can be specified in <link linkend="PQF">PQF</link>
1717 notation by using the
1718 function <function>ZOOM_query_prefix</function>.
1719 The <function>ZOOM_query_cql</function> specifies a CQL
1720 query to be sent to the server/target.
1721 More query types will be added in future versions of &yaz;, such as
1722 <link linkend="CCL">CCL</link> to RPN-mapping, native CCL query,
1723 etc. In addition to a search, a sort criteria may be set. Function
1724 <function>ZOOM_query_sortby</function> enables Z39.50 sorting and
1725 it takes sort criteria using the same string notation as
1726 yaz-client's <link linkend="sortspec">sort command</link>.
1728 <para id="zoom.query.sortby2">
1729 <function>ZOOM_query_sortby2</function> is similar to
1730 <function>ZOOM_query_sortby</function> but allows a strategy for
1731 sorting. The reason for the strategy parameter is that some
1732 protocols offer multiple ways of performing sorting.
1733 For example, Z39.50 has the standard sort, which is performed after
1734 search on an existing result set.
1735 It's also possible to use CQL in Z39.50 as the query type and use
1736 CQL's SORTBY keyword. Finally, Index Data's
1737 Zebra server also allows sorting to be specified as part of RPN (Type 7).
1739 <table id="zoom-sort-strategy" frame="top">
1740 <title>ZOOM sort strategy</title>
1742 <colspec colwidth="2*" colname="name"/>
1743 <colspec colwidth="5*" colname="description"/>
1747 <entry>Description</entry>
1752 <entry>z39.50</entry><entry>Z39.50 resultset sort</entry>
1755 <entry>type7</entry><entry>Sorting embedded in RPN(Type-7)</entry>
1758 <entry>cql</entry><entry>CQL SORTBY</entry>
1761 <entry>sru11</entry><entry>SRU sortKeys parameter</entry>
1764 <entry>solr</entry><entry>Solr sort</entry>
1767 <entry>embed</entry><entry>type7 for Z39.50, cql for SRU,
1768 solr for Solr protocol</entry>
1774 <sect1 id="zoom.resultsets"><title>Result sets</title>
1776 The result set object is a container for records returned from
1780 ZOOM_resultset ZOOM_connection_search(ZOOM_connection, ZOOM_query q);
1782 ZOOM_resultset ZOOM_connection_search_pqf(ZOOM_connection c,
1784 void ZOOM_resultset_destroy(ZOOM_resultset r);
1787 Function <function>ZOOM_connection_search</function> creates
1788 a result set, given a connection and query.
1789 Destroy a result set by calling
1790 <function>ZOOM_resultset_destroy</function>.
1791 Simple clients using PQF only, may use the function
1792 <function>ZOOM_connection_search_pqf</function> in which case
1793 creating query objects is not necessary.
1796 void ZOOM_resultset_option_set(ZOOM_resultset r,
1797 const char *key, const char *val);
1799 const char *ZOOM_resultset_option_get(ZOOM_resultset r, const char *key);
1801 size_t ZOOM_resultset_size(ZOOM_resultset r);
1804 Functions <function>ZOOM_resultset_options_set</function> and
1805 <function>ZOOM_resultset_get</function> sets and gets an option
1806 for a result set similar to <function>ZOOM_connection_option_get</function>
1807 and <function>ZOOM_connection_option_set</function>.
1810 The number of hits, also called result-count, is returned by
1811 function <function>ZOOM_resultset_size</function>.
1813 <table id="zoom.resultset.options"
1814 frame="top"><title>ZOOM Result set Options</title>
1816 <colspec colwidth="4*" colname="name"></colspec>
1817 <colspec colwidth="7*" colname="description"></colspec>
1818 <colspec colwidth="2*" colname="default"></colspec>
1821 <entry>Option</entry>
1822 <entry>Description</entry>
1823 <entry>Default</entry>
1828 start</entry><entry>Offset of first record to be
1829 retrieved from target. First record has offset 0 unlike the
1830 protocol specifications where first record has position 1.
1831 This option affects ZOOM_resultset_search and
1832 ZOOM_resultset_search_pqf and must be set before any of
1833 these functions are invoked. If a range of
1834 records must be fetched manually after search,
1835 function ZOOM_resultset_records should be used.
1836 </entry><entry>0</entry></row>
1838 count</entry><entry>Number of records to be retrieved.
1839 This option affects ZOOM_resultset_search and
1840 ZOOM_resultset_search_pqf and must be set before any of
1841 these functions are invoked.
1842 </entry><entry>0</entry></row>
1844 presentChunk</entry><entry>The number of records to be
1845 requested from the server in each chunk (present request). The
1846 value 0 means to request all the records in a single chunk.
1847 (The old <literal>step</literal>
1848 option is also supported for the benefit of old applications.)
1849 </entry><entry>0</entry></row>
1851 elementSetName</entry><entry>Element-Set name of records.
1852 Most targets should honor element set name <literal>B</literal>
1853 and <literal>F</literal> for brief and full respectively.
1854 </entry><entry>none</entry></row>
1856 preferredRecordSyntax</entry><entry>Preferred Syntax, such as
1857 <literal>USMARC</literal>, <literal>SUTRS</literal>, etc.
1858 </entry><entry>none</entry></row>
1860 schema</entry><entry>Schema for retrieval, such as
1861 <literal>Gils-schema</literal>, <literal>Geo-schema</literal>, etc.
1862 </entry><entry>none</entry></row>
1864 setname</entry><entry>Name of Result Set (Result Set ID).
1865 If this option isn't set, the ZOOM module will automatically
1866 allocate a result set name.
1867 </entry><entry>default</entry></row>
1869 rpnCharset</entry><entry>Character set for RPN terms.
1870 If this is set, ZOOM C will assume that the ZOOM application is
1871 running UTF-8. Terms in RPN queries are then converted to the
1872 rpnCharset. If this is unset, ZOOM C will not assume any encoding
1873 of RPN terms and no conversion is performed.
1874 </entry><entry>none</entry></row>
1879 For servers that support Search Info report, the following
1880 options may be read using <function>ZOOM_resultset_get</function>.
1881 This detailed information is read after a successful search has
1885 This information is a list of of items, where each item is
1886 information about a term or subquery. All items in the list
1888 <literal>SearchResult.</literal><replaceable>no</replaceable>
1889 where no presents the item number (0=first, 1=second).
1890 Read <literal>searchresult.size</literal> to determine the
1893 <table id="zoom.search.info.report.options"
1894 frame="top"><title>Search Info Report Options</title>
1896 <colspec colwidth="4*" colname="name"></colspec>
1897 <colspec colwidth="7*" colname="description"></colspec>
1900 <entry>Option</entry>
1901 <entry>Description</entry>
1906 <entry>searchresult.size</entry>
1908 number of search result entries. This option is non-existent
1909 if no entries are returned by the server.
1913 <entry>searchresult.<replaceable>no</replaceable>.id</entry>
1914 <entry>sub query ID</entry>
1917 <entry>searchresult.<replaceable>no</replaceable>.count</entry>
1918 <entry>result count for item (number of hits)</entry>
1921 <entry>searchresult.<replaceable>no</replaceable>.subquery.term</entry>
1922 <entry>subquery term</entry>
1926 searchresult.<replaceable>no</replaceable>.interpretation.term
1928 <entry>interpretation term</entry>
1932 searchresult.<replaceable>no</replaceable>.recommendation.term
1934 <entry>recommendation term</entry>
1939 <sect2 id="zoom.z3950.resultset.sort">
1940 <title>Z39.50 Result-set Sort</title>
1942 void ZOOM_resultset_sort(ZOOM_resultset r,
1943 const char *sort_type, const char *sort_spec);
1945 int ZOOM_resultset_sort1(ZOOM_resultset r,
1946 const char *sort_type, const char *sort_spec);
1949 <function>ZOOM_resultset_sort</function> and
1950 <function>ZOOM_resultset_sort1</function> both sort an existing
1951 result-set. The sort_type parameter is not used. Set it to "yaz".
1952 The sort_spec is same notation as ZOOM_query_sortby and identical
1953 to that offered by yaz-client's
1954 <link linkend="sortspec">sort command</link>.
1957 These functions only work for Z39.50. Use the more generic utility
1958 <link linkend="zoom.query.sortby2">
1959 <function>ZOOM_query_sortby2</function></link>
1960 for other protocols (and even Z39.50).
1963 <sect2 id="zoom.z3950.resultset.behavior">
1964 <title>Z39.50 Protocol behavior</title>
1966 The creation of a result set involves at least a SearchRequest
1967 - SearchResponse protocol handshake. Following that, if a sort
1968 criteria was specified as part of the query, a SortRequest -
1969 SortResponse handshake takes place. Note that it is necessary to
1970 perform sorting before any retrieval takes place, so no records will
1971 be returned from the target as part of the SearchResponse because these
1972 would be unsorted. Hence, piggyback is disabled when sort criteria
1973 are set. Following Search - and a possible sort - Retrieval takes
1974 place - as one or more Present Requests/Response pairs being
1978 The API allows for two different modes for retrieval. A high level
1979 mode which is somewhat more powerful and a low level one.
1980 The low level is enabled when searching on a Connection object
1981 for which the settings
1982 <literal>smallSetUpperBound</literal>,
1983 <literal>mediumSetPresentNumber</literal> and
1984 <literal>largeSetLowerBound</literal> are set. The low level mode
1985 thus allows you to precisely set how records are returned as part
1986 of a search response as offered by the Z39.50 protocol.
1987 Since the client may be retrieving records as part of the
1988 search response, this mode doesn't work well if sorting is used.
1991 The high-level mode allows you to fetch a range of records from
1992 the result set with a given start offset. When you use this mode
1993 the client will automatically use piggyback if that is possible
1994 with the target, and perform one or more present requests as needed.
1995 Even if the target returns fewer records as part of a present response
1996 because of a record size limit, etc. the client will repeat sending
1997 present requests. As an example, if option <literal>start</literal>
1998 is 0 (default) and <literal>count</literal> is 4, and
1999 <literal>piggyback</literal> is 1 (default) and no sorting criteria
2000 is specified, then the client will attempt to retrieve the 4
2001 records as part the search response (using piggyback). On the other
2002 hand, if either <literal>start</literal> is positive or if
2003 a sorting criteria is set, or if <literal>piggyback</literal>
2004 is 0, then the client will not perform piggyback but send Present
2008 If either of the options <literal>mediumSetElementSetName</literal> and
2009 <literal>smallSetElementSetName</literal> are unset, the value
2010 of option <literal>elementSetName</literal> is used for piggyback
2011 searches. This means that for the high-level mode you only have
2012 to specify one elementSetName option rather than three.
2015 <sect2 id="zoom.sru.resultset.behavior">
2016 <title>SRU Protocol behavior</title>
2018 Current version of &yaz; does not take advantage of a result set id
2019 returned by the SRU server. Future versions might do, however.
2020 Since the ZOOM driver does not save result set IDs, any
2021 present (retrieval) is transformed to a SRU SearchRetrieveRequest
2022 with same query but, possibly, different offsets.
2025 Option <literal>schema</literal> specifies SRU schema
2026 for retrieval. However, options <literal>elementSetName</literal> and
2027 <literal>preferredRecordSyntax</literal> are ignored.
2030 Options <literal>start</literal> and <literal>count</literal>
2031 are supported by SRU.
2032 The remaining options
2033 <literal>piggyback</literal>,
2034 <literal>smallSetUpperBound</literal>,
2035 <literal>largeSetLowerBound</literal>,
2036 <literal>mediumSetPresentNumber</literal>,
2037 <literal>mediumSetElementSetName</literal>,
2038 <literal>smallSetElementSetName</literal> are
2042 SRU supports CQL queries, <emphasis>not</emphasis> PQF.
2043 If PQF is used, however, the PQF query is transferred anyway
2044 using non-standard element <literal>pQuery</literal> in
2045 SRU SearchRetrieveRequest.
2048 Solr queries need to be done in Solr query format.
2051 Unfortunately, SRU and Solr do not define a database setting. Hence,
2052 <literal>databaseName</literal> is unsupported and ignored.
2053 However, the path part in host parameter for functions
2054 <function>ZOOM_connecton_new</function> and
2055 <function>ZOOM_connection_connect</function> acts as a
2056 database (at least for the &yaz; SRU server).
2060 <sect1 id="zoom.records">
2061 <title>Records</title>
2063 A record object is a retrieval record on the client side -
2064 created from result sets.
2067 void ZOOM_resultset_records(ZOOM_resultset r,
2069 size_t start, size_t count);
2070 ZOOM_record ZOOM_resultset_record(ZOOM_resultset s, size_t pos);
2072 const char *ZOOM_record_get(ZOOM_record rec, const char *type,
2075 int ZOOM_record_error(ZOOM_record rec, const char **msg,
2076 const char **addinfo, const char **diagset);
2078 ZOOM_record ZOOM_record_clone(ZOOM_record rec);
2080 void ZOOM_record_destroy(ZOOM_record rec);
2083 References to temporary records are returned by functions
2084 <function>ZOOM_resultset_records</function> or
2085 <function>ZOOM_resultset_record</function>.
2088 If a persistent reference to a record is desired
2089 <function>ZOOM_record_clone</function> should be used.
2090 It returns a record reference that should be destroyed
2091 by a call to <function>ZOOM_record_destroy</function>.
2094 A single record is returned by function
2095 <function>ZOOM_resultset_record</function> that takes a
2096 position as argument. First record has position zero.
2097 If no record could be obtained <literal>NULL</literal> is returned.
2100 Error information for a record can be checked with
2101 <function>ZOOM_record_error</function> which returns non-zero
2102 (error code) if record is in error, called <emphasis>Surrogate
2103 Diagnostics</emphasis> in Z39.50.
2106 Function <function>ZOOM_resultset_records</function> retrieves
2107 a number of records from a result set. Parameter <literal>start</literal>
2108 and <literal>count</literal> specifies the range of records to
2109 be returned. Upon completion, the array
2110 <literal>recs[0], ..recs[count-1]</literal>
2111 holds record objects for the records. The array of records
2112 <literal>recs</literal> should be allocated prior the call
2113 <function>ZOOM_resultset_records</function>. Note that for those
2114 records that couldn't be retrieved from the target,
2115 <literal>recs[ ..]</literal> is set to <literal>NULL</literal>.
2117 <para id="zoom.record.get">
2118 In order to extract information about a single record,
2119 <function>ZOOM_record_get</function> is provided. The
2120 function returns a pointer to certain record information. The
2121 nature (type) of the pointer depends on the parameter,
2122 <parameter>type</parameter>.
2125 The <parameter>type</parameter> is a string of the format:
2128 <replaceable>format</replaceable>[;charset=<replaceable>from</replaceable>[/<replaceable>opacfrom</replaceable>][,<replaceable>to</replaceable>]][;format=<replaceable>v</replaceable>][;base64=<replaceable>xpath</replaceable>]
2131 If <literal>charset</literal> is given, then <replaceable>from</replaceable>
2132 specifies the character set of the record in its original form
2133 (as returned by the server), <replaceable>to</replaceable> specifies
2134 the output (returned) character set encoding.
2135 If <replaceable>to</replaceable> is omitted, then UTF-8 is assumed.
2136 If charset is not given, then no character set conversion takes place.
2137 OPAC records may be returned in a different
2138 set from the bibliographic MARC record. If this is this the case,
2139 <replaceable>opacfrom</replaceable> should be set to the character set
2140 of the OPAC record part.
2144 The <literal>format</literal> is generic but can only be used to
2145 specify XML indentation when the value <replaceable>v</replaceable>
2146 is 1 (<literal>format=1</literal>).
2149 The <literal>base64</literal> allows a full record to be extracted
2150 from base64-encoded string in an XML document.
2154 Specifying the OPAC record character set requires YAZ 4.1.5 or later.
2157 Specifying the base64 parameter requires YAZ 4.2.35 or later.
2161 The format argument controls whether record data should be XML
2162 pretty-printed (post process operation).
2163 It is enabled only if format value <replaceable>v</replaceable> is
2164 <literal>1</literal> and the record content is XML well-formed.
2167 In addition, for certain types, the length
2168 <literal>len</literal> passed will be set to the size in bytes of
2169 the returned information.
2172 The following are the supported values for <replaceable>form</replaceable>.
2174 <varlistentry><term><literal>database</literal></term>
2175 <listitem><para>The Database of the record is returned
2176 as a C null-terminated string. Return type
2177 <literal>const char *</literal>.
2180 <varlistentry><term><literal>syntax</literal></term>
2181 <listitem><para>The transfer syntax of the record is returned
2182 as a C null-terminated string containing the symbolic name of
2183 the record syntax, e.g. <literal>Usmarc</literal>. Return type
2185 <literal>const char *</literal>.
2188 <varlistentry><term><literal>schema</literal></term>
2189 <listitem><para>The schema of the record is returned
2190 as a C null-terminated string. Return type is
2191 <literal>const char *</literal>.
2194 <varlistentry><term><literal>render</literal></term>
2195 <listitem><para>The record is returned in a display friendly
2196 format. Upon completion, buffer is returned
2197 (type <literal>const char *</literal>) and length is stored in
2198 <literal>*len</literal>.
2201 <varlistentry><term><literal>raw</literal></term>
2202 <listitem><para>The record is returned in the internal
2203 YAZ specific format. For GRS-1, Explain, and others, the
2204 raw data is returned as type
2205 <literal>Z_External *</literal> which is just the type for
2206 the member <literal>retrievalRecord</literal> in
2207 type <literal>NamePlusRecord</literal>.
2208 For SUTRS and octet aligned record (including all MARCs) the
2209 octet buffer is returned and the length of the buffer.
2212 <varlistentry><term><literal>xml</literal></term>
2213 <listitem><para>The record is returned in XML if possible.
2214 SRU, Solr and Z39.50 records with transfer syntax XML are
2215 returned verbatim. MARC records are returned in
2216 <ulink url="&url.marcxml;">
2219 (converted from ISO2709 to MARCXML by YAZ).
2220 OPAC records are also converted to XML and the
2221 bibliographic record is converted to MARCXML (when possible).
2222 GRS-1 records are not supported for this form.
2223 Upon completion, the XML buffer is returned
2224 (type <literal>const char *</literal>) and length is stored in
2225 <literal>*len</literal>.
2228 <varlistentry><term><literal>opac</literal></term>
2229 <listitem><para>OPAC information for record is returned in XML
2230 if an OPAC record is present at the position given. If no
2231 OPAC record is present, a NULL pointer is returned.
2234 <varlistentry><term><literal>txml</literal></term>
2235 <listitem><para>The record is returned in TurboMARC if possible.
2236 SRU and Z39.50 records with transfer syntax XML are
2237 returned verbatim. MARC records are returned in
2238 <link linkend="tools.turbomarc">
2241 (converted from ISO2709 to TurboMARC by YAZ).
2242 Upon completion, the XML buffer is returned
2243 (type <literal>const char *</literal>) and length is stored in
2244 <literal>*len</literal>.
2247 <varlistentry><term><literal>json</literal></term>
2248 <listitem><para>Like xml, but MARC records are converted to
2249 <ulink url="&url.marc_in_json;">MARC-in-JSON</ulink>.
2257 <ulink url="&url.marc21;">MARC21</ulink>
2259 <ulink url="&url.marc8;">MARC-8</ulink>
2260 character set encoding.
2261 An application that wishes to display in Latin-1 would use
2263 render; charset=marc8,iso-8859-1
2266 <sect2 id="zoom.z3950.record.behavior">
2267 <title>Z39.50 Protocol behavior</title>
2269 The functions <function>ZOOM_resultset_record</function> and
2270 <function>ZOOM_resultset_records</function> inspects the client-side
2271 record cache. Records not found in cache are fetched using
2273 The functions may block (and perform network I/O) - even though option
2274 <literal>async</literal> is 1, because they return records objects.
2275 (And there's no way to return records objects without retrieving them!)
2278 There is a trick, however, in the usage of function
2279 <function>ZOOM_resultset_records</function> that allows for
2280 delayed retrieval (and makes it non-blocking). By using
2281 a null pointer for <parameter>recs</parameter> you're indicating
2282 you're not interested in getting records objects
2283 <emphasis>now</emphasis>.
2286 <sect2 id="zoom.sru.record.behavior">
2287 <title>SRU/Solr Protocol behavior</title>
2289 The ZOOM driver for SRU/Solr treats records returned by a SRU/Solr server
2290 as if they where Z39.50 records with transfer syntax XML and
2291 no element set name or database name.
2295 <sect1 id="zoom.facets"><title>Facets</title>
2297 Facet operations is not part of the official ZOOM specification, but
2298 is an Index Data extension for YAZ-based Z39.50 targets,
2299 <ulink url="&url.solr;">Solr</ulink> and SRU 2.0 targets.
2301 Facets may be requestd by the
2302 <link linkend="zoom.facets.option">facets</link> option before a
2304 For inspection of the returned facets, the following functions are
2308 ZOOM_facet_field *ZOOM_resultset_facets(ZOOM_resultset r);
2310 ZOOM_facet_field ZOOM_resultset_get_facet_field(ZOOM_resultset r,
2311 const char *facet_name);
2313 ZOOM_facet_field ZOOM_resultset_get_facet_field_by_index(ZOOM_resultset r,
2316 size_t ZOOM_resultset_facets_size(ZOOM_resultset r);
2318 const char *ZOOM_facet_field_name(ZOOM_facet_field facet_field);
2320 size_t ZOOM_facet_field_term_count(ZOOM_facet_field facet_field);
2322 const char *ZOOM_facet_field_get_term(ZOOM_facet_field facet_field,
2323 size_t idx, int *freq);
2326 References to temporary structures are returned by all functions.
2327 They are only valid as long the Result set is valid.
2328 <function>ZOOM_resultset_get_facet_field</function> or
2329 <function>ZOOM_resultset_get_facet_field_by_index</function>.
2330 <function>ZOOM_resultset_facets</function>.
2331 <function>ZOOM_facet_field_name</function>.
2332 <function>ZOOM_facet_field_get_term</function>.
2334 <para id="zoom.resultset.get_facet_field">
2335 A single Facet field is returned by function
2336 <function>ZOOM_resultset_get_facet_field</function> or
2337 <function>ZOOM_resultset_get_facet_field_by_index</function> that takes
2338 a result set and facet name or positive index respectively. First
2339 facet has position zero. If no facet could be obtained (invalid name
2340 or index out of bounds) <literal>NULL</literal> is returned.
2342 <para id="zoom.resultset.facets">
2343 An array of facets field can be returned by
2344 <function>ZOOM_resultset_facets</function>. The length of the array is
2345 given by <function>ZOOM_resultset_facets_size</function>. The array is
2346 zero-based and the last entry will be at
2347 <function>ZOOM_resultset_facets_size(result_set)</function>-1.
2349 <para id="zoom.resultset.facets_names">
2350 It is possible to interate over facets by name, by calling
2351 <function>ZOOM_resultset_facets_names</function>.
2352 This will return a const array of char * where each string can be used
2353 as parameter for <function>ZOOM_resultset_get_facet_field</function>.
2356 Function <function>ZOOM_facet_field_name</function> gets the request
2357 facet name from a returned facet field.
2360 Function <function>ZOOM_facet_field_get_term</function> returns the
2361 idx'th term and term count for a facet field.
2362 Idx must between 0 and
2363 <function>ZOOM_facet_field_term_count</function>-1, otherwise the
2364 returned reference will be <literal>NULL</literal>. On a valid idx, the
2365 value of the freq reference will be the term count.
2366 The <literal>freq</literal> parameter must be valid pointer to integer.
2369 <sect1 id="zoom.scan"><title>Scan</title>
2371 This section describes an interface for Scan. Scan is not an
2372 official part of the ZOOM model yet. The result of a scan operation
2373 is the <literal>ZOOM_scanset</literal> which is a set of terms
2374 returned by a target.
2378 The Scan interface is supported for both Z39.50, SRU and Solr.
2382 ZOOM_scanset ZOOM_connection_scan(ZOOM_connection c,
2383 const char *startpqf);
2385 ZOOM_scanset ZOOM_connection_scan1(ZOOM_connection c,
2388 size_t ZOOM_scanset_size(ZOOM_scanset scan);
2390 const char *ZOOM_scanset_term(ZOOM_scanset scan, size_t pos,
2391 size_t *occ, size_t *len);
2393 const char *ZOOM_scanset_display_term(ZOOM_scanset scan, size_t pos,
2394 size_t *occ, size_t *len);
2396 void ZOOM_scanset_destroy(ZOOM_scanset scan);
2398 const char *ZOOM_scanset_option_get(ZOOM_scanset scan,
2401 void ZOOM_scanset_option_set(ZOOM_scanset scan, const char *key,
2405 The scan set is created by function
2406 <function>ZOOM_connection_scan</function> which performs a scan
2407 operation on the connection using the specified
2408 <parameter>startpqf</parameter>.
2409 If the operation was successful, the size of the scan set can be
2410 retrieved by a call to <function>ZOOM_scanset_size</function>.
2411 Like result sets, the items are numbered 0..size-1.
2412 To obtain information about a particular scan term, call function
2413 <function>ZOOM_scanset_term</function>. This function takes
2414 a scan set offset <literal>pos</literal> and returns a pointer
2415 to a <emphasis>raw term</emphasis> or <literal>NULL</literal> if
2417 If present, the <literal>occ</literal> and <literal>len</literal>
2418 are set to the number of occurrences and the length
2419 of the actual term respectively.
2420 <function>ZOOM_scanset_display_term</function> is similar to
2421 <function>ZOOM_scanset_term</function> except that it returns
2422 the <emphasis>display term</emphasis> rather than the raw term.
2423 In a few cases, the term is different from display term. Always
2424 use the display term for display and the raw term for subsequent
2425 scan operations (to get more terms, next scan result, etc).
2428 A scan set may be freed by a call to function
2429 <function>ZOOM_scanset_destroy</function>.
2430 Functions <function>ZOOM_scanset_option_get</function> and
2431 <function>ZOOM_scanset_option_set</function> retrieves and sets
2432 an option respectively.
2435 The <parameter>startpqf</parameter> is a subset of PQF, namely
2436 the Attributes+Term part. Multiple <literal>@attr</literal> can
2437 be used. For example to scan in title (complete) phrases:
2439 @attr 1=4 @attr 6=2 "science o"
2443 The <function>ZOOM_connecton_scan1</function> is a newer and
2444 more generic alternative to <function>ZOOM_connection_scan</function>
2445 which allows to use both CQL and PQF for Scan.
2447 <table frame="top" id="zoom.scanset.options">
2448 <title>ZOOM Scan Set Options</title>
2450 <colspec colwidth="4*" colname="name"></colspec>
2451 <colspec colwidth="7*" colname="description"></colspec>
2452 <colspec colwidth="2*" colname="default"></colspec>
2455 <entry>Option</entry>
2456 <entry>Description</entry>
2457 <entry>Default</entry>
2462 number</entry><entry>Number of Scan Terms requested in next scan.
2463 After scan it holds the actual number of terms returned.
2464 </entry><entry>20</entry></row>
2466 position</entry><entry>Preferred Position of term in response
2467 in next scan; actual position after completion of scan.
2468 </entry><entry>1</entry></row>
2470 stepSize</entry><entry>Step Size
2471 </entry><entry>0</entry></row>
2473 scanStatus</entry><entry>An integer indicating the Scan Status
2475 </entry><entry>0</entry></row>
2477 rpnCharset</entry><entry>Character set for RPN terms.
2478 If this is set, ZOOM C will assume that the ZOOM application is
2479 running UTF-8. Terms in RPN queries are then converted to the
2480 rpnCharset. If this is unset, ZOOM C will not assume any encoding
2481 of RPN terms and no conversion is performed.
2482 </entry><entry>none</entry></row>
2487 <sect1 id="zoom.extendedservices">
2488 <title>Extended Services</title>
2490 ZOOM offers an interface to a subset of the Z39.50 extended services
2491 as well as a few privately defined ones:
2496 Z39.50 Item Order (ILL).
2497 See <xref linkend="zoom.item.order"/>.
2502 Record Update. This allows a client to insert, modify or delete
2504 See <xref linkend="zoom.record.update"/>.
2509 Database Create. This a non-standard feature. Allows a client
2510 to create a database.
2511 See <xref linkend="zoom.database.create"/>.
2516 Database Drop. This a non-standard feature. Allows a client
2517 to delete/drop a database.
2518 See <xref linkend="zoom.database.drop"/>.
2523 Commit operation. This a non-standard feature. Allows a client
2524 to commit operations.
2525 See <xref linkend="zoom.commit"/>.
2528 <!-- all the ILL PDU options should go here too -->
2531 To create an extended service operation, a <literal>ZOOM_package</literal>
2532 must be created. The operation is a five step operation. The
2533 package is created, package is configured by means of options,
2534 the package is sent, result is inspected (by means of options),
2535 the package is destroyed.
2538 ZOOM_package ZOOM_connection_package(ZOOM_connection c,
2539 ZOOM_options options);
2541 const char *ZOOM_package_option_get(ZOOM_package p,
2543 void ZOOM_package_option_set(ZOOM_package p, const char *key,
2545 void ZOOM_package_send(ZOOM_package p, const char *type);
2547 void ZOOM_package_destroy(ZOOM_package p);
2550 The <function>ZOOM_connection_package</function> creates a
2551 package for the connection given using the options specified.
2554 Functions <function>ZOOM_package_option_get</function> and
2555 <function>ZOOM_package_option_set</function> gets and sets
2559 <function>ZOOM_package_send</function> sends
2560 the package the via connection specified in
2561 <function>ZOOM_connection_package</function>.
2562 The <parameter>type</parameter> specifies the actual extended service
2563 package type to be sent.
2565 <table frame="top" id="zoom.extendedservices.type">
2566 <title>Extended Service Type</title>
2568 <colspec colwidth="3*" colname="value"></colspec>
2569 <colspec colwidth="7*" colname="description"></colspec>
2573 <entry>Description</entry>
2578 <entry>itemorder</entry><entry>Item Order</entry>
2581 <entry>update</entry><entry>Record Update</entry>
2584 <entry>create</entry><entry>Database Create</entry>
2587 <entry>drop</entry><entry>Database Drop</entry>
2590 <entry>commit</entry><entry>Commit Operation</entry>
2596 <table frame="top" id="zoom.extendedservices.options">
2597 <title>Extended Service Common Options</title>
2599 <colspec colwidth="4*" colname="name"></colspec>
2600 <colspec colwidth="7*" colname="description"></colspec>
2601 <colspec colwidth="3*" colname="default"></colspec>
2604 <entry>Option</entry>
2605 <entry>Description</entry>
2606 <entry>Default</entry>
2611 <entry>package-name</entry>
2612 <entry>Extended Service Request package name. Must be specified
2613 as part of a request.</entry>
2617 <entry>user-id</entry>
2618 <entry>User ID of Extended Service Package. Is a request option.</entry>
2622 <entry>function</entry>
2624 Function of package - one of <literal>create</literal>,
2625 <literal>delete</literal>, <literal>modify</literal>. Is
2628 <entry><literal>create</literal></entry>
2631 <entry>waitAction</entry>
2633 Wait action for package. Possible values:
2634 <literal>wait</literal>, <literal>waitIfPossible</literal>,
2635 <literal>dontWait</literal> or <literal>dontReturnPackage</literal>.
2637 <entry><literal>waitIfPossible</literal></entry>
2640 <entry>targetReference</entry>
2642 Target Reference. This is part of the response as returned
2643 by the server. Read it after a successful operation.
2645 <entry><literal>none</literal></entry>
2650 <sect2 id="zoom.item.order">
2651 <title>Item Order</title>
2653 For Item Order, <literal>type</literal> must be set to
2654 <literal>itemorder</literal> in
2655 <function>ZOOM_package_send</function>.
2658 <table frame="top" id="zoom.item.order.options">
2659 <title>Item Order Options</title>
2661 <colspec colwidth="4*" colname="name"></colspec>
2662 <colspec colwidth="7*" colname="description"></colspec>
2663 <colspec colwidth="3*" colname="default"></colspec>
2666 <entry>Option</entry>
2667 <entry>Description</entry>
2668 <entry>Default</entry>
2673 <entry>contact-name</entry>
2674 <entry>ILL contact name</entry>
2678 <entry>contact-phone</entry>
2679 <entry>ILL contact phone</entry>
2683 <entry>contact-email</entry>
2684 <entry>ILL contact email</entry>
2688 <entry>itemorder-setname</entry>
2689 <entry>Name of result set for record</entry>
2690 <entry>default</entry>
2693 <entry>itemorder-item</entry>
2694 <entry>Position for item (record) requested. An integer</entry>
2701 There are two variants of item order: ILL-variant and
2702 XML document variant. In order to use the XML variant the setting
2703 <literal>doc</literal> must hold the XML item order document. If that
2704 setting is unset, the ILL-variant is used.
2707 <table frame="top" id="zoom.illrequest.options">
2708 <title>ILL Request Options</title>
2710 <colspec colwidth="4*" colname="name"></colspec>
2713 <entry>Option</entry>
2717 <row><entry>protocol-version-num</entry></row>
2718 <row><entry>transaction-id,initial-requester-id,person-or-institution-symbol,person</entry></row>
2719 <row><entry>transaction-id,initial-requester-id,person-or-institution-symbol,institution</entry></row>
2720 <row><entry>transaction-id,initial-requester-id,name-of-person-or-institution,name-of-person</entry></row>
2721 <row><entry>transaction-id,initial-requester-id,name-of-person-or-institution,name-of-institution</entry></row>
2722 <row><entry>transaction-id,transaction-group-qualifier</entry></row>
2723 <row><entry>transaction-id,transaction-qualifier</entry></row>
2724 <row><entry>transaction-id,sub-transaction-qualifier</entry></row>
2725 <row><entry>service-date-time,this,date</entry></row>
2726 <row><entry>service-date-time,this,time</entry></row>
2727 <row><entry>service-date-time,original,date</entry></row>
2728 <row><entry>service-date-time,original,time</entry></row>
2729 <row><entry>requester-id,person-or-institution-symbol,person</entry></row>
2730 <row><entry>requester-id,person-or-institution-symbol,institution</entry></row>
2731 <row><entry>requester-id,name-of-person-or-institution,name-of-person</entry></row>
2732 <row><entry>requester-id,name-of-person-or-institution,name-of-institution</entry></row>
2733 <row><entry>responder-id,person-or-institution-symbol,person</entry></row>
2734 <row><entry>responder-id,person-or-institution-symbol,institution</entry></row>
2735 <row><entry>responder-id,name-of-person-or-institution,name-of-person</entry></row>
2736 <row><entry>responder-id,name-of-person-or-institution,name-of-institution</entry></row>
2737 <row><entry>transaction-type</entry></row>
2738 <row><entry>delivery-address,postal-address,name-of-person-or-institution,name-of-person</entry></row>
2739 <row><entry>delivery-address,postal-address,name-of-person-or-institution,name-of-institution</entry></row>
2740 <row><entry>delivery-address,postal-address,extended-postal-delivery-address</entry></row>
2741 <row><entry>delivery-address,postal-address,street-and-number</entry></row>
2742 <row><entry>delivery-address,postal-address,post-office-box</entry></row>
2743 <row><entry>delivery-address,postal-address,city</entry></row>
2744 <row><entry>delivery-address,postal-address,region</entry></row>
2745 <row><entry>delivery-address,postal-address,country</entry></row>
2746 <row><entry>delivery-address,postal-address,postal-code</entry></row>
2747 <row><entry>delivery-address,electronic-address,telecom-service-identifier</entry></row>
2748 <row><entry>delivery-address,electronic-address,telecom-service-addreess</entry></row>
2749 <row><entry>billing-address,postal-address,name-of-person-or-institution,name-of-person</entry></row>
2750 <row><entry>billing-address,postal-address,name-of-person-or-institution,name-of-institution</entry></row>
2751 <row><entry>billing-address,postal-address,extended-postal-delivery-address</entry></row>
2752 <row><entry>billing-address,postal-address,street-and-number</entry></row>
2753 <row><entry>billing-address,postal-address,post-office-box</entry></row>
2754 <row><entry>billing-address,postal-address,city</entry></row>
2755 <row><entry>billing-address,postal-address,region</entry></row>
2756 <row><entry>billing-address,postal-address,country</entry></row>
2757 <row><entry>billing-address,postal-address,postal-code</entry></row>
2758 <row><entry>billing-address,electronic-address,telecom-service-identifier</entry></row>
2759 <row><entry>billing-address,electronic-address,telecom-service-addreess</entry></row>
2760 <row><entry>ill-service-type</entry></row>
2761 <row><entry>requester-optional-messages,can-send-RECEIVED</entry></row>
2762 <row><entry>requester-optional-messages,can-send-RETURNED</entry></row>
2763 <row><entry>requester-optional-messages,requester-SHIPPED</entry></row>
2764 <row><entry>requester-optional-messages,requester-CHECKED-IN</entry></row>
2765 <row><entry>search-type,level-of-service</entry></row>
2766 <row><entry>search-type,need-before-date</entry></row>
2767 <row><entry>search-type,expiry-date</entry></row>
2768 <row><entry>search-type,expiry-flag</entry></row>
2769 <row><entry>place-on-hold</entry></row>
2770 <row><entry>client-id,client-name</entry></row>
2771 <row><entry>client-id,client-status</entry></row>
2772 <row><entry>client-id,client-identifier</entry></row>
2773 <row><entry>item-id,item-type</entry></row>
2774 <row><entry>item-id,call-number</entry></row>
2775 <row><entry>item-id,author</entry></row>
2776 <row><entry>item-id,title</entry></row>
2777 <row><entry>item-id,sub-title</entry></row>
2778 <row><entry>item-id,sponsoring-body</entry></row>
2779 <row><entry>item-id,place-of-publication</entry></row>
2780 <row><entry>item-id,publisher</entry></row>
2781 <row><entry>item-id,series-title-number</entry></row>
2782 <row><entry>item-id,volume-issue</entry></row>
2783 <row><entry>item-id,edition</entry></row>
2784 <row><entry>item-id,publication-date</entry></row>
2785 <row><entry>item-id,publication-date-of-component</entry></row>
2786 <row><entry>item-id,author-of-article</entry></row>
2787 <row><entry>item-id,title-of-article</entry></row>
2788 <row><entry>item-id,pagination</entry></row>
2789 <row><entry>item-id,ISBN</entry></row>
2790 <row><entry>item-id,ISSN</entry></row>
2791 <row><entry>item-id,additional-no-letters</entry></row>
2792 <row><entry>item-id,verification-reference-source</entry></row>
2793 <row><entry>copyright-complicance</entry></row>
2794 <row><entry>retry-flag</entry></row>
2795 <row><entry>forward-flag</entry></row>
2796 <row><entry>requester-note</entry></row>
2797 <row><entry>forward-note</entry></row>
2802 <sect2 id="zoom.record.update">
2803 <title>Record Update</title>
2805 For Record Update, <literal>type</literal> must be set to
2806 <literal>update</literal> in
2807 <function>ZOOM_package_send</function>.
2809 <table frame="top" id="zoom.record.update.options">
2810 <title>Record Update Options</title>
2812 <colspec colwidth="4*" colname="name"></colspec>
2813 <colspec colwidth="7*" colname="description"></colspec>
2814 <colspec colwidth="3*" colname="default"></colspec>
2817 <entry>Option</entry>
2818 <entry>Description</entry>
2819 <entry>Default</entry>
2824 <entry>action</entry>
2826 The update action. One of
2827 <literal>specialUpdate</literal>,
2828 <literal>recordInsert</literal>,
2829 <literal>recordReplace</literal>,
2830 <literal>recordDelete</literal>,
2831 <literal>elementUpdate</literal>.
2833 <entry><literal>specialUpdate (recordInsert for updateVersion=1 which does not support specialUpdate)</literal></entry>
2836 <entry>recordIdOpaque</entry>
2837 <entry>Opaque Record ID</entry>
2841 <entry>recordIdNumber</entry>
2842 <entry>Record ID number</entry>
2846 <entry>record</entry>
2847 <entry>The record itself</entry>
2851 <entry>recordOpaque</entry>
2852 <entry>Specifies an opaque record which is
2853 encoded as an ASN.1 ANY type with the OID as tiven by option
2854 <literal>syntax</literal> (see below).
2855 Option <literal>recordOpaque</literal> is an alternative
2856 to record - and <literal>record</literal> option (above) is
2857 ignored if recordOpaque is set. This option is only available in
2858 YAZ 3.0.35 and later, and is meant to facilitate Updates with
2864 <entry>syntax</entry>
2865 <entry>The record syntax (transfer syntax). Is a string that
2866 is a known record syntax.
2868 <entry>no syntax</entry>
2871 <entry>databaseName</entry>
2872 <entry>Database from connection object</entry>
2873 <entry>Default</entry>
2876 <entry>correlationInfo.note</entry>
2877 <entry>Correlation Info Note (string)</entry>
2881 <entry>correlationInfo.id</entry>
2882 <entry>Correlation Info ID (integer)</entry>
2886 <entry>elementSetName</entry>
2887 <entry>Element Set for Record</entry>
2891 <entry>updateVersion</entry>
2892 <entry>Record Update version which holds one of the values
2893 1, 2 or 3. Each version has a distinct OID:
2895 (<ulink url="&url.z39.50.extupdate1;">first version</ulink>) ,
2897 (second version) and
2898 1.2.840.10003.9.5.1.1
2899 (<ulink url="&url.z39.50.extupdate3;">third and
2900 newest version</ulink>).
2910 <sect2 id="zoom.database.create"><title>Database Create</title>
2912 For Database Create, <literal>type</literal> must be set to
2913 <literal>create</literal> in
2914 <function>ZOOM_package_send</function>.
2917 <table frame="top" id="zoom.database.create.options">
2918 <title>Database Create Options</title>
2920 <colspec colwidth="4*" colname="name"></colspec>
2921 <colspec colwidth="7*" colname="description"></colspec>
2922 <colspec colwidth="3*" colname="default"></colspec>
2925 <entry>Option</entry>
2926 <entry>Description</entry>
2927 <entry>Default</entry>
2932 <entry>databaseName</entry>
2933 <entry>Database from connection object</entry>
2934 <entry>Default</entry>
2940 <sect2 id="zoom.database.drop">
2941 <title>Database Drop</title>
2943 For Database Drop, <literal>type</literal> must be set to
2944 <literal>drop</literal> in
2945 <function>ZOOM_package_send</function>.
2947 <table frame="top" id="zoom.database.drop.options">
2948 <title>Database Drop Options</title>
2950 <colspec colwidth="4*" colname="name"></colspec>
2951 <colspec colwidth="7*" colname="description"></colspec>
2952 <colspec colwidth="3*" colname="default"></colspec>
2955 <entry>Option</entry>
2956 <entry>Description</entry>
2957 <entry>Default</entry>
2962 <entry>databaseName</entry>
2963 <entry>Database from connection object</entry>
2964 <entry>Default</entry>
2970 <sect2 id="zoom.commit">
2971 <title>Commit Operation</title>
2973 For Commit, <literal>type</literal> must be set to
2974 <literal>commit</literal> in
2975 <function>ZOOM_package_send</function>.
2978 <sect2 id="zoom.extended.services.behavior">
2979 <title>Protocol behavior</title>
2981 All the extended services are Z39.50-only.
2985 The database create, drop, and commit services are privately defined
2987 Refer to <filename>esadmin.asn</filename> in YAZ for the ASN.1
2993 <sect1 id="zoom.options">
2994 <title>Options</title>
2996 Most &zoom; objects provide a way to specify options to change behavior.
2997 From an implementation point of view, a set of options is just like
2998 an associative array / hash.
3001 ZOOM_options ZOOM_options_create(void);
3003 ZOOM_options ZOOM_options_create_with_parent(ZOOM_options parent);
3005 void ZOOM_options_destroy(ZOOM_options opt);
3008 const char *ZOOM_options_get(ZOOM_options opt, const char *name);
3010 void ZOOM_options_set(ZOOM_options opt, const char *name,
3014 typedef const char *(*ZOOM_options_callback)
3015 (void *handle, const char *name);
3017 ZOOM_options_callback
3018 ZOOM_options_set_callback(ZOOM_options opt,
3019 ZOOM_options_callback c,
3023 <sect1 id="zoom.queryconversions">
3024 <title>Query conversions</title>
3026 int ZOOM_query_cql2rpn(ZOOM_query s, const char *cql_str,
3027 ZOOM_connection conn);
3029 int ZOOM_query_ccl2rpn(ZOOM_query s, const char *ccl_str,
3031 int *ccl_error, const char **error_string,
3035 <function>ZOOM_query_cql2rpn</function> translates the CQL string,
3036 client-side, into RPN which may be passed to the server.
3037 This is useful for servers that don't themselves
3038 support CQL, for which <function>ZOOM_query_cql</function> is useless.
3039 'conn' is used only as a place to stash diagnostics if compilation
3040 fails; if this information is not needed, a null pointer may be used.
3041 The CQL conversion is driven by option <literal>cqlfile</literal> from
3042 connection conn. This specifies a conversion file (e.g. pqf.properties)
3043 which <emphasis>must</emphasis> be present.
3046 <function>ZOOM_query_ccl2rpn</function> translates the CCL string,
3047 client-side, into RPN which may be passed to the server.
3048 The conversion is driven by the specification given by
3049 <literal>config</literal>. Upon completion 0 is returned on success; -1
3050 is returned on failure. On failure <literal>error_string</literal> and
3051 <literal>error_pos</literal> hold the error message and position of
3052 first error in original CCL string.
3055 <sect1 id="zoom.events"><title>Events</title>
3057 If you're developing non-blocking applications, you have to deal
3061 int ZOOM_event(int no, ZOOM_connection *cs);
3064 The <function>ZOOM_event</function> executes pending events for
3065 a number of connections. Supply the number of connections in
3066 <literal>no</literal> and an array of connections in
3067 <literal>cs</literal> (<literal>cs[0] ... cs[no-1]</literal>).
3068 A pending event could be sending a search, receiving a response,
3070 When an event has occurred for one of the connections, this function
3071 returns a positive integer <literal>n</literal> denoting that an event
3072 occurred for connection <literal>cs[n-1]</literal>.
3073 When no events are pending for the connections, a value of zero is
3075 To ensure that all outstanding requests are performed, call this function
3076 repeatedly until zero is returned.
3079 If <function>ZOOM_event</function> returns, and returns non-zero, the
3080 last event that occurred can be expected.
3083 int ZOOM_connection_last_event(ZOOM_connection cs);
3086 <function>ZOOM_connection_last_event</function> returns an event type
3087 (integer) for the last event.
3090 <table frame="top" id="zoom.event.ids">
3091 <title>ZOOM Event IDs</title>
3093 <colspec colwidth="4*" colname="name"></colspec>
3094 <colspec colwidth="7*" colname="description"></colspec>
3097 <entry>Event</entry>
3098 <entry>Description</entry>
3103 <entry>ZOOM_EVENT_NONE</entry>
3104 <entry>No event has occurred</entry>
3107 <entry>ZOOM_EVENT_CONNECT</entry>
3108 <entry>TCP/IP connect has initiated</entry>
3111 <entry>ZOOM_EVENT_SEND_DATA</entry>
3112 <entry>Data has been transmitted (sending)</entry>
3115 <entry>ZOOM_EVENT_RECV_DATA</entry>
3116 <entry>Data has been received</entry>
3119 <entry>ZOOM_EVENT_TIMEOUT</entry>
3120 <entry>Timeout</entry>
3123 <entry>ZOOM_EVENT_UNKNOWN</entry>
3124 <entry>Unknown event</entry>
3127 <entry>ZOOM_EVENT_SEND_APDU</entry>
3128 <entry>An APDU has been transmitted (sending)</entry>
3131 <entry>ZOOM_EVENT_RECV_APDU</entry>
3132 <entry>An APDU has been received</entry>
3135 <entry>ZOOM_EVENT_RECV_RECORD</entry>
3136 <entry>A result-set record has been received</entry>
3139 <entry>ZOOM_EVENT_RECV_SEARCH</entry>
3140 <entry>A search result has been received</entry>
3147 <chapter id="server">
3148 <title>Generic server</title>
3149 <sect1 id="server.introduction"><title>Introduction</title>
3151 If you aren't into documentation, a good way to learn how the
3152 back end interface works is to look at the <filename>backend.h</filename>
3153 file. Then, look at the small dummy-server in
3154 <filename>ztest/ztest.c</filename>. The <filename>backend.h</filename>
3155 file also makes a good reference, once you've chewed your way through
3156 the prose of this file.
3159 If you have a database system that you would like to make available by
3160 means of Z39.50 or SRU, &yaz; basically offers two options. You
3161 can use the APIs provided by the &asn;, &odr;, and &comstack;
3163 create and decode PDUs, and exchange them with a client.
3164 Using this low-level interface gives you access to all fields and
3165 options of the protocol, and you can construct your server as close
3166 to your existing database as you like.
3167 It is also a fairly involved process, requiring
3168 you to set up an event-handling mechanism, protocol state machine,
3169 etc. To simplify server implementation, we have implemented a compact
3170 and simple, but reasonably full-functioned server-frontend that will
3171 handle most of the protocol mechanics, while leaving you to
3172 concentrate on your database interface.
3176 The backend interface was designed in anticipation of a specific
3177 integration task, while still attempting to achieve some degree of
3178 generality. We realize fully that there are points where the
3179 interface can be improved significantly. If you have specific
3180 functions or parameters that you think could be useful, send us a
3181 mail (or better, sign on to the mailing list referred to in the
3182 top-level README file). We will try to fit good suggestions into future
3183 releases, to the extent that it can be done without requiring
3184 too many structural changes in existing applications.
3189 The &yaz; server does not support XCQL.
3193 <sect1 id="server.frontend">
3194 <title>The Database Frontend</title>
3196 We refer to this software as a generic database frontend. Your
3197 database system is the <emphasis>backend database</emphasis>, and the
3198 interface between the two is called the <emphasis>backend API</emphasis>.
3199 The backend API consists of a small number of function handlers and
3200 structure definitions. You are required to provide the
3201 <function>main()</function> routine for the server (which can be
3202 quite simple), as well as a set of handlers to match each of the
3204 The interface functions that you write can use any mechanism you like
3205 to communicate with your database system: You might link the whole
3206 thing together with your database application and access it by
3207 function calls; you might use IPC to talk to a database server
3208 somewhere; or you might link with third-party software that handles
3209 the communication for you (like a commercial database client library).
3210 At any rate, the handlers will perform the tasks of:
3223 Scanning the database index (optional - if you wish to implement SCAN).
3226 Extended Services (optional).
3229 Result-Set Delete (optional).
3232 Result-Set Sort (optional).
3235 Return Explain for SRU (optional).
3239 (more functions will be added in time to support as much of
3240 Z39.50-1995 as possible).
3243 <sect1 id="server.backend">
3244 <title>The Backend API</title>
3246 The header file that you need to use the interface are in the
3247 <filename>include/yaz</filename> directory. It's called
3248 <filename>backend.h</filename>. It will include other files from
3249 the <filename>include/yaz</filename> directory, so you'll
3250 probably want to use the -I option of your compiler to tell it
3251 where to find the files. When you run
3252 <literal>make</literal> in the top-level &yaz; directory,
3253 everything you need to create your server is to link with the
3254 <filename>lib/libyaz.la</filename> library.
3257 <sect1 id="server.main">
3258 <title>Your main() Routine</title>
3260 As mentioned, your <function>main()</function> routine can be quite brief.
3261 If you want to initialize global parameters, or read global configuration
3262 tables, this is the place to do it. At the end of the routine, you should
3266 int statserv_main(int argc, char **argv,
3267 bend_initresult *(*bend_init)(bend_initrequest *r),
3268 void (*bend_close)(void *handle));
3271 The third and fourth arguments are pointers to handlers. Handler
3272 <function>bend_init</function> is called whenever the server receives
3273 an Initialize Request, so it serves as a Z39.50 session initializer. The
3274 <function>bend_close</function> handler is called when the session is
3278 <function>statserv_main</function> will establish listening sockets
3279 according to the parameters given. When connection requests are received,
3280 the event handler will typically <function>fork()</function> and
3281 create a sub-process to handle a new connection.
3282 Alternatively the server may be setup to create threads for each
3284 If you do use global variables and forking, you should be aware, then,
3285 that these cannot be shared between associations, unless you explicitly
3286 disable forking by command line parameters.
3289 The server provides a mechanism for controlling some of its behavior
3290 without using command-line options. The function
3293 statserv_options_block *statserv_getcontrol(void);
3296 will return a pointer to a <literal>struct statserv_options_block</literal>
3297 describing the current default settings of the server. The structure
3298 contains these elements:
3301 <term><literal>int dynamic</literal></term>
3303 A boolean value, which determines whether the server
3304 will fork on each incoming request (TRUE), or not (FALSE). Default is
3305 TRUE. This flag is only read by UNIX-based servers (WIN32-based servers
3310 <term><literal>int threads</literal></term>
3312 A boolean value, which determines whether the server
3313 will create a thread on each incoming request (TRUE), or not (FALSE).
3314 Default is FALSE. This flag is only read by UNIX-based servers
3315 that offer POSIX Threads support.
3316 WIN32-based servers always operate in threaded mode.
3320 <term><literal>int inetd</literal></term>
3322 A boolean value, which determines whether the server
3323 will operate under a UNIX INET daemon (inetd). Default is FALSE.
3327 <term><literal>char logfile[ODR_MAXNAME+1]</literal></term>
3328 <listitem><para>File for diagnostic output ("": stderr).
3332 <term><literal>char apdufile[ODR_MAXNAME+1]</literal></term>
3334 Name of file for logging incoming and outgoing APDUs
3335 ("": don't log APDUs, "-":
3336 <literal>stderr</literal>).
3340 <term><literal>char default_listen[1024]</literal></term>
3341 <listitem><para>Same form as the command-line specification of
3342 listener address. "": no default listener address.
3343 Default is to listen at "tcp:@:9999". You can only
3344 specify one default listener address in this fashion.
3348 <term><literal>enum oid_proto default_proto;</literal></term>
3349 <listitem><para>Either <literal>PROTO_Z3950</literal> or
3350 <literal>PROTO_SR</literal>.
3351 Default is <literal>PROTO_Z39_50</literal>.
3355 <term><literal>int idle_timeout;</literal></term>
3356 <listitem><para>Maximum session idle-time, in minutes. Zero indicates
3357 no (infinite) timeout. Default is 15 minutes.
3361 <term><literal>int maxrecordsize;</literal></term>
3362 <listitem><para>Maximum permissible record (message) size. Default
3363 is 64 MB. This amount of memory will only be allocated if a
3364 client requests a very large amount of records in one operation
3366 Set it to a lower number if you are worried about resource
3367 consumption on your host system.
3371 <term><literal>char configname[ODR_MAXNAME+1]</literal></term>
3372 <listitem><para>Passed to the backend when a new connection is received.
3376 <term><literal>char setuid[ODR_MAXNAME+1]</literal></term>
3377 <listitem><para>Set user id to the user specified, after binding
3378 the listener addresses.
3383 <literal>void (*bend_start)(struct statserv_options_block *p)</literal>
3385 <listitem><para>Pointer to function which is called after the
3386 command line options have been parsed - but before the server
3388 For forked UNIX servers, this handler is called in the mother
3389 process; for threaded servers, this handler is called in the
3391 The default value of this pointer is NULL in which case it
3392 isn't invoked by the frontend server.
3393 When the server operates as an NT service, this handler is called
3394 whenever the service is started.
3399 <literal>void (*bend_stop)(struct statserv_options_block *p)</literal>
3401 <listitem><para>Pointer to function which is called whenever the server
3402 has stopped listening for incoming connections. This function pointer
3403 has a default value of NULL in which case it isn't called.
3404 When the server operates as an NT service, this handler is called
3405 whenever the service is stopped.
3409 <term><literal>void *handle</literal></term>
3410 <listitem><para>User defined pointer (default value NULL).
3411 This is a per-server handle that can be used to specify "user-data".
3412 Do not confuse this with the session-handle as returned by bend_init.
3418 The pointer returned by <literal>statserv_getcontrol</literal> points to
3419 a static area. You are allowed to change the contents of the structure,
3420 but the changes will not take effect until you call
3423 void statserv_setcontrol(statserv_options_block *block);
3427 You should generally update this structure before calling
3428 <function>statserv_main()</function>.
3432 <sect1 id="server.backendfunctions">
3433 <title>The Backend Functions</title>
3435 For each service of the protocol, the backend interface declares one or
3436 two functions. You are required to provide implementations of the
3437 functions representing the services that you wish to implement.
3439 <sect2 id="server.init">
3442 bend_initresult (*bend_init)(bend_initrequest *r);
3445 This handler is called once for each new connection request, after
3446 a new process/thread has been created, and an Initialize Request has
3447 been received from the client. The pointer to the
3448 <function>bend_init</function> handler is passed in the call to
3449 <function>statserv_start</function>.
3452 This handler is also called when operating in SRU mode - when
3453 a connection has been made (even though SRU does not offer
3457 Unlike previous versions of YAZ, the <function>bend_init</function> also
3458 serves as a handler that defines the Z39.50 services that the backend
3459 intends to support. Pointers to <emphasis>all</emphasis> service handlers,
3460 including search - and fetch must be specified here in this handler.
3463 The request - and result structures are defined as
3466 typedef struct bend_initrequest
3468 /** \brief user/name/password to be read */
3469 Z_IdAuthentication *auth;
3470 /** \brief encoding stream (for results) */
3472 /** \brief printing stream */
3474 /** \brief decoding stream (use stream for results) */
3476 /** \brief reference ID */
3477 Z_ReferenceId *referenceId;
3478 /** \brief peer address of client */
3481 /** \brief character set and language negotiation
3483 see include/yaz/z-charneg.h
3485 Z_CharSetandLanguageNegotiation *charneg_request;
3487 /** \brief character negotiation response */
3488 Z_External *charneg_response;
3490 /** \brief character set (encoding) for query terms
3492 This is NULL by default. It should be set to the native character
3493 set that the backend assumes for query terms */
3494 char *query_charset;
3496 /** \brief whehter query_charset also applies to recors
3498 Is 0 (No) by default. Set to 1 (yes) if records is in the same
3499 character set as queries. If in doubt, use 0 (No).
3501 int records_in_same_charset;
3503 char *implementation_id;
3504 char *implementation_name;
3505 char *implementation_version;
3507 /** \brief Z39.50 sort handler */
3508 int (*bend_sort)(void *handle, bend_sort_rr *rr);
3509 /** \brief SRU/Z39.50 search handler */
3510 int (*bend_search)(void *handle, bend_search_rr *rr);
3511 /** \brief SRU/Z39.50 fetch handler */
3512 int (*bend_fetch)(void *handle, bend_fetch_rr *rr);
3513 /** \brief SRU/Z39.50 present handler */
3514 int (*bend_present)(void *handle, bend_present_rr *rr);
3515 /** \brief Z39.50 extended services handler */
3516 int (*bend_esrequest) (void *handle, bend_esrequest_rr *rr);
3517 /** \brief Z39.50 delete result set handler */
3518 int (*bend_delete)(void *handle, bend_delete_rr *rr);
3519 /** \brief Z39.50 scan handler */
3520 int (*bend_scan)(void *handle, bend_scan_rr *rr);
3521 /** \brief Z39.50 segment facility handler */
3522 int (*bend_segment)(void *handle, bend_segment_rr *rr);
3523 /** \brief SRU explain handler */
3524 int (*bend_explain)(void *handle, bend_explain_rr *rr);
3525 /** \brief SRU scan handler */
3526 int (*bend_srw_scan)(void *handle, bend_scan_rr *rr);
3527 /** \brief SRU record update handler */
3528 int (*bend_srw_update)(void *handle, bend_update_rr *rr);
3530 /** \brief whether named result sets are supported (0=disable, 1=enable) */
3531 int named_result_sets;
3534 typedef struct bend_initresult
3536 int errcode; /* 0==OK */
3537 char *errstring; /* system error string or NULL */
3538 void *handle; /* private handle to the backend module */
3542 In general, the server frontend expects that the
3543 <literal>bend_*result</literal> pointer that you return is valid at
3544 least until the next call to a <literal>bend_* function</literal>.
3545 This applies to all of the functions described herein. The parameter
3546 structure passed to you in the call belongs to the server frontend, and
3547 you should not make assumptions about its contents after the current
3548 function call has completed. In other words, if you want to retain any
3549 of the contents of a request structure, you should copy them.
3552 The <literal>errcode</literal> should be zero if the initialization of
3553 the backend went well. Any other value will be interpreted as an error.
3554 The <literal>errstring</literal> isn't used in the current version, but
3555 one option would be to stick it in the initResponse as a VisibleString.
3556 The <literal>handle</literal> is the most important parameter. It should
3557 be set to some value that uniquely identifies the current session to
3558 the backend implementation. It is used by the frontend server in any
3559 future calls to a backend function.
3560 The typical use is to set it to point to a dynamically allocated state
3561 structure that is private to your backend module.
3564 The <literal>auth</literal> member holds the authentication information
3565 part of the Z39.50 Initialize Request. Interpret this if your server
3566 requires authentication.
3569 The members <literal>peer_name</literal>,
3570 <literal>implementation_id</literal>,
3571 <literal>implementation_name</literal> and
3572 <literal>implementation_version</literal> holds
3573 DNS of client, ID of implementor, name
3574 of client (Z39.50) implementation - and version.
3577 The <literal>bend_</literal> - members are set to NULL when
3578 <function>bend_init</function> is called. Modify the pointers by
3579 setting them to point to backend functions.
3582 <sect2 id="server.search.retrieve">
3583 <title>Search and Retrieve</title>
3585 We now describe the handlers that are required to support search -
3586 and retrieve. You must support two functions - one for search - and one
3587 for fetch (retrieval of one record). If desirable you can provide a
3588 third handler which is called when a present request is received which
3589 allows you to optimize retrieval of multiple-records.
3592 int (*bend_search) (void *handle, bend_search_rr *rr);
3595 char *setname; /* name to give to this set */
3596 int replace_set; /* replace set, if it already exists */
3597 int num_bases; /* number of databases in list */
3598 char **basenames; /* databases to search */
3599 Z_ReferenceId *referenceId;/* reference ID */
3600 Z_Query *query; /* query structure */
3601 ODR stream; /* encode stream */
3602 ODR decode; /* decode stream */
3603 ODR print; /* print stream */
3605 bend_request request;
3606 bend_association association;
3608 int hits; /* number of hits */
3609 int errcode; /* 0==OK */
3610 char *errstring; /* system error string or NULL */
3611 Z_OtherInformation *search_info; /* additional search info */
3612 char *srw_sortKeys; /* holds SRU/SRW sortKeys info */
3613 char *srw_setname; /* holds SRU/SRW generated resultsetID */
3614 int *srw_setnameIdleTime; /* holds SRU/SRW life-time */
3615 int estimated_hit_count; /* if hit count is estimated */
3616 int partial_resultset; /* if result set is partial */
3620 The <function>bend_search</function> handler is a fairly close
3621 approximation of a protocol Z39.50 Search Request - and Response PDUs.
3622 The <literal>setname</literal> is the resultSetName from the protocol.
3623 You are required to establish a mapping between the set name and whatever
3624 your backend database likes to use.
3625 Similarly, the <literal>replace_set</literal> is a boolean value
3626 corresponding to the resultSetIndicator field in the protocol.
3627 <literal>num_bases/basenames</literal> is a length of/array of character
3628 pointers to the database names provided by the client.
3629 The <literal>query</literal> is the full query structure as defined in
3630 the protocol ASN.1 specification.
3631 It can be either of the possible query types, and it's up to you to
3632 determine if you can handle the provided query type.
3633 Rather than reproduce the C interface here, we'll refer you to the
3634 structure definitions in the file
3635 <filename>include/yaz/z-core.h</filename>. If you want to look at the
3636 attributeSetId OID of the RPN query, you can either match it against
3637 your own internal tables, or you can use the <link linkend="tools.oid">
3641 The structure contains a number of hits, and an
3642 <literal>errcode/errstring</literal> pair. If an error occurs
3643 during the search, or if you're unhappy with the request, you should
3644 set the errcode to a value from the BIB-1 diagnostic set. The value
3645 will then be returned to the user in a nonsurrogate diagnostic record
3646 in the response. The <literal>errstring</literal>, if provided, will
3647 go in the addinfo field. Look at the protocol definition for the
3648 defined error codes, and the suggested uses of the addinfo field.
3651 The <function>bend_search</function> handler is also called when
3652 the frontend server receives a SRU SearchRetrieveRequest.
3653 For SRU, a CQL query is usually provided by the client.
3654 The CQL query is available as part of <literal>Z_Query</literal>
3655 structure (note that CQL is now part of Z39.50 via an external).
3656 To support CQL in existing implementations that only do Type-1,
3657 we refer to the CQL-to-PQF tool described
3658 <link linkend="cql.to.pqf">here</link>.
3661 To maintain backwards compatibility, the frontend server
3662 of yaz always assume that error codes are BIB-1 diagnostics.
3663 For SRU operation, a Bib-1 diagnostic code is mapped to
3667 int (*bend_fetch) (void *handle, bend_fetch_rr *rr);
3669 typedef struct bend_fetch_rr {
3670 char *setname; /* set name */
3671 int number; /* record number */
3672 Z_ReferenceId *referenceId;/* reference ID */
3673 Odr_oid *request_format; /* format, transfer syntax (OID) */
3674 Z_RecordComposition *comp; /* Formatting instructions */
3675 ODR stream; /* encoding stream - memory source if req */
3676 ODR print; /* printing stream */
3678 char *basename; /* name of database that provided record */
3679 int len; /* length of record or -1 if structured */
3680 char *record; /* record */
3681 int last_in_set; /* is it? */
3682 Odr_oid *output_format; /* response format/syntax (OID) */
3683 int errcode; /* 0==success */
3684 char *errstring; /* system error string or NULL */
3685 int surrogate_flag; /* surrogate diagnostic */
3686 char *schema; /* string record schema input/output */
3690 The frontend server calls the <function>bend_fetch</function> handler
3691 when it needs database records to fulfill a Z39.50 Search Request, a
3692 Z39.50 Present Request or a SRU SearchRetrieveRequest.
3693 The <literal>setname</literal> is simply the name of the result set
3694 that holds the reference to the desired record.
3695 The <literal>number</literal> is the offset into the set (with 1
3696 being the first record in the set). The <literal>format</literal> field
3697 is the record format requested by the client (See
3698 <xref linkend="tools.oid"/>).
3699 A value of NULL for <literal>format</literal> indicates that the
3700 client did not request a specific format.
3701 The <literal>stream</literal> argument is an &odr; stream which
3702 should be used for allocating space for structured data records.
3703 The stream will be reset when all records have been assembled, and
3704 the response package has been transmitted.
3705 For unstructured data, the backend is responsible for maintaining a
3706 static or dynamic buffer for the record between calls.
3709 If a SRU SearchRetrieveRequest is received by the frontend server,
3710 the <literal>referenceId</literal> is NULL and the
3711 <literal>format</literal> (transfer syntax) is the OID for XML.
3712 The schema for SRU is stored in both the
3713 <literal>Z_RecordComposition</literal>
3714 structure and <literal>schema</literal> (simple string).
3717 In the structure, the <literal>basename</literal> is the name of the
3718 database that holds the
3719 record. <literal>len</literal> is the length of the record returned, in
3720 bytes, and <literal>record</literal> is a pointer to the record.
3721 <literal>last_in_set</literal> should be nonzero only if the record
3722 returned is the last one in the given result set.
3723 <literal>errcode</literal> and <literal>errstring</literal>, if
3724 given, will be interpreted as a global error pertaining to the
3725 set, and will be returned in a non-surrogate-diagnostic.
3726 If you wish to return the error as a surrogate-diagnostic
3727 (local error) you can do this by setting
3728 <literal>surrogate_flag</literal> to 1 also.
3731 If the <literal>len</literal> field has the value -1, then
3732 <literal>record</literal> is assumed to point to a constructed data
3733 type. The <literal>format</literal> field will be used to determine
3734 which encoder should be used to serialize the data.
3738 If your backend generates structured records, it should use
3739 <function>odr_malloc()</function> on the provided stream for allocating
3740 data: This allows the frontend server to keep track of the record sizes.
3744 The <literal>format</literal> field is mapped to an object identifier
3745 in the direct reference of the resulting EXTERNAL representation
3750 The current version of &yaz; only supports the direct reference mode.
3754 int (*bend_present) (void *handle, bend_present_rr *rr);
3757 char *setname; /* set name */
3759 int number; /* record number */
3760 Odr_oid *format; /* format, transfer syntax (OID) */
3761 Z_ReferenceId *referenceId;/* reference ID */
3762 Z_RecordComposition *comp; /* Formatting instructions */
3763 ODR stream; /* encoding stream - memory source if required */
3764 ODR print; /* printing stream */
3765 bend_request request;
3766 bend_association association;
3768 int hits; /* number of hits */
3769 int errcode; /* 0==OK */
3770 char *errstring; /* system error string or NULL */
3774 The <function>bend_present</function> handler is called when
3775 the server receives a Z39.50 Present Request.
3776 The <literal>setname</literal>,
3777 <literal>start</literal> and <literal>number</literal> is the
3778 name of the result set - start position - and number of records to
3779 be retrieved respectively. <literal>format</literal> and
3780 <literal>comp</literal> is the preferred transfer syntax and element
3781 specifications of the present request.
3784 Note that this is handler serves as a supplement for
3785 <function>bend_fetch</function> and need not to be defined in order to
3786 support search - and retrieve.
3789 <sect2 id="server.delete">
3790 <title>Delete</title>
3792 For back-ends that supports delete of a result set, only one handler
3796 int (*bend_delete)(void *handle, bend_delete_rr *rr);
3798 typedef struct bend_delete_rr {
3802 Z_ReferenceId *referenceId;
3803 int delete_status; /* status for the whole operation */
3804 int *statuses; /* status each set - indexed as setnames */
3811 The delete set function definition is rather primitive, mostly because
3812 we have had no practical need for it as of yet. If someone wants
3813 to provide a full delete service, we'd be happy to add the
3814 extra parameters that are required. Are there clients out there
3815 that will actually delete sets they no longer need?
3819 <sect2 id="server.scan">
3822 For servers that wish to offer the scan service one handler
3826 int (*bend_scan)(void *handle, bend_scan_rr *rr);
3829 BEND_SCAN_SUCCESS, /* ok */
3830 BEND_SCAN_PARTIAL /* not all entries could be found */
3833 typedef struct bend_scan_rr {
3834 int num_bases; /* number of elements in databaselist */
3835 char **basenames; /* databases to search */
3836 Odr_oid *attributeset;
3837 Z_ReferenceId *referenceId; /* reference ID */
3838 Z_AttributesPlusTerm *term;
3839 ODR stream; /* encoding stream - memory source if required */
3840 ODR print; /* printing stream */
3842 int *step_size; /* step size */
3843 int term_position; /* desired index of term in result list/returned */
3844 int num_entries; /* number of entries requested/returned */
3846 /* scan term entries. The called handler does not have
3847 to allocate this. Size of entries is num_entries (see above) */
3848 struct scan_entry *entries;
3849 bend_scan_status status;
3852 char *scanClause; /* CQL scan clause */
3853 char *setname; /* Scan in result set (NULL if omitted) */
3857 This backend server handles both Z39.50 scan
3858 and SRU scan. In order for a handler to distinguish between SRU (CQL) scan
3859 Z39.50 Scan, it must check for a non-NULL value of
3860 <literal>scanClause</literal>.
3864 If designed today, it would be a choice using a union or similar,
3865 but that would break binary compatibility with existing servers.
3870 <sect1 id="server.invocation">
3871 <title>Application Invocation</title>
3873 The finished application has the following
3874 invocation syntax (by way of <function>statserv_main()</function>):
3882 A listener specification consists of a transport mode followed by a
3883 colon (:) followed by a listener address. The transport mode is
3884 either <literal>tcp</literal>, <literal>unix:</literal> or
3885 <literal>ssl</literal>.
3888 For TCP and SSL, an address has the form
3891 hostname | IP-number [: portnumber]
3894 The port number defaults to 210 (standard Z39.50 port).
3897 For UNIX, the address is the filename of socket.
3900 For TCP/IP and SSL, the special hostnames <literal>@</literal>,
3901 maps to <literal>IN6ADDR_ANY_INIT</literal> with
3902 IPV4 binding as well (bindv6only=0),
3903 The special hostname <literal>@4</literal> binds to
3904 <literal>INADDR_ANY</literal> (IPV4 only listener).
3905 The special hostname <literal>@6</literal> binds to
3906 <literal>IN6ADDR_ANY_INIT</literal> with bindv6only=1 (IPV6 only listener).
3908 <example id="server.example.running.unix">
3909 <title>Running the GFS on Unix</title>
3911 Assuming the server application <replaceable>appname</replaceable> is
3912 started as root, the following will make it listen on port 210.
3913 The server will change identity to <literal>nobody</literal>
3914 and write its log to <filename>/var/log/app.log</filename>.
3916 application -l /var/log/app.log -u nobody tcp:@:210
3920 The server will accept Z39.50 requests and offer SRU service on port 210.
3923 <example id="server.example.apache.sru">
3924 <title>Setting up Apache as SRU Frontend</title>
3926 If you use <ulink url="&url.apache;">Apache</ulink>
3927 as your public web server and want to offer HTTP port 80
3928 access to the YAZ server on 210, you can use the
3929 <ulink url="&url.apache.directive.proxypass;">
3930 <literal>ProxyPass</literal></ulink>
3932 If you have virtual host
3933 <literal>srw.mydomain</literal> you can use the following directives
3934 in Apache's httpd.conf:
3937 ErrorLog /home/srw/logs/error_log
3938 TransferLog /home/srw/logs/access_log
3939 ProxyPass / http://srw.mydomain:210/
3944 The above is for the Apache 1.3 series.
3947 <example id="server.example.local.access">
3948 <title>Running a server with local access only</title>
3950 A server that is only being accessed from the local host should listen
3951 on UNIX file socket rather than an Internet socket. To listen on
3952 <filename>/tmp/mysocket</filename> start the server as follows:
3954 application unix:/tmp/mysocket
3959 <sect1 id="server.vhosts">
3960 <title>GFS Configuration and Virtual Hosts</title>
3965 <title>The Z39.50 ASN.1 Module</title>
3966 <sect1 id="asn.introduction">
3967 <title>Introduction</title>
3969 The &asn; module provides you with a set of C struct definitions for the
3970 various PDUs of the Z39.50 protocol, as well as for the complex types
3971 appearing within the PDUs. For the primitive data types, the C
3972 representation often takes the form of an ordinary C language type,
3973 such as <literal>Odr_int</literal> which is equivalent to an integral
3974 C integer. For ASN.1 constructs that have no direct
3975 representation in C, such as general octet strings and bit strings,
3976 the &odr; module (see section <link linkend="odr">The ODR Module</link>)
3977 provides auxiliary definitions.
3980 The &asn; module is located in sub directory <filename>z39.50</filename>.
3981 There you'll find C files that implement encoders and decoders for the
3982 Z39.50 types. You'll also find the protocol definitions:
3983 <filename>z3950v3.asn</filename>, <filename>esupdate.asn</filename>,
3987 <sect1 id="asn.preparing">
3988 <title>Preparing PDUs</title>
3990 A structure representing a complex ASN.1 type doesn't in itself contain the
3991 members of that type. Instead, the structure contains
3992 <emphasis>pointers</emphasis> to the members of the type.
3993 This is necessary, in part, to allow a mechanism for specifying which
3994 of the optional structure (SEQUENCE) members are present, and which
3995 are not. It follows that you will need to somehow provide space for
3996 the individual members of the structure, and set the pointers to
3997 refer to the members.
4000 The conversion routines don't care how you allocate and maintain your
4001 C structures - they just follow the pointers that you provide.
4002 Depending on the complexity of your application, and your personal
4003 taste, there are at least three different approaches that you may take
4004 when you allocate the structures.
4007 You can use static or automatic local variables in the function that
4008 prepares the PDU. This is a simple approach, and it provides the most
4009 efficient form of memory management. While it works well for flat
4010 PDUs like the InitReqest, it will generally not be sufficient for say,
4011 the generation of an arbitrarily complex RPN query structure.
4014 You can individually create the structure and its members using the
4015 <function>malloc(2)</function> function. If you want to ensure that
4016 the data is freed when it is no longer needed, you will have to
4017 define a function that individually releases each member of a
4018 structure before freeing the structure itself.
4021 You can use the <function>odr_malloc()</function> function (see
4022 <xref linkend="odr.use"/> for details). When you use
4023 <function>odr_malloc()</function>, you can release all of the
4024 allocated data in a single operation, independent of any pointers and
4025 relations between the data. The <function>odr_malloc()</function> function
4026 is based on a "nibble-memory"
4027 scheme, in which large portions of memory are allocated, and then
4028 gradually handed out with each call to <function>odr_malloc()</function>.
4029 The next time you call <function>odr_reset()</function>, all of the
4030 memory allocated since the last call is recycled for future use (actually,
4031 it is placed on a free-list).
4034 You can combine all of the methods described here. This will often be
4035 the most practical approach. For instance, you might use
4036 <function>odr_malloc()</function> to allocate an entire structure and
4037 some of its elements, while you leave other elements pointing to global
4038 or per-session default variables.
4041 The &asn; module provides an important aid in creating new PDUs. For
4042 each of the PDU types (say, <function>Z_InitRequest</function>), a
4043 function is provided that allocates and initializes an instance of
4044 that PDU type for you. In the case of the InitRequest, the function is
4045 simply named <function>zget_InitRequest()</function>, and it sets up
4046 reasonable default value for all of the mandatory members. The optional
4047 members are generally initialized to null pointers. This last aspect
4048 is very important: it ensures that if the PDU definitions are
4049 extended after you finish your implementation (to accommodate
4050 new versions of the protocol, say), you won't get into trouble with
4051 uninitialized pointers in your structures. The functions use
4052 <function>odr_malloc()</function> to
4053 allocate the PDUs and its members, so you can free everything again with a
4054 single call to <function>odr_reset()</function>. We strongly recommend
4055 that you use the <literal>zget_*</literal>
4056 functions whenever you are preparing a PDU (in a C++ API, the
4057 <literal>zget_</literal>
4058 functions would probably be promoted to constructors for the
4062 The prototype for the individual PDU types generally look like this:
4065 Z_<type> *zget_<type>(ODR o);
4071 Z_InitRequest *zget_InitRequest(ODR o);
4074 The &odr; handle should generally be your encoding stream, but it
4078 As well as the individual PDU functions, a function
4079 <function>zget_APDU()</function> is provided, which allocates
4080 a top-level Z-APDU of the type requested:
4083 Z_APDU *zget_APDU(ODR o, int which);
4086 The <varname>which</varname> parameter is (of course) the discriminator
4087 belonging to the <varname>Z_APDU</varname> <literal>CHOICE</literal> type.
4088 All of the interface described here is provided by the &asn; module, and
4089 you access it through the <filename>proto.h</filename> header file.
4092 <sect1 id="asn.external">
4093 <title>EXTERNAL Data</title>
4095 In order to achieve extensibility and adaptability to different
4096 application domains, the new version of the protocol defines many
4097 structures outside of the main ASN.1 specification, referencing them
4098 through ASN.1 EXTERNAL constructs. To simplify the construction and
4099 access to the externally referenced data, the &asn; module defines a
4100 specialized version of the EXTERNAL construct, called
4101 <literal>Z_External</literal>.It is defined thus:
4104 typedef struct Z_External
4106 Odr_oid *direct_reference;
4107 int *indirect_reference;
4112 Z_External_single = 0,
4114 Z_External_arbitrary,
4116 /* Specific types */
4118 Z_External_explainRecord,
4119 Z_External_resourceReport1,
4120 Z_External_resourceReport2
4128 Odr_any *single_ASN1_type;
4129 Odr_oct *octet_aligned;
4130 Odr_bitmask *arbitrary;
4132 /* Specific types */
4134 Z_ExplainRecord *explainRecord;
4135 Z_ResourceReport1 *resourceReport1;
4136 Z_ResourceReport2 *resourceReport2;
4144 When decoding, the &asn; module will attempt to determine which
4145 syntax describes the data by looking at the reference fields
4146 (currently only the direct-reference). For ASN.1 structured data, you
4147 need only consult the <literal>which</literal> field to determine the
4148 type of data. You can the access the data directly through the union.
4149 When constructing data for encoding, you set the union pointer to point
4150 to the data, and set the <literal>which</literal> field accordingly.
4151 Remember also to set the direct (or indirect) reference to the correct
4152 OID for the data type.
4153 For non-ASN.1 data such as MARC records, use the
4154 <literal>octet_aligned</literal> arm of the union.
4157 Some servers return ASN.1 structured data values (e.g. database
4158 records) as BER-encoded records placed in the
4159 <literal>octet-aligned</literal> branch of the EXTERNAL CHOICE.
4160 The ASN-module will <emphasis>not</emphasis> automatically decode
4161 these records. To help you decode the records in the application, the
4165 Z_ext_typeent *z_ext_gettypebyref(const oid *oid);
4168 can be used to retrieve information about the known, external data
4169 types. The function returns a pointer to a static area, or NULL, if no
4170 match for the given direct reference is found. The
4171 <literal>Z_ext_typeent</literal>
4175 typedef struct Z_ext_typeent
4177 int oid[OID_SIZE]; /* the direct-reference OID. */
4178 int what; /* discriminator value for the external CHOICE */
4179 Odr_fun fun; /* decoder function */
4183 The <literal>what</literal> member contains the
4184 <literal>Z_External</literal> union discriminator value for the
4185 given type: For the SUTRS record syntax, the value would be
4186 <literal>Z_External_sutrs</literal>.
4187 The <literal>fun</literal> member contains a pointer to the
4188 function which encodes/decodes the given type. Again, for the SUTRS
4189 record syntax, the value of <literal>fun</literal> would be
4190 <literal>z_SUTRS</literal> (a function pointer).
4193 If you receive an EXTERNAL which contains an octet-string value that
4194 you suspect of being an ASN.1-structured data value, you can use
4195 <literal>z_ext_gettypebyref</literal> to look for the provided
4197 If the return value is different from NULL, you can use the provided
4198 function to decode the BER string (see <xref linkend="odr.use"/>
4202 If you want to <emphasis>send</emphasis> EXTERNALs containing
4203 ASN.1-structured values in the octet-aligned branch of the CHOICE, this
4204 is possible too. However, on the encoding phase, it requires a somewhat
4205 involved juggling around of the various buffers involved.
4208 If you need to add new, externally defined data types, you must update
4209 the struct above, in the source file <filename>prt-ext.h</filename>, as
4210 well as the encoder/decoder in the file <filename>prt-ext.c</filename>.
4211 When changing the latter, remember to update both the
4212 <literal>arm</literal> arrary and the list
4213 <literal>type_table</literal>, which drives the CHOICE biasing that
4214 is necessary to tell the different, structured types apart
4219 Eventually, the EXTERNAL processing will most likely
4220 automatically insert the correct OIDs or indirect-refs. First,
4221 however, we need to determine how application-context management
4222 (specifically the presentation-context-list) should fit into the
4227 <sect1 id="asn.pdu">
4228 <title>PDU Contents Table</title>
4230 We include, for reference, a listing of the fields of each top-level
4231 PDU, as well as their default settings.
4233 <table frame="top" id="asn.default.initialize.request">
4234 <title>Default settings for PDU Initialize Request</title>
4236 <colspec colwidth="7*" colname="field"></colspec>
4237 <colspec colwidth="5*" colname="type"></colspec>
4238 <colspec colwidth="7*" colname="value"></colspec>
4241 <entry>Field</entry>
4243 <entry>Default Value</entry>
4248 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4251 protocolVersion</entry><entry>Odr_bitmask</entry><entry>Empty bitmask
4254 options</entry><entry>Odr_bitmask</entry><entry>Empty bitmask
4257 preferredMessageSize</entry><entry>Odr_int</entry><entry>30*1024
4260 maximumRecordSize</entry><entry>Odr_int</entry><entry>30*1024
4263 idAuthentication</entry><entry>Z_IdAuthentication</entry><entry>NULL
4266 implementationId</entry><entry>char*</entry><entry>"81"
4269 implementationName</entry><entry>char*</entry><entry>"YAZ"
4272 implementationVersion</entry><entry>char*</entry><entry>YAZ_VERSION
4275 userInformationField</entry><entry>Z_UserInformation</entry><entry>NULL
4278 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4283 <table frame="top" id="asn.default.initialize.response">
4284 <title>Default settings for PDU Initialize Response</title>
4286 <colspec colwidth="7*" colname="field"></colspec>
4287 <colspec colwidth="5*" colname="type"></colspec>
4288 <colspec colwidth="7*" colname="value"></colspec>
4291 <entry>Field</entry>
4293 <entry>Default Value</entry>
4298 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4301 protocolVersion</entry><entry>Odr_bitmask</entry><entry>Empty bitmask
4304 options</entry><entry>Odr_bitmask</entry><entry>Empty bitmask
4307 preferredMessageSize</entry><entry>Odr_int</entry><entry>30*1024
4310 maximumRecordSize</entry><entry>Odr_int</entry><entry>30*1024
4313 result</entry><entry>Odr_bool</entry><entry>TRUE
4316 implementationId</entry><entry>char*</entry><entry>"id)"
4319 implementationName</entry><entry>char*</entry><entry>"YAZ"
4322 implementationVersion</entry><entry>char*</entry><entry>YAZ_VERSION
4325 userInformationField</entry><entry>Z_UserInformation</entry><entry>NULL
4328 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4333 <table frame="top" id="asn.default.search.request">
4334 <title>Default settings for PDU Search Request</title>
4336 <colspec colwidth="7*" colname="field"></colspec>
4337 <colspec colwidth="5*" colname="type"></colspec>
4338 <colspec colwidth="7*" colname="value"></colspec>
4341 <entry>Field</entry>
4343 <entry>Default Value</entry>
4348 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4351 smallSetUpperBound</entry><entry>Odr_int</entry><entry>0
4354 largeSetLowerBound</entry><entry>Odr_int</entry><entry>1
4357 mediumSetPresentNumber</entry><entry>Odr_int</entry><entry>0
4360 replaceIndicator</entry><entry>Odr_bool</entry><entry>TRUE
4363 resultSetName</entry><entry>char *</entry><entry>"default"
4366 num_databaseNames</entry><entry>Odr_int</entry><entry>0
4369 databaseNames</entry><entry>char **</entry><entry>NULL
4372 smallSetElementSetNames</entry><entry>Z_ElementSetNames
4376 mediumSetElementSetNames</entry><entry>Z_ElementSetNames
4380 preferredRecordSyntax</entry><entry>Odr_oid</entry><entry>NULL
4383 query</entry><entry>Z_Query</entry><entry>NULL
4386 additionalSearchInfo</entry><entry>Z_OtherInformation
4390 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4395 <table frame="top" id="asn.default.search.response">
4396 <title>Default settings for PDU Search Response</title>
4398 <colspec colwidth="7*" colname="field"></colspec>
4399 <colspec colwidth="5*" colname="type"></colspec>
4400 <colspec colwidth="7*" colname="value"></colspec>
4403 <entry>Field</entry>
4405 <entry>Default Value</entry>
4410 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4413 resultCount</entry><entry>Odr_int</entry><entry>0
4416 numberOfRecordsReturned</entry><entry>Odr_int</entry><entry>0
4419 nextResultSetPosition</entry><entry>Odr_int</entry><entry>0
4422 searchStatus</entry><entry>Odr_bool</entry><entry>TRUE
4425 resultSetStatus</entry><entry>Odr_int</entry><entry>NULL
4428 presentStatus</entry><entry>Odr_int</entry><entry>NULL
4431 records</entry><entry>Z_Records</entry><entry>NULL
4434 additionalSearchInfo</entry>
4435 <entry>Z_OtherInformation</entry><entry>NULL
4438 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4443 <table frame="top" id="asn.default.present.request">
4444 <title>Default settings for PDU Present Request</title>
4446 <colspec colwidth="7*" colname="field"></colspec>
4447 <colspec colwidth="5*" colname="type"></colspec>
4448 <colspec colwidth="7*" colname="value"></colspec>
4451 <entry>Field</entry>
4453 <entry>Default Value</entry>
4458 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4461 resultSetId</entry><entry>char*</entry><entry>"default"
4464 resultSetStartPoint</entry><entry>Odr_int</entry><entry>1
4467 numberOfRecordsRequested</entry><entry>Odr_int</entry><entry>10
4470 num_ranges</entry><entry>Odr_int</entry><entry>0
4473 additionalRanges</entry><entry>Z_Range</entry><entry>NULL
4476 recordComposition</entry><entry>Z_RecordComposition</entry><entry>NULL
4479 preferredRecordSyntax</entry><entry>Odr_oid</entry><entry>NULL
4482 maxSegmentCount</entry><entry>Odr_int</entry><entry>NULL
4485 maxRecordSize</entry><entry>Odr_int</entry><entry>NULL
4488 maxSegmentSize</entry><entry>Odr_int</entry><entry>NULL
4491 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4496 <table frame="top" id="asn.default.present.response">
4497 <title>Default settings for PDU Present Response</title>
4499 <colspec colwidth="7*" colname="field"></colspec>
4500 <colspec colwidth="5*" colname="type"></colspec>
4501 <colspec colwidth="7*" colname="value"></colspec>
4504 <entry>Field</entry>
4506 <entry>Default Value</entry>
4511 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4514 numberOfRecordsReturned</entry><entry>Odr_int</entry><entry>0
4517 nextResultSetPosition</entry><entry>Odr_int</entry><entry>0
4520 presentStatus</entry><entry>Odr_int</entry><entry>Z_PresentStatus_success
4523 records</entry><entry>Z_Records</entry><entry>NULL
4526 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4531 <table frame="top" id="asn.default.delete.result.set.request">
4532 <title>Default settings for Delete Result Set Request</title>
4534 <colspec colwidth="7*" colname="field"></colspec>
4535 <colspec colwidth="5*" colname="type"></colspec>
4536 <colspec colwidth="7*" colname="value"></colspec>
4539 <entry>Field</entry>
4541 <entry>Default Value</entry>
4545 <row><entry>referenceId
4546 </entry><entry>Z_ReferenceId</entry><entry>NULL
4549 deleteFunction</entry><entry>Odr_int</entry><entry>Z_DeleteResultSetRequest_list
4552 num_ids</entry><entry>Odr_int</entry><entry>0
4555 resultSetList</entry><entry>char**</entry><entry>NULL
4558 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4563 <table frame="top" id="asn.default.delete.result.set.response">
4564 <title>Default settings for Delete Result Set Response</title>
4566 <colspec colwidth="7*" colname="field"></colspec>
4567 <colspec colwidth="5*" colname="type"></colspec>
4568 <colspec colwidth="7*" colname="value"></colspec>
4571 <entry>Field</entry>
4573 <entry>Default Value</entry>
4578 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4581 deleteOperationStatus</entry><entry>Odr_int</entry>
4582 <entry>Z_DeleteStatus_success</entry></row>
4584 num_statuses</entry><entry>Odr_int</entry><entry>0
4587 deleteListStatuses</entry><entry>Z_ListStatus**</entry><entry>NULL
4590 numberNotDeleted</entry><entry>Odr_int</entry><entry>NULL
4593 num_bulkStatuses</entry><entry>Odr_int</entry><entry>0
4596 bulkStatuses</entry><entry>Z_ListStatus</entry><entry>NUL
4599 deleteMessage</entry><entry>char*</entry><entry>NULL
4602 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4607 <table frame="top" id="asn.default.scan.request">
4608 <title>Default settings for Scan Request</title>
4610 <colspec colwidth="7*" colname="field"></colspec>
4611 <colspec colwidth="5*" colname="type"></colspec>
4612 <colspec colwidth="7*" colname="value"></colspec>
4615 <entry>Field</entry>
4617 <entry>Default Value</entry>
4622 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4625 num_databaseNames</entry><entry>Odr_int</entry><entry>0
4628 databaseNames</entry><entry>char**</entry><entry>NULL
4631 attributeSet</entry><entry>Odr_oid</entry><entry>NULL
4634 termListAndStartPoint</entry><entry>Z_AttributesPlus...
4635 </entry><entry>NULL</entry></row>
4637 stepSize</entry><entry>Odr_int</entry><entry>NULL
4640 numberOfTermsRequested</entry><entry>Odr_int</entry><entry>20
4643 preferredPositionInResponse</entry><entry>Odr_int</entry><entry>NULL
4646 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4651 <table frame="top" id="asn.default.scan.response">
4652 <title>Default settings for Scan Response</title>
4654 <colspec colwidth="7*" colname="field"></colspec>
4655 <colspec colwidth="5*" colname="type"></colspec>
4656 <colspec colwidth="7*" colname="value"></colspec>
4659 <entry>Field</entry>
4661 <entry>Default Value</entry>
4666 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4669 stepSize</entry><entry>Odr_int</entry><entry>NULL
4672 scanStatus</entry><entry>Odr_int</entry><entry>Z_Scan_success
4675 numberOfEntriesReturned</entry><entry>Odr_int</entry><entry>0
4678 positionOfTerm</entry><entry>Odr_int</entry><entry>NULL
4681 entries</entry><entry>Z_ListEntris</entry><entry>NULL
4684 attributeSet</entry><entry>Odr_oid</entry><entry>NULL
4687 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4692 <table frame="top" id="asn.default.trigger.resource.control.request">
4693 <title>Default settings for Trigger Resource Control Request</title>
4695 <colspec colwidth="7*" colname="field"></colspec>
4696 <colspec colwidth="5*" colname="type"></colspec>
4697 <colspec colwidth="7*" colname="value"></colspec>
4700 <entry>Field</entry>
4702 <entry>Default Value</entry>
4707 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4710 requestedAction</entry><entry>Odr_int</entry><entry>
4711 Z_TriggerResourceCtrl_resou..
4714 prefResourceReportFormat</entry><entry>Odr_oid</entry><entry>NULL
4717 resultSetWanted</entry><entry>Odr_bool</entry><entry>NULL
4720 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4725 <table frame="top" id="asn.default.resource.control.request">
4726 <title>Default settings for Resource Control Request</title>
4728 <colspec colwidth="7*" colname="field"></colspec>
4729 <colspec colwidth="5*" colname="type"></colspec>
4730 <colspec colwidth="7*" colname="value"></colspec>
4733 <entry>Field</entry>
4735 <entry>Default Value</entry>
4740 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4743 suspendedFlag</entry><entry>Odr_bool</entry><entry>NULL
4746 resourceReport</entry><entry>Z_External</entry><entry>NULL
4749 partialResultsAvailable</entry><entry>Odr_int</entry><entry>NULL
4752 responseRequired</entry><entry>Odr_bool</entry><entry>FALSE
4755 triggeredRequestFlag</entry><entry>Odr_bool</entry><entry>NULL
4758 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4763 <table frame="top" id="asn.default.resource.control.response">
4764 <title>Default settings for Resource Control Response</title>
4766 <colspec colwidth="7*" colname="field"></colspec>
4767 <colspec colwidth="5*" colname="type"></colspec>
4768 <colspec colwidth="7*" colname="value"></colspec>
4771 <entry>Field</entry>
4773 <entry>Default Value</entry>
4778 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4781 continueFlag</entry><entry>bool_t</entry><entry>TRUE
4784 resultSetWanted</entry><entry>bool_t</entry><entry>NULL
4787 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4792 <table frame="top" id="asn.default.access.control.request">
4793 <title>Default settings for Access Control Request</title>
4795 <colspec colwidth="7*" colname="field"></colspec>
4796 <colspec colwidth="5*" colname="type"></colspec>
4797 <colspec colwidth="7*" colname="value"></colspec>
4800 <entry>Field</entry>
4802 <entry>Default Value</entry>
4807 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4810 which</entry><entry>enum</entry><entry>Z_AccessRequest_simpleForm;
4813 u</entry><entry>union</entry><entry>NULL
4816 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4821 <table frame="top" id="asn.default.access.control.response">
4822 <title>Default settings for Access Control Response</title>
4824 <colspec colwidth="7*" colname="field"></colspec>
4825 <colspec colwidth="5*" colname="type"></colspec>
4826 <colspec colwidth="7*" colname="value"></colspec>
4829 <entry>Field</entry>
4831 <entry>Default Value</entry>
4836 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4839 which</entry><entry>enum</entry><entry>Z_AccessResponse_simpleForm
4842 u</entry><entry>union</entry><entry>NULL
4845 diagnostic</entry><entry>Z_DiagRec</entry><entry>NULL
4848 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4853 <table frame="top" id="asn.default.segment">
4854 <title>Default settings for Segment</title>
4856 <colspec colwidth="7*" colname="field"></colspec>
4857 <colspec colwidth="5*" colname="type"></colspec>
4858 <colspec colwidth="7*" colname="value"></colspec>
4861 <entry>Field</entry>
4863 <entry>Default Value</entry>
4868 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4871 numberOfRecordsReturned</entry><entry>Odr_int</entry><entry>value=0
4874 num_segmentRecords</entry><entry>Odr_int</entry><entry>0
4877 segmentRecords</entry><entry>Z_NamePlusRecord</entry><entry>NULL
4879 <row><entry>otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4884 <table frame="top" id="asn.default.close">
4885 <title>Default settings for Close</title>
4887 <colspec colwidth="7*" colname="field"></colspec>
4888 <colspec colwidth="5*" colname="type"></colspec>
4889 <colspec colwidth="7*" colname="value"></colspec>
4892 <entry>Field</entry>
4894 <entry>Default Value</entry>
4899 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4902 closeReason</entry><entry>Odr_int</entry><entry>Z_Close_finished
4905 diagnosticInformation</entry><entry>char*</entry><entry>NULL
4908 resourceReportFormat</entry><entry>Odr_oid</entry><entry>NULL
4911 resourceFormat</entry><entry>Z_External</entry><entry>NULL
4914 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4922 <title>SOAP and SRU</title>
4923 <sect1 id="soap.introduction">
4924 <title>Introduction</title>
4926 &yaz; uses a very simple implementation of
4927 <ulink url="&url.soap;">SOAP</ulink> that only
4928 (currently) supports what is sufficient to offer SRU SOAP functionality.
4929 The implementation uses the
4930 <ulink url="&url.libxml2.api.tree;">tree API</ulink> of
4931 libxml2 to encode and decode SOAP packages.
4934 Like the Z39.50 ASN.1 module, the &yaz; SRU implementation uses
4935 simple C structs to represent SOAP packages as well as
4939 <sect1 id="soap.http">
4942 &yaz; only offers HTTP as transport carrier for SOAP, but it is
4943 relatively easy to change that.
4946 The following definition of <literal>Z_GDU</literal> (Generic Data
4947 Unit) allows for both HTTP and Z39.50 in one packet.
4950 #include <yaz/zgdu.h>
4952 #define Z_GDU_Z3950 1
4953 #define Z_GDU_HTTP_Request 2
4954 #define Z_GDU_HTTP_Response 3
4959 Z_HTTP_Request *HTTP_Request;
4960 Z_HTTP_Response *HTTP_Response;
4965 The corresponding Z_GDU encoder/decoder is <function>z_GDU</function>.
4966 The <literal>z3950</literal> is any of the known BER encoded Z39.50
4968 <literal>HTTP_Request</literal> and <literal>HTTP_Response</literal>
4969 is the HTTP Request and Response respectively.
4972 <sect1 id="soap.xml">
4973 <title>SOAP Packages</title>
4975 Every SOAP package in &yaz; is represented as follows:
4977 #include <yaz/soap.h>
4991 #define Z_SOAP_fault 1
4992 #define Z_SOAP_generic 2
4993 #define Z_SOAP_error 3
4997 Z_SOAP_Fault *fault;
4998 Z_SOAP_Generic *generic;
4999 Z_SOAP_Fault *soap_error;
5006 The <literal>fault</literal> and <literal>soap_error</literal>
5007 arms both represent a SOAP fault - struct
5008 <literal>Z_SOAP_Fault</literal>. Any other generic
5009 (valid) package is represented by <literal>Z_SOAP_Generic</literal>.
5012 The <literal>ns</literal> as part of <literal>Z_SOAP</literal>
5013 is the namespace for SOAP itself and reflects the SOAP
5014 version. For version 1.1 it is
5015 <literal>http://schemas.xmlsoap.org/soap/envelope/</literal>,
5016 for version 1.2 it is
5017 <literal>http://www.w3.org/2001/06/soap-envelope</literal>.
5020 int z_soap_codec(ODR o, Z_SOAP **pp,
5021 char **content_buf, int *content_len,
5022 Z_SOAP_Handler *handlers);
5025 The <literal>content_buf</literal> and <literal>content_len</literal>
5026 is XML buffer and length of buffer respectively.
5029 The <literal>handlers</literal> is a list of SOAP codec
5030 handlers - one handler for each service namespace. For SRU SOAP, the
5031 namespace would be <literal>http://www.loc.gov/zing/srw/v1.0/</literal>.
5034 When decoding, the <function>z_soap_codec</function>
5035 inspects the XML content
5036 and tries to match one of the services namespaces of the
5037 supplied handlers. If there is a match. a handler function
5038 is invoked which decodes that particular SOAP package.
5039 If successful, the returned <literal>Z_SOAP</literal> package will be
5040 of type <literal>Z_SOAP_Generic</literal>.
5041 Member <literal>no</literal> is
5042 set the offset of the handler that matched; <literal>ns</literal>
5043 is set to namespace of the matching handler; the void pointer
5044 <literal>p</literal> is set to the C data structure assocatiated
5048 When a NULL namespace is met (member <literal>ns</literal> below),
5049 that specifies end-of-list.
5052 Each handler is defined as follows:
5060 The <literal>ns</literal> is the namespace of the service associated with
5061 handler <literal>f</literal>. The <literal>client_data</literal>
5062 is user-defined data which is passed to the handler.
5065 The prototype for a SOAP service handler is:
5067 int handler(ODR o, void * ptr, void **handler_data,
5068 void *client_data, const char *ns);
5070 The <parameter>o</parameter> specifies the mode (decode/encode)
5071 as usual. The second argument, <parameter>ptr</parameter>,
5072 is a libxml2 tree node pointer (<literal>xmlNodePtr</literal>)
5073 and is a pointer to the <literal>Body</literal> element
5074 of the SOAP package. The <parameter>handler_data</parameter>
5075 is an opaque pointer to C definitions associated with the
5076 SOAP service. The <parameter>client_data</parameter> is the pointer
5077 which was set as part of the <literal>Z_SOAP_handler</literal>.
5078 Finally, <parameter>ns</parameter> is the service namespace.
5081 <sect1 id="soap.srw">
5084 SRU SOAP is just one implementation of a SOAP handler as described
5085 in the previous section.
5086 The encoder/decoder handler for SRU is defined as
5089 #include <yaz/srw.h>
5091 int yaz_srw_codec(ODR o, void * pptr,
5092 Z_SRW_GDU **handler_data,
5093 void *client_data, const char *ns);
5095 Here, <literal>Z_SRW_GDU</literal> is either
5096 searchRetrieveRequest or a searchRetrieveResponse.
5100 The xQuery and xSortKeys are not handled yet by
5101 the SRW implementation of &yaz;. Explain is also missing.
5102 Future versions of &yaz; will include these features.
5106 The definition of searchRetrieveRequest is:
5110 #define Z_SRW_query_type_cql 1
5111 #define Z_SRW_query_type_xcql 2
5112 #define Z_SRW_query_type_pqf 3
5120 #define Z_SRW_sort_type_none 1
5121 #define Z_SRW_sort_type_sort 2
5122 #define Z_SRW_sort_type_xSort 3
5130 int *maximumRecords;
5132 char *recordPacking;
5134 } Z_SRW_searchRetrieveRequest;
5136 Please observe that data of type xsd:string is represented
5137 as a char pointer (<literal>char *</literal>). A null pointer
5138 means that the element is absent.
5139 Data of type xsd:integer is represented as a pointer to
5140 an int (<literal>int *</literal>). Again, a null pointer
5141 is used for absent elements.
5144 The SearchRetrieveResponse has the following definition.
5147 int * numberOfRecords;
5149 int * resultSetIdleTime;
5151 Z_SRW_record *records;
5154 Z_SRW_diagnostic *diagnostics;
5155 int num_diagnostics;
5156 int *nextRecordPosition;
5157 } Z_SRW_searchRetrieveResponse;
5159 The <literal>num_records</literal> and <literal>num_diagnostics</literal>
5160 is number of returned records and diagnostics respectively, and also
5161 correspond to the "size of" arrays <literal>records</literal>
5162 and <literal>diagnostics</literal>.
5165 A retrieval record is defined as follows:
5169 char *recordData_buf;
5171 int *recordPosition;
5174 The record data is defined as a buffer of some length so that
5175 data can be of any type. SRW 1.0 currenly doesn't allow for this
5176 (only XML), but future versions might do.
5179 And, a diagnostic as:
5189 <chapter id="tools">
5190 <title>Supporting Tools</title>
5192 In support of the service API - primarily the ASN module, which
5193 provides the programmatic interface to the Z39.50 APDUs, &yaz; contains
5194 a collection of tools that support the development of applications.
5196 <sect1 id="tools.query">
5197 <title>Query Syntax Parsers</title>
5199 Since the type-1 (RPN) query structure has no direct, useful string
5200 representation, every origin application needs to provide some form of
5201 mapping from a local query notation or representation to a
5202 <token>Z_RPNQuery</token> structure. Some programmers will prefer to
5203 construct the query manually, perhaps using
5204 <function>odr_malloc()</function> to simplify memory management.
5205 The &yaz; distribution includes three separate, query-generating tools
5206 that may be of use to you.
5209 <title>Prefix Query Format</title>
5211 Since RPN or reverse polish notation is really just a fancy way of
5212 describing a suffix notation format (operator follows operands), it
5213 would seem that the confusion is total when we now introduce a prefix
5214 notation for RPN. The reason is one of simple laziness - it's somewhat
5215 simpler to interpret a prefix format, and this utility was designed
5216 for maximum simplicity, to provide a baseline representation for use
5217 in simple test applications and scripting environments (like Tcl). The
5218 demonstration client included with YAZ uses the PQF.
5222 The PQF has been adopted by other parties developing Z39.50
5223 software. It is often referred to as Prefix Query Notation
5228 The PQF is defined by the pquery module in the YAZ library.
5229 There are two sets of functions that have similar behavior. First
5230 set operates on a PQF parser handle, second set doesn't. First set
5231 of functions are more flexible than the second set. Second set
5232 is obsolete and is only provided to ensure backwards compatibility.
5235 First set of functions all operate on a PQF parser handle:
5238 #include <yaz/pquery.h>
5240 YAZ_PQF_Parser yaz_pqf_create(void);
5242 void yaz_pqf_destroy(YAZ_PQF_Parser p);
5244 Z_RPNQuery *yaz_pqf_parse(YAZ_PQF_Parser p, ODR o, const char *qbuf);
5246 Z_AttributesPlusTerm *yaz_pqf_scan(YAZ_PQF_Parser p, ODR o,
5247 Odr_oid **attributeSetId, const char *qbuf);
5249 int yaz_pqf_error(YAZ_PQF_Parser p, const char **msg, size_t *off);
5252 A PQF parser is created and destructed by functions
5253 <function>yaz_pqf_create</function> and
5254 <function>yaz_pqf_destroy</function> respectively.
5255 Function <function>yaz_pqf_parse</function> parses the query given
5256 by string <literal>qbuf</literal>. If parsing was successful,
5257 a Z39.50 RPN Query is returned which is created using ODR stream
5258 <literal>o</literal>. If parsing failed, a NULL pointer is
5260 Function <function>yaz_pqf_scan</function> takes a scan query in
5261 <literal>qbuf</literal>. If parsing was successful, the function
5262 returns attributes plus term pointer and modifies
5263 <literal>attributeSetId</literal> to hold attribute set for the
5264 scan request - both allocated using ODR stream <literal>o</literal>.
5265 If parsing failed, yaz_pqf_scan returns a NULL pointer.
5266 Error information for bad queries can be obtained by a call to
5267 <function>yaz_pqf_error</function> which returns an error code and
5268 modifies <literal>*msg</literal> to point to an error description,
5269 and modifies <literal>*off</literal> to the offset within the last
5270 query where parsing failed.
5273 The second set of functions are declared as follows:
5276 #include <yaz/pquery.h>
5278 Z_RPNQuery *p_query_rpn(ODR o, oid_proto proto, const char *qbuf);
5280 Z_AttributesPlusTerm *p_query_scan(ODR o, oid_proto proto,
5281 Odr_oid **attributeSetP, const char *qbuf);
5283 int p_query_attset(const char *arg);
5286 The function <function>p_query_rpn()</function> takes as arguments an
5287 &odr; stream (see section <link linkend="odr">The ODR Module</link>)
5288 to provide a memory source (the structure created is released on
5289 the next call to <function>odr_reset()</function> on the stream), a
5290 protocol identifier (one of the constants <token>PROTO_Z3950</token> and
5291 <token>PROTO_SR</token>), an attribute set reference, and
5292 finally a null-terminated string holding the query string.
5295 If the parse went well, <function>p_query_rpn()</function> returns a
5296 pointer to a <literal>Z_RPNQuery</literal> structure which can be
5297 placed directly into a <literal>Z_SearchRequest</literal>.
5298 If parsing failed, due to syntax error, a NULL pointer is returned.
5301 The <literal>p_query_attset</literal> specifies which attribute set
5302 to use if the query doesn't specify one by the
5303 <literal>@attrset</literal> operator.
5304 The <literal>p_query_attset</literal> returns 0 if the argument is a
5305 valid attribute set specifier; otherwise the function returns -1.
5308 The grammar of the PQF is as follows:
5311 query ::= top-set query-struct.
5313 top-set ::= [ '@attrset' string ]
5315 query-struct ::= attr-spec | simple | complex | '@term' term-type query
5317 attr-spec ::= '@attr' [ string ] string query-struct
5319 complex ::= operator query-struct query-struct.
5321 operator ::= '@and' | '@or' | '@not' | '@prox' proximity.
5323 simple ::= result-set | term.
5325 result-set ::= '@set' string.
5329 proximity ::= exclusion distance ordered relation which-code unit-code.
5331 exclusion ::= '1' | '0' | 'void'.
5333 distance ::= integer.
5335 ordered ::= '1' | '0'.
5337 relation ::= integer.
5339 which-code ::= 'known' | 'private' | integer.
5341 unit-code ::= integer.
5343 term-type ::= 'general' | 'numeric' | 'string' | 'oid' | 'datetime' | 'null'.
5346 You will note that the syntax above is a fairly faithful
5347 representation of RPN, except for the Attribute, which has been
5348 moved a step away from the term, allowing you to associate one or more
5349 attributes with an entire query structure. The parser will
5350 automatically apply the given attributes to each term as required.
5353 The @attr operator is followed by an attribute specification
5354 (<literal>attr-spec</literal> above). The specification consists
5355 of an optional attribute set, an attribute type-value pair and
5356 a sub-query. The attribute type-value pair is packed in one string:
5357 an attribute type, an equals sign, and an attribute value, like this:
5358 <literal>@attr 1=1003</literal>.
5359 The type is always an integer, but the value may be either an
5360 integer or a string (if it doesn't start with a digit character).
5361 A string attribute-value is encoded as a Type-1 "complex"
5362 attribute with the list of values containing the single string
5363 specified, and including no semantic indicators.
5366 Version 3 of the Z39.50 specification defines various encoding of terms.
5367 Use <literal>@term </literal> <replaceable>type</replaceable>
5368 <replaceable>string</replaceable>,
5369 where type is one of: <literal>general</literal>,
5370 <literal>numeric</literal> or <literal>string</literal>
5371 (for InternationalString).
5372 If no term type has been given, the <literal>general</literal> form
5373 is used. This is the only encoding allowed in both versions 2 and 3
5374 of the Z39.50 standard.
5376 <sect3 id="PQF-prox">
5377 <title>Using Proximity Operators with PQF</title>
5380 This is an advanced topic, describing how to construct
5381 queries that make very specific requirements on the
5382 relative location of their operands.
5383 You may wish to skip this section and go straight to
5384 <link linkend="pqf-examples">the example PQF queries</link>.
5389 Most Z39.50 servers do not support proximity searching, or
5390 support only a small subset of the full functionality that
5391 can be expressed using the PQF proximity operator. Be
5392 aware that the ability to <emphasis>express</emphasis> a
5393 query in PQF is no guarantee that any given server will
5394 be able to <emphasis>execute</emphasis> it.
5400 The proximity operator <literal>@prox</literal> is a special
5401 and more restrictive version of the conjunction operator
5402 <literal>@and</literal>. Its semantics are described in
5403 section 3.7.2 (Proximity) of Z39.50 the standard itself, which
5404 can be read on-line at
5405 <ulink url="&url.z39.50.proximity;"/>
5408 In PQF, the proximity operation is represented by a sequence
5411 @prox <replaceable>exclusion</replaceable> <replaceable>distance</replaceable> <replaceable>ordered</replaceable> <replaceable>relation</replaceable> <replaceable>which-code</replaceable> <replaceable>unit-code</replaceable>
5413 in which the meanings of the parameters are as described in
5414 the standard, and they can take the following values:
5417 <formalpara><title>exclusion</title>
5419 0 = false (i.e. the proximity condition specified by the
5420 remaining parameters must be satisfied) or
5421 1 = true (the proximity condition specified by the
5422 remaining parameters must <emphasis>not</emphasis> be
5428 <formalpara><title>distance</title><para>
5429 An integer specifying the difference between the locations
5430 of the operands: e.g. two adjacent words would have
5431 distance=1 since their locations differ by one unit.
5433 </formalpara></listitem>
5435 <formalpara><title>ordered</title><para>
5436 1 = ordered (the operands must occur in the order the
5437 query specifies them) or
5438 0 = unordered (they may appear in either order).
5443 <formalpara><title>relation</title><para>
5444 Recognised values are
5446 2 (lessThanOrEqual),
5448 4 (greaterThanOrEqual),
5455 <formalpara><title>which-code</title><para>
5456 <literal>known</literal>
5458 <literal>k</literal>
5459 (the unit-code parameter is taken from the well-known list
5460 of alternatives described below) or
5461 <literal>private</literal>
5463 <literal>p</literal>
5464 (the unit-code parameter has semantics specific to an
5465 out-of-band agreement such as a profile).
5470 <formalpara><title>unit-code</title><para>
5471 If the which-code parameter is <literal>known</literal>
5472 then the recognised values are
5482 10 (elementType) and
5484 If which-code is <literal>private</literal> then the
5485 acceptable values are determined by the profile.
5490 (The numeric values of the relation and well-known unit-code
5491 parameters are taken straight from
5492 <ulink url="&url.z39.50.proximity.asn1;"
5493 >the ASN.1</ulink> of the proximity structure in the standard.)
5496 <sect3 id="pqf-examples">
5497 <title>PQF queries</title>
5498 <example id="example.pqf.simple.terms">
5499 <title>PQF queries using simple terms</title>
5508 <example id="pqf.example.pqf.boolean.operators">
5509 <title>PQF boolean operators</title>
5512 @or "dylan" "zimmerman"
5514 @and @or dylan zimmerman when
5516 @and when @or dylan zimmerman
5520 <example id="example.pqf.result.sets">
5521 <title>PQF references to result sets</title>
5526 @and @set seta @set setb
5530 <example id="example.pqf.attributes">
5531 <title>Attributes for terms</title>
5536 @attr 1=4 @attr 4=1 "self portrait"
5538 @attrset exp1 @attr 1=1 CategoryList
5540 @attr gils 1=2008 Copenhagen
5542 @attr 1=/book/title computer
5546 <example id="example.pqf.proximity">
5547 <title>PQF Proximity queries</title>
5550 @prox 0 3 1 2 k 2 dylan zimmerman
5552 Here the parameters 0, 3, 1, 2, k and 2 represent exclusion,
5553 distance, ordered, relation, which-code and unit-code, in that
5557 <para>exclusion = 0: the proximity condition must hold</para>
5560 <para>distance = 3: the terms must be three units apart</para>
5564 ordered = 1: they must occur in the order they are specified
5569 relation = 2: lessThanOrEqual (to the distance of 3 units)
5574 which-code is "known", so the standard unit-codes are used
5578 <para>unit-code = 2: word.</para>
5581 So the whole proximity query means that the words
5582 <literal>dylan</literal> and <literal>zimmerman</literal> must
5583 both occur in the record, in that order, differing in position
5584 by three or fewer words (i.e. with two or fewer words between
5585 them.) The query would find "Bob Dylan, aka. Robert
5586 Zimmerman", but not "Bob Dylan, born as Robert Zimmerman"
5587 since the distance in this case is four.
5590 <example id="example.pqf.search.term.type">
5591 <title>PQF specification of search term type</title>
5594 @term string "a UTF-8 string, maybe?"
5598 <example id="example.pqf.mixed.queries">
5599 <title>PQF mixed queries</title>
5602 @or @and bob dylan @set Result-1
5604 @attr 4=1 @and @attr 1=1 "bob dylan" @attr 1=4 "slow train coming"
5606 @and @attr 2=4 @attr gils 1=2038 -114 @attr 2=2 @attr gils 1=2039 -109
5608 The last of these examples is a spatial search: in
5609 <ulink url="http://www.gils.net/prof_v2.html#sec_7_4"
5610 >the GILS attribute set</ulink>,
5612 2038 indicates West Bounding Coordinate and
5613 2030 indicates East Bounding Coordinate,
5614 so the query is for areas extending from -114 degrees longitude
5615 to no more than -109 degrees longitude.
5620 <sect2 id="CCL"><title>CCL</title>
5622 Not all users enjoy typing in prefix query structures and numerical
5623 attribute values, even in a minimalistic test client. In the library
5624 world, the more intuitive Common Command Language - CCL (ISO 8777)
5625 has enjoyed some popularity - especially before the widespread
5626 availability of graphical interfaces. It is still useful in
5627 applications where you for some reason or other need to provide a
5628 symbolic language for expressing boolean query structures.
5630 <sect3 id="ccl.syntax">
5631 <title>CCL Syntax</title>
5633 The CCL parser obeys the following grammar for the FIND argument.
5634 The syntax is annotated using lines prefixed by
5635 <literal>--</literal>.
5638 CCL-Find ::= CCL-Find Op Elements
5641 Op ::= "and" | "or" | "not"
5642 -- The above means that Elements are separated by boolean operators.
5644 Elements ::= '(' CCL-Find ')'
5647 | Qualifiers Relation Terms
5648 | Qualifiers Relation '(' CCL-Find ')'
5649 | Qualifiers '=' string '-' string
5650 -- Elements is either a recursive definition, a result set reference, a
5651 -- list of terms, qualifiers followed by terms, qualifiers followed
5652 -- by a recursive definition or qualifiers in a range (lower - upper).
5654 Set ::= 'set' = string
5655 -- Reference to a result set
5657 Terms ::= Terms Prox Term
5659 -- Proximity of terms.
5661 Term ::= Term string
5663 -- This basically means that a term may include a blank
5665 Qualifiers ::= Qualifiers ',' string
5667 -- Qualifiers is a list of strings separated by comma
5669 Relation ::= '=' | '>=' | '<=' | '<>' | '>' | '<'
5670 -- Relational operators. This really doesn't follow the ISO8777
5674 -- Proximity operator
5677 <example id="example.ccl.queries">
5678 <title>CCL queries</title>
5680 The following queries are all valid:
5691 (dylan and bob) or set=1
5700 Assuming that the qualifiers <literal>ti</literal>
5701 and <literal>au</literal>
5702 and <literal>date</literal> are defined, we may use:
5707 au=(bob dylan and slow train coming)
5709 date>1980 and (ti=((self portrait)))
5713 <sect3 id="ccl.qualifiers">
5714 <title>CCL Qualifiers</title>
5716 Qualifiers are used to direct the search to a particular searchable
5717 index, such as title (ti) and author indexes (au). The CCL standard
5718 itself doesn't specify a particular set of qualifiers, but it does
5719 suggest a few short-hand notations. You can customize the CCL parser
5720 to support a particular set of qualifiers to reflect the current target
5721 profile. Traditionally, a qualifier would map to a particular
5722 use-attribute within the BIB-1 attribute set. It is also
5723 possible to set other attributes, such as the structure
5727 A CCL profile is a set of predefined CCL qualifiers that may be
5728 read from a file or set in the CCL API.
5729 The YAZ client reads its CCL qualifiers from a file named
5730 <filename>default.bib</filename>. There are four types of
5731 lines in a CCL profile: qualifier specification,
5732 qualifier alias, comments and directives.
5734 <sect4 id="ccl.qualifier.specification">
5735 <title>Qualifier specification</title>
5737 A qualifier specification is of the form:
5740 <replaceable>qualifier-name</replaceable>
5741 [<replaceable>attributeset</replaceable><literal>,</literal>]<replaceable>type</replaceable><literal>=</literal><replaceable>val</replaceable>
5742 [<replaceable>attributeset</replaceable><literal>,</literal>]<replaceable>type</replaceable><literal>=</literal><replaceable>val</replaceable> ...
5745 where <replaceable>qualifier-name</replaceable> is the name of the
5746 qualifier to be used (e.g. <literal>ti</literal>),
5747 <replaceable>type</replaceable> is attribute type in the attribute
5748 set (Bib-1 is used if no attribute set is given) and
5749 <replaceable>val</replaceable> is attribute value.
5750 The <replaceable>type</replaceable> can be specified as an
5751 integer, or as a single-letter:
5752 <literal>u</literal> for use,
5753 <literal>r</literal> for relation, <literal>p</literal> for position,
5754 <literal>s</literal> for structure,<literal>t</literal> for truncation,
5755 or <literal>c</literal> for completeness.
5756 The attributes for the special qualifier name <literal>term</literal>
5757 are used when no CCL qualifier is given in a query.
5758 <table id="ccl.common.bib1.attributes">
5759 <title>Common Bib-1 attributes</title>
5761 <colspec colwidth="2*" colname="type"></colspec>
5762 <colspec colwidth="9*" colname="description"></colspec>
5766 <entry>Description</entry>
5771 <entry><literal>u=</literal><replaceable>value</replaceable></entry>
5773 Use attribute (1). Common use attributes are
5774 1 Personal-name, 4 Title, 7 ISBN, 8 ISSN, 30 Date,
5775 62 Subject, 1003 Author, 1016 Any. Specify value
5780 <entry><literal>r=</literal><replaceable>value</replaceable></entry>
5782 Relation attribute (2). Common values are
5783 1 <, 2 <=, 3 =, 4 >=, 5 >, 6 <>,
5784 100 phonetic, 101 stem, 102 relevance, 103 always matches.
5788 <entry><literal>p=</literal><replaceable>value</replaceable></entry>
5790 Position attribute (3). Values: 1 first in field, 2
5791 first in any subfield, 3 any position in field.
5795 <entry><literal>s=</literal><replaceable>value</replaceable></entry>
5797 Structure attribute (4). Values: 1 phrase, 2 word,
5798 3 key, 4 year, 5 date, 6 word list, 100 date (un),
5799 101 name (norm), 102 name (un), 103 structure, 104 urx,
5800 105 free-form-text, 106 document-text, 107 local-number,
5801 108 string, 109 numeric string.
5805 <entry><literal>t=</literal><replaceable>value</replaceable></entry>
5807 Truncation attribute (5). Values: 1 right, 2 left,
5808 3 left and right, 100 none, 101 process #, 102 regular-1,
5809 103 regular-2, 104 CCL.
5813 <entry><literal>c=</literal><replaceable>value</replaceable></entry>
5815 Completeness attribute (6). Values: 1 incomplete subfield,
5816 2 complete subfield, 3 complete field.
5824 Refer to <xref linkend="bib1"/> or the complete
5825 <ulink url="&url.z39.50.attset.bib1;">list of Bib-1 attributes</ulink>
5828 It is also possible to specify non-numeric attribute values,
5829 which are used in combination with certain types.
5830 The special combinations are:
5831 <table id="ccl.special.attribute.combos">
5832 <title>Special attribute combos</title>
5834 <colspec colwidth="2*" colname="name"></colspec>
5835 <colspec colwidth="9*" colname="description"></colspec>
5839 <entry>Description</entry>
5844 <entry><literal>s=pw</literal></entry>
5846 The structure is set to either word or phrase depending
5847 on the number of tokens in a term (phrase-word).
5851 <entry><literal>s=al</literal></entry>
5853 Each token in the term is ANDed (and-list).
5854 This does not set the structure at all.
5857 <row><entry><literal>s=ol</literal></entry>
5859 Each token in the term is ORed (or-list).
5860 This does not set the structure at all.
5863 <row><entry><literal>s=ag</literal></entry>
5865 Tokens that appears as phrases (with blank in them) gets
5866 structure phrase attached (4=1). Tokens that appear to be words
5867 gets structure word attached (4=2). Phrases and words are
5868 ANDed. This is a variant of s=al and s=pw, with the main
5869 difference that words are not split (with operator AND)
5870 but instead kept in one RPN token. This facility appeared
5874 <row><entry><literal>s=sl</literal></entry>
5876 Tokens are split into sub-phrases of all combinations - in order.
5877 This facility appeared in YAZ 5.14.0.
5880 <row><entry><literal>r=o</literal></entry>
5882 Allows ranges and the operators greater-than, less-than, ...
5884 This sets Bib-1 relation attribute accordingly (relation
5885 ordered). A query construct is only treated as a range if
5886 dash is used and that is surrounded by white-space. So
5887 <literal>-1980</literal> is treated as term
5888 <literal>"-1980"</literal> not <literal><= 1980</literal>.
5889 If <literal>- 1980</literal> is used, however, that is
5893 <row><entry><literal>r=r</literal></entry>
5895 Similar to <literal>r=o</literal> but assumes that terms
5896 are non-negative (not prefixed with <literal>-</literal>).
5897 Thus, a dash will always be treated as a range.
5898 The construct <literal>1980-1990</literal> is
5899 treated as a range with <literal>r=r</literal> but as a
5900 single term <literal>"1980-1990"</literal> with
5901 <literal>r=o</literal>. The special attribute
5902 <literal>r=r</literal> is available in YAZ 2.0.24 or later.
5905 <row><entry><literal>r=omiteq</literal></entry>
5907 This will omit relation=equals (@attr 2=3) when r=o / r=r
5908 is used. This is useful for servers that somehow break
5909 when an explicit relation=equals is used. Omitting the
5910 relation is usually safe because "equals" is the default
5911 behavior. This tweak was added in YAZ version 5.1.2.
5914 <row><entry><literal>t=l</literal></entry>
5916 Allows term to be left-truncated.
5917 If term is of the form <literal>?x</literal>, the resulting
5918 Type-1 term is <literal>x</literal> and truncation is left.
5921 <row><entry><literal>t=r</literal></entry>
5923 Allows term to be right-truncated.
5924 If term is of the form <literal>x?</literal>, the resulting
5925 Type-1 term is <literal>x</literal> and truncation is right.
5928 <row><entry><literal>t=n</literal></entry>
5930 If term is does not include <literal>?</literal>, the
5931 truncation attribute is set to none (100).
5934 <row><entry><literal>t=b</literal></entry>
5936 Allows term to be both left-and-right truncated.
5937 If term is of the form <literal>?x?</literal>, the
5938 resulting term is <literal>x</literal> and trunctation is
5939 set to both left and right.
5942 <row><entry><literal>t=x</literal></entry>
5944 Allows masking anywhere in a term, thus fully supporting
5945 # (mask one character) and ? (zero or more of any).
5946 If masking is used, truncation is set to 102 (regexp-1 in term)
5947 and the term is converted accordingly to a regular expression.
5950 <row><entry><literal>t=z</literal></entry>
5952 Allows masking anywhere in a term, thus fully supporting
5953 # (mask one character) and ? (zero or more of any).
5954 If masking is used, truncation is set to 104 (Z39.58 in term)
5955 and the term is converted accordingly to Z39.58 masking term -
5956 actually the same truncation as CCL itself.
5963 <example id="example.ccl.profile">
5964 <title>CCL profile</title>
5966 Consider the following definition:
5976 <literal>ti</literal> and <literal>au</literal> both set
5977 structure attribute to phrase (s=1).
5978 <literal>ti</literal>
5979 sets the use-attribute to 4. <literal>au</literal> sets the
5981 When no qualifiers are used in the query, the structure-attribute is
5982 set to free-form-text (105) (rule for <literal>term</literal>).
5983 The <literal>date</literal> sets the relation attribute to
5984 the relation used in the CCL query and sets the use attribute
5988 You can combine attributes. To Search for "ranked title" you
5991 ti,ranked=knuth computer
5993 which will set relation=ranked, use=title, structure=phrase.
6000 is a valid query. But
6008 <sect4 id="ccl.qualifier.alias">
6009 <title>Qualifier alias</title>
6011 A qualifier alias is of the form:
6014 <replaceable>q</replaceable>
6015 <replaceable>q1</replaceable> <replaceable>q2</replaceable> ..
6018 which declares <replaceable>q</replaceable> to
6019 be an alias for <replaceable>q1</replaceable>,
6020 <replaceable>q2</replaceable>... such that the CCL
6021 query <replaceable>q=x</replaceable> is equivalent to
6022 <replaceable>q1=x or q2=x or ...</replaceable>.
6025 <sect4 id="ccl.comments">
6026 <title>Comments</title>
6028 Lines with white space or lines that begin with
6029 character <literal>#</literal> are treated as comments.
6032 <sect4 id="ccl.directives">
6033 <title>Directives</title>
6035 Directive specifications takes the form
6037 <para><literal>@</literal><replaceable>directive</replaceable> <replaceable>value</replaceable>
6039 <table id="ccl.directives.table">
6040 <title>CCL directives</title>
6042 <colspec colwidth="2*" colname="name"></colspec>
6043 <colspec colwidth="8*" colname="description"></colspec>
6044 <colspec colwidth="1*" colname="default"></colspec>
6048 <entry>Description</entry>
6049 <entry>Default</entry>
6054 <entry>truncation</entry>
6055 <entry>Truncation character</entry>
6056 <entry><literal>?</literal></entry>
6060 <entry>Masking character. Requires YAZ 4.2.58 or later</entry>
6061 <entry><literal>#</literal></entry>
6064 <entry>field</entry>
6065 <entry>Specifies how multiple fields are to be
6066 combined. There are two modes: <literal>or</literal>:
6067 multiple qualifier fields are ORed,
6068 <literal>merge</literal>: attributes for the qualifier
6069 fields are merged and assigned to one term.
6071 <entry><literal>merge</literal></entry>
6075 <entry>Specifies if CCL operators and qualifiers should be
6076 compared with case sensitivity or not. Specify 1 for
6077 case sensitive; 0 for case insensitive.</entry>
6078 <entry><literal>1</literal></entry>
6082 <entry>Specifies token for CCL operator AND.</entry>
6083 <entry><literal>and</literal></entry>
6087 <entry>Specifies token for CCL operator OR.</entry>
6088 <entry><literal>or</literal></entry>
6092 <entry>Specifies token for CCL operator NOT.</entry>
6093 <entry><literal>not</literal></entry>
6097 <entry>Specifies token for CCL operator SET.</entry>
6098 <entry><literal>set</literal></entry>
6105 <sect3 id="ccl.api">
6106 <title>CCL API</title>
6108 All public definitions can be found in the header file
6109 <filename>ccl.h</filename>. A profile identifier is of type
6110 <literal>CCL_bibset</literal>. A profile must be created with the call
6111 to the function <function>ccl_qual_mk</function> which returns a profile
6112 handle of type <literal>CCL_bibset</literal>.
6115 To read a file containing qualifier definitions the function
6116 <function>ccl_qual_file</function> may be convenient. This function
6117 takes an already opened <literal>FILE</literal> handle pointer as
6118 argument along with a <literal>CCL_bibset</literal> handle.
6121 To parse a simple string with a FIND query use the function
6124 struct ccl_rpn_node *ccl_find_str(CCL_bibset bibset, const char *str,
6125 int *error, int *pos);
6128 which takes the CCL profile (<literal>bibset</literal>) and query
6129 (<literal>str</literal>) as input. Upon successful completion the RPN
6130 tree is returned. If an error occurs, such as a syntax error, the integer
6131 pointed to by <literal>error</literal> holds the error code and
6132 <literal>pos</literal> holds the offset inside query string in which
6136 An English representation of the error may be obtained by calling
6137 the <literal>ccl_err_msg</literal> function. The error codes are
6138 listed in <filename>ccl.h</filename>.
6141 To convert the CCL RPN tree (type
6142 <literal>struct ccl_rpn_node *</literal>)
6143 to the Z_RPNQuery of YAZ the function <function>ccl_rpn_query</function>
6144 must be used. This function which is part of YAZ is implemented in
6145 <filename>yaz-ccl.c</filename>.
6146 After calling this function the CCL RPN tree is probably no longer
6147 needed. The <literal>ccl_rpn_delete</literal> destroys the CCL RPN tree.
6150 A CCL profile may be destroyed by calling the
6151 <function>ccl_qual_rm</function> function.
6154 The token names for the CCL operators may be changed by setting the
6155 globals (all type <literal>char *</literal>)
6156 <literal>ccl_token_and</literal>, <literal>ccl_token_or</literal>,
6157 <literal>ccl_token_not</literal> and <literal>ccl_token_set</literal>.
6158 An operator may have aliases, i.e. there may be more than one name for
6159 the operator. To do this, separate each alias with a space character.
6166 <ulink url="&url.cql;">CQL</ulink>
6167 - Common Query Language - was defined for the
6168 <ulink url="&url.sru;">SRU</ulink> protocol.
6169 In many ways CQL has a similar syntax to CCL.
6170 The objective of CQL is different. Where CCL aims to be
6171 an end-user language, CQL is <emphasis>the</emphasis> protocol
6172 query language for SRU.
6176 If you are new to CQL, read the
6177 <ulink url="&url.cql.intro;">Gentle Introduction</ulink>.
6181 The CQL parser in &yaz; provides the following:
6185 It parses and validates a CQL query.
6190 It generates a C structure that allows you to convert
6191 a CQL query to some other query language, such as SQL.
6196 The parser converts a valid CQL query to PQF, thus providing a
6197 way to use CQL for both SRU servers and Z39.50 targets at the
6203 The parser converts CQL to XCQL.
6204 XCQL is an XML representation of CQL.
6205 XCQL is part of the SRU specification. However, since SRU
6206 supports CQL only, we don't expect XCQL to be widely used.
6207 Furthermore, CQL has the advantage over XCQL that it is
6213 <sect3 id="cql.parsing">
6214 <title>CQL parsing</title>
6216 A CQL parser is represented by the <literal>CQL_parser</literal>
6217 handle. Its contents should be considered &yaz; internal (private).
6219 #include <yaz/cql.h>
6221 typedef struct cql_parser *CQL_parser;
6223 CQL_parser cql_parser_create(void);
6224 void cql_parser_destroy(CQL_parser cp);
6226 A parser is created by <function>cql_parser_create</function> and
6227 is destroyed by <function>cql_parser_destroy</function>.
6230 To parse a CQL query string, the following function
6233 int cql_parser_string(CQL_parser cp, const char *str);
6235 A CQL query is parsed by the <function>cql_parser_string</function>
6236 which takes a query <parameter>str</parameter>.
6237 If the query was valid (no syntax errors), then zero is returned;
6238 otherwise -1 is returned to indicate a syntax error.
6242 int cql_parser_stream(CQL_parser cp,
6243 int (*getbyte)(void *client_data),
6244 void (*ungetbyte)(int b, void *client_data),
6247 int cql_parser_stdio(CQL_parser cp, FILE *f);
6249 The functions <function>cql_parser_stream</function> and
6250 <function>cql_parser_stdio</function> parse a CQL query
6251 - just like <function>cql_parser_string</function>.
6252 The only difference is that the CQL query can be
6253 fed to the parser in different ways.
6254 The <function>cql_parser_stream</function> uses a generic
6255 byte stream as input. The <function>cql_parser_stdio</function>
6256 uses a <literal>FILE</literal> handle which is opened for reading.
6259 <sect3 id="cql.tree">
6260 <title>CQL tree</title>
6262 If the query string is valid, the CQL parser
6263 generates a tree representing the structure of the
6268 struct cql_node *cql_parser_result(CQL_parser cp);
6270 <function>cql_parser_result</function> returns
6271 a pointer to the root node of the resulting tree.
6274 Each node in a CQL tree is represented by a
6275 <literal>struct cql_node</literal>.
6276 It is defined as follows:
6278 #define CQL_NODE_ST 1
6279 #define CQL_NODE_BOOL 2
6280 #define CQL_NODE_SORT 3
6290 struct cql_node *modifiers;
6294 struct cql_node *left;
6295 struct cql_node *right;
6296 struct cql_node *modifiers;
6300 struct cql_node *next;
6301 struct cql_node *modifiers;
6302 struct cql_node *search;
6307 There are three node types: search term (ST), boolean (BOOL)
6309 A modifier is treated as a search term too.
6312 The search term node has five members:
6316 <literal>index</literal>: index for search term.
6317 If an index is unspecified for a search term,
6318 <literal>index</literal> will be NULL.
6323 <literal>index_uri</literal>: index URi for search term
6324 or NULL if none could be resolved for the index.
6329 <literal>term</literal>: the search term itself.
6334 <literal>relation</literal>: relation for search term.
6339 <literal>relation_uri</literal>: relation URI for search term.
6344 <literal>modifiers</literal>: relation modifiers for search
6345 term. The <literal>modifiers</literal> list itself of cql_nodes
6346 each of type <literal>ST</literal>.
6352 The boolean node represents <literal>and</literal>,
6353 <literal>or</literal>, <literal>not</literal> +
6358 <literal>left</literal> and <literal>right</literal>: left
6359 - and right operand respectively.
6364 <literal>modifiers</literal>: proximity arguments.
6370 The sort node represents both the SORTBY clause.
6373 <sect3 id="cql.to.pqf">
6374 <title>CQL to PQF conversion</title>
6376 Conversion to PQF (and Z39.50 RPN) is tricky by the fact
6377 that the resulting RPN depends on the Z39.50 target
6378 capabilities (combinations of supported attributes).
6379 In addition, the CQL and SRU operates on index prefixes
6380 (URI or strings), whereas the RPN uses Object Identifiers
6384 The CQL library of &yaz; defines a <literal>cql_transform_t</literal>
6385 type. It represents a particular mapping between CQL and RPN.
6386 This handle is created and destroyed by the functions:
6388 cql_transform_t cql_transform_open_FILE (FILE *f);
6389 cql_transform_t cql_transform_open_fname(const char *fname);
6390 void cql_transform_close(cql_transform_t ct);
6392 The first two functions create a tranformation handle from
6393 either an already open FILE or from a filename respectively.
6396 The handle is destroyed by <function>cql_transform_close</function>
6397 in which case no further reference of the handle is allowed.
6400 When a <literal>cql_transform_t</literal> handle has been created
6401 you can convert to RPN.
6403 int cql_transform_buf(cql_transform_t ct,
6404 struct cql_node *cn, char *out, int max);
6406 This function converts the CQL tree <literal>cn</literal>
6407 using handle <literal>ct</literal>.
6408 For the resulting PQF, you supply a buffer <literal>out</literal>
6409 which must be able to hold at at least <literal>max</literal>
6413 If conversion failed, <function>cql_transform_buf</function>
6414 returns a non-zero SRU error code; otherwise zero is returned
6415 (conversion successful). The meanings of the numeric error
6416 codes are listed in the SRU specification somewhere (no
6417 direct link anymore).
6420 If conversion fails, more information can be obtained by calling
6422 int cql_transform_error(cql_transform_t ct, char **addinfop);
6424 This function returns the most recently returned numeric
6425 error-code and sets the string-pointer at
6426 <literal>*addinfop</literal> to point to a string containing
6427 additional information about the error that occurred: for
6428 example, if the error code is 15 ("Illegal or unsupported context
6429 set"), the additional information is the name of the requested
6430 context set that was not recognised.
6433 The SRU error-codes may be translated into brief human-readable
6434 error messages using
6436 const char *cql_strerror(int code);
6440 If you wish to be able to produce a PQF result in a different
6441 way, there are two alternatives.
6443 void cql_transform_pr(cql_transform_t ct,
6444 struct cql_node *cn,
6445 void (*pr)(const char *buf, void *client_data),
6448 int cql_transform_FILE(cql_transform_t ct,
6449 struct cql_node *cn, FILE *f);
6451 The former function produces output to a user-defined
6452 output stream. The latter writes the result to an already
6453 open <literal>FILE</literal>.
6456 <sect3 id="cql.to.rpn">
6457 <title>Specification of CQL to RPN mappings</title>
6459 The file supplied to functions
6460 <function>cql_transform_open_FILE</function>,
6461 <function>cql_transform_open_fname</function> follows
6462 a structure found in many Unix utilities.
6463 It consists of mapping specifications - one per line.
6464 Lines starting with <literal>#</literal> are ignored (comments).
6467 Each line is of the form
6469 <replaceable>CQL pattern</replaceable><literal> = </literal> <replaceable> RPN equivalent</replaceable>
6473 An RPN pattern is a simple attribute list. Each attribute pair
6476 [<replaceable>set</replaceable>] <replaceable>type</replaceable><literal>=</literal><replaceable>value</replaceable>
6478 The attribute <replaceable>set</replaceable> is optional.
6479 The <replaceable>type</replaceable> is the attribute type,
6480 <replaceable>value</replaceable> the attribute value.
6483 The character <literal>*</literal> (asterisk) has special meaning
6484 when used in the RPN pattern.
6485 Each occurrence of <literal>*</literal> is substituted with the
6486 CQL matching name (index, relation, qualifier etc).
6487 This facility can be used to copy a CQL name verbatim to the RPN result.
6490 The following CQL patterns are recognized:
6494 <literal>index.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
6498 This pattern is invoked when a CQL index, such as
6499 dc.title is converted. <replaceable>set</replaceable>
6500 and <replaceable>name</replaceable> are the context set and index
6502 Typically, the RPN specifies an equivalent use attribute.
6505 For terms not bound by an index, the pattern
6506 <literal>index.cql.serverChoice</literal> is used.
6507 Here, the prefix <literal>cql</literal> is defined as
6508 <literal>http://www.loc.gov/zing/cql/cql-indexes/v1.0/</literal>.
6509 If this pattern is not defined, the mapping will fail.
6513 <literal>index.</literal><replaceable>set</replaceable><literal>.*</literal>
6514 is used when no other index pattern is matched.
6520 <literal>qualifier.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
6525 For backwards compatibility, this is recognised as a synonym of
6526 <literal>index.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
6532 <literal>relation.</literal><replaceable>relation</replaceable>
6536 This pattern specifies how a CQL relation is mapped to RPN.
6537 The <replaceable>pattern</replaceable> is name of relation
6538 operator. Since <literal>=</literal> is used as
6539 separator between CQL pattern and RPN, CQL relations
6540 including <literal>=</literal> cannot be
6541 used directly. To avoid a conflict, the names
6542 <literal>ge</literal>,
6543 <literal>eq</literal>,
6544 <literal>le</literal>,
6545 must be used for CQL operators, greater-than-or-equal,
6546 equal, less-than-or-equal respectively.
6547 The RPN pattern is supposed to include a relation attribute.
6550 For terms not bound by a relation, the pattern
6551 <literal>relation.scr</literal> is used. If the pattern
6552 is not defined, the mapping will fail.
6555 The special pattern, <literal>relation.*</literal> is used
6556 when no other relation pattern is matched.
6562 <literal>relationModifier.</literal><replaceable>mod</replaceable>
6566 This pattern specifies how a CQL relation modifier is mapped to RPN.
6567 The RPN pattern is usually a relation attribute.
6573 <literal>structure.</literal><replaceable>type</replaceable>
6577 This pattern specifies how a CQL structure is mapped to RPN.
6578 Note that this CQL pattern is somewhat similar to
6579 CQL pattern <literal>relation</literal>.
6580 The <replaceable>type</replaceable> is a CQL relation.
6583 The pattern, <literal>structure.*</literal> is used
6584 when no other structure pattern is matched.
6585 Usually, the RPN equivalent specifies a structure attribute.
6591 <literal>position.</literal><replaceable>type</replaceable>
6595 This pattern specifies how the anchor (position) of
6596 CQL is mapped to RPN.
6597 The <replaceable>type</replaceable> is one
6598 of <literal>first</literal>, <literal>any</literal>,
6599 <literal>last</literal>, <literal>firstAndLast</literal>.
6602 The pattern, <literal>position.*</literal> is used
6603 when no other position pattern is matched.
6609 <literal>set.</literal><replaceable>prefix</replaceable>
6613 This specification defines a CQL context set for a given prefix.
6614 The value on the right hand side is the URI for the set -
6615 <emphasis>not</emphasis> RPN. All prefixes used in
6616 index patterns must be defined this way.
6622 <literal>set</literal>
6626 This specification defines a default CQL context set for index names.
6627 The value on the right hand side is the URI for the set.
6633 <example id="example.cql.to.rpn.mapping">
6634 <title>CQL to RPN mapping file</title>
6636 This simple file defines two context sets, three indexes and three
6637 relations, a position pattern and a default structure.
6639 <programlisting><![CDATA[
6640 set.cql = http://www.loc.gov/zing/cql/context-sets/cql/v1.1/
6641 set.dc = http://www.loc.gov/zing/cql/dc-indexes/v1.0/
6643 index.cql.serverChoice = 1=1016
6644 index.dc.title = 1=4
6645 index.dc.subject = 1=21
6651 position.any = 3=3 6=1
6657 With the mappings above, the CQL query
6661 is converted to the PQF:
6663 @attr 1=1016 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "computer"
6665 by rules <literal>index.cql.serverChoice</literal>,
6666 <literal>relation.scr</literal>, <literal>structure.*</literal>,
6667 <literal>position.any</literal>.
6674 is rejected, since <literal>position.right</literal> is
6680 >my = "http://www.loc.gov/zing/cql/dc-indexes/v1.0/" my.title = x
6684 @attr 1=4 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "x"
6688 <example id="example.cql.to.rpn.string">
6689 <title>CQL to RPN string attributes</title>
6691 In this example we allow any index to be passed to RPN as
6694 <programlisting><![CDATA[
6695 # Identifiers for prefixes used in this file. (index.*)
6696 set.cql = info:srw/cql-context-set/1/cql-v1.1
6697 set.rpn = http://bogus/rpn
6698 set = http://bogus/rpn
6700 # The default index when none is specified by the query
6701 index.cql.serverChoice = 1=any
6710 The <literal>http://bogus/rpn</literal> context set is also the default
6711 so we can make queries such as
6715 which is converted to
6717 @attr 2=3 @attr 4=1 @attr 3=3 @attr 1=title "a"
6721 <example id="example.cql.to.rpn.bathprofile">
6722 <title>CQL to RPN using Bath Profile</title>
6724 The file <filename>etc/pqf.properties</filename> has mappings from
6725 the Bath Profile and Dublin Core to RPN.
6726 If YAZ is installed as a package it's usually located
6727 in <filename>/usr/share/yaz/etc</filename> and part of the
6728 development package, such as <literal>libyaz-dev</literal>.
6732 <sect3 id="cql.xcql">
6733 <title>CQL to XCQL conversion</title>
6735 Conversion from CQL to XCQL is trivial and does not
6736 require a mapping to be defined.
6737 There are three functions to choose from depending on the
6738 way you wish to store the resulting output (XML buffer
6741 int cql_to_xml_buf(struct cql_node *cn, char *out, int max);
6742 void cql_to_xml(struct cql_node *cn,
6743 void (*pr)(const char *buf, void *client_data),
6745 void cql_to_xml_stdio(struct cql_node *cn, FILE *f);
6747 Function <function>cql_to_xml_buf</function> converts
6748 to XCQL and stores the result in a user-supplied buffer of a given
6752 <function>cql_to_xml</function> writes the result in
6753 a user-defined output stream.
6754 <function>cql_to_xml_stdio</function> writes to a
6758 <sect3 id="rpn.to.cql">
6759 <title>PQF to CQL conversion</title>
6761 Conversion from PQF to CQL is offered by the two functions shown
6762 below. The former uses a generic stream for result. The latter
6763 puts result in a WRBUF (string container).
6765 #include <yaz/rpn2cql.h>
6767 int cql_transform_rpn2cql_stream(cql_transform_t ct,
6768 void (*pr)(const char *buf, void *client_data),
6772 int cql_transform_rpn2cql_wrbuf(cql_transform_t ct,
6776 The configuration is the same as used in CQL to PQF conversions.
6781 <sect1 id="tools.oid">
6782 <title>Object Identifiers</title>
6784 The basic YAZ representation of an OID is an array of integers,
6785 terminated with the value -1. This integer is of type
6786 <literal>Odr_oid</literal>.
6789 Fundamental OID operations and the type <literal>Odr_oid</literal>
6790 are defined in <filename>yaz/oid_util.h</filename>.
6793 An OID can either be declared as a automatic variable or it can
6794 be allocated using the memory utilities or ODR/NMEM. It's
6795 guaranteed that an OID can fit in <literal>OID_SIZE</literal> integers.
6797 <example id="tools.oid.bib1.1"><title>Create OID on stack</title>
6799 We can create an OID for the Bib-1 attribute set with:
6801 Odr_oid bib1[OID_SIZE];
6813 And OID may also be filled from a string-based representation using
6814 dots (.). This is achieved by the function
6816 int oid_dotstring_to_oid(const char *name, Odr_oid *oid);
6818 This functions returns 0 if name could be converted; -1 otherwise.
6820 <example id="tools.oid.bib1.2"><title>Using oid_oiddotstring_to_oid</title>
6822 We can fill the Bib-1 attribute set OID more easily with:
6824 Odr_oid bib1[OID_SIZE];
6825 oid_oiddotstring_to_oid("1.2.840.10003.3.1", bib1);
6830 We can also allocate an OID dynamically on an ODR stream with:
6832 Odr_oid *odr_getoidbystr(ODR o, const char *str);
6834 This creates an OID from a string-based representation using dots.
6835 This function take an &odr; stream as parameter. This stream is used to
6836 allocate memory for the data elements, which is released on a
6837 subsequent call to <function>odr_reset()</function> on that stream.
6839 <example id="tools.oid.bib1.3">
6840 <title>Using odr_getoidbystr</title>
6842 We can create an OID for the Bib-1 attribute set with:
6844 Odr_oid *bib1 = odr_getoidbystr(odr, "1.2.840.10003.3.1");
6851 char *oid_oid_to_dotstring(const Odr_oid *oid, char *oidbuf)
6853 does the reverse of <function>oid_oiddotstring_to_oid</function>. It
6854 converts an OID to the string-based representation using dots.
6855 The supplied char buffer <literal>oidbuf</literal> holds the resulting
6856 string and must be at least <literal>OID_STR_MAX</literal> in size.
6859 OIDs can be copied with <function>oid_oidcpy</function> which takes
6860 two OID lists as arguments. Alternatively, an OID copy can be allocated
6861 on an ODR stream with:
6863 Odr_oid *odr_oiddup(ODR odr, const Odr_oid *o);
6867 OIDs can be compared with <function>oid_oidcmp</function> which returns
6868 zero if the two OIDs provided are identical; non-zero otherwise.
6870 <sect2 id="tools.oid.database">
6871 <title>OID database</title>
6873 From YAZ version 3 and later, the oident system has been replaced
6874 by an OID database. OID database is a misnomer .. the old odient
6875 system was also a database.
6878 The OID database is really just a map between named Object Identifiers
6879 (string) and their OID raw equivalents. Most operations either
6880 convert from string to OID or other way around.
6883 Unfortunately, whenever we supply a string we must also specify the
6884 <emphasis>OID class</emphasis>. The class is necessary because some
6885 strings correspond to multiple OIDs. An example of such a string is
6886 <literal>Bib-1</literal> which may either be an attribute-set
6887 or a diagnostic-set.
6890 Applications using the YAZ database should include
6891 <filename>yaz/oid_db.h</filename>.
6894 A YAZ database handle is of type <literal>yaz_oid_db_t</literal>.
6895 Actually that's a pointer. You need not deal with that.
6896 YAZ has a built-in database which can be considered "constant" for
6898 We can get hold of that by using function <function>yaz_oid_std</function>.
6901 All functions with prefix <function>yaz_string_to_oid</function>
6902 converts from class + string to OID. We have variants of this
6903 operation due to different memory allocation strategies.
6906 All functions with prefix
6907 <function>yaz_oid_to_string</function> converts from OID to string
6910 <example id="tools.oid.bib1.4">
6911 <title>Create OID with YAZ DB</title>
6913 We can create an OID for the Bib-1 attribute set on the ODR stream
6917 yaz_string_to_oid_odr(yaz_oid_std(), CLASS_ATTSET, "Bib-1", odr);
6919 This is more complex than using <function>odr_getoidbystr</function>.
6920 You would only use <function>yaz_string_to_oid_odr</function> when the
6921 string (here Bib-1) is supplied by a user or configuration.
6925 <sect2 id="tools.oid.std">
6926 <title>Standard OIDs</title>
6928 All the object identifers in the standard OID database as returned
6929 by <function>yaz_oid_std</function> can be referenced directly in a
6930 program as a constant OID.
6931 Each constant OID is prefixed with <literal>yaz_oid_</literal> -
6932 followed by OID class (lowercase) - then by OID name (normalized and
6936 See <xref linkend="list-oids"/> for list of all object identifiers
6938 These are declared in <filename>yaz/oid_std.h</filename> but are
6939 included by <filename>yaz/oid_db.h</filename> as well.
6941 <example id="tools.oid.bib1.5">
6942 <title>Use a built-in OID</title>
6944 We can allocate our own OID filled with the constant OID for
6947 Odr_oid *bib1 = odr_oiddup(o, yaz_oid_attset_bib1);
6953 <sect1 id="tools.nmem">
6954 <title>Nibble Memory</title>
6956 Sometimes when you need to allocate and construct a large,
6957 interconnected complex of structures, it can be a bit of a pain to
6958 release the associated memory again. For the structures describing the
6959 Z39.50 PDUs and related structures, it is convenient to use the
6960 memory-management system of the &odr; subsystem (see
6961 <xref linkend="odr.use"/>). However, in some circumstances
6962 where you might otherwise benefit from using a simple nibble-memory
6963 management system, it may be impractical to use
6964 <function>odr_malloc()</function> and <function>odr_reset()</function>.
6965 For this purpose, the memory manager which also supports the &odr;
6966 streams is made available in the NMEM module. The external interface
6967 to this module is given in the <filename>nmem.h</filename> file.
6970 The following prototypes are given:
6973 NMEM nmem_create(void);
6974 void nmem_destroy(NMEM n);
6975 void *nmem_malloc(NMEM n, size_t size);
6976 void nmem_reset(NMEM n);
6977 size_t nmem_total(NMEM n);
6978 void nmem_init(void);
6979 void nmem_exit(void);
6982 The <function>nmem_create()</function> function returns a pointer to a
6983 memory control handle, which can be released again by
6984 <function>nmem_destroy()</function> when no longer needed.
6985 The function <function>nmem_malloc()</function> allocates a block of
6986 memory of the requested size. A call to <function>nmem_reset()</function>
6987 or <function>nmem_destroy()</function> will release all memory allocated
6988 on the handle since it was created (or since the last call to
6989 <function>nmem_reset()</function>. The function
6990 <function>nmem_total()</function> returns the number of bytes currently
6991 allocated on the handle.
6994 The nibble-memory pool is shared amongst threads. POSIX
6995 mutex'es and WIN32 Critical sections are introduced to keep the
6996 module thread safe. Function <function>nmem_init()</function>
6997 initializes the nibble-memory library and it is called automatically
6998 the first time the <literal>YAZ.DLL</literal> is loaded. &yaz; uses
6999 function <function>DllMain</function> to achieve this. You should
7000 <emphasis>not</emphasis> call <function>nmem_init</function> or
7001 <function>nmem_exit</function> unless you're absolute sure what
7002 you're doing. Note that in previous &yaz; versions you'd have to call
7003 <function>nmem_init</function> yourself.
7006 <sect1 id="tools.log">
7009 &yaz; has evolved a fairly complex log system which should be useful both
7010 for debugging &yaz; itself, debugging applications that use &yaz;, and for
7011 production use of those applications.
7014 The log functions are declared in header <filename>yaz/log.h</filename>
7015 and implemented in <filename>src/log.c</filename>.
7016 Due to name clash with syslog and some math utilities the logging
7017 interface has been modified as of YAZ 2.0.29. The obsolete interface
7018 is still available in header file <filename>yaz/log.h</filename>.
7019 The key points of the interface are:
7022 void yaz_log(int level, const char *fmt, ...)
7023 void yaz_log_init(int level, const char *prefix, const char *name);
7024 void yaz_log_init_file(const char *fname);
7025 void yaz_log_init_level(int level);
7026 void yaz_log_init_prefix(const char *prefix);
7027 void yaz_log_time_format(const char *fmt);
7028 void yaz_log_init_max_size(int mx);
7030 int yaz_log_mask_str(const char *str);
7031 int yaz_log_module_level(const char *name);
7034 The reason for the whole log module is the <function>yaz_log</function>
7035 function. It takes a bitmask indicating the log levels, a
7036 <literal>printf</literal>-like format string, and a variable number of
7040 The <literal>log level</literal> is a bit mask, that says on which level(s)
7041 the log entry should be made, and optionally set some behaviour of the
7042 logging. In the most simple cases, it can be one of <literal>YLOG_FATAL,
7043 YLOG_DEBUG, YLOG_WARN, YLOG_LOG</literal>. Those can be combined with bits
7044 that modify the way the log entry is written:<literal>YLOG_ERRNO,
7045 YLOG_NOTIME, YLOG_FLUSH</literal>.
7046 Most of the rest of the bits are deprecated, and should not be used. Use
7047 the dynamic log levels instead.
7050 Applications that use &yaz;, should not use the LOG_LOG for ordinary
7051 messages, but should make use of the dynamic loglevel system. This consists
7052 of two parts, defining the loglevel and checking it.
7055 To define the log levels, the (main) program should pass a string to
7056 <function>yaz_log_mask_str</function> to define which log levels are to be
7057 logged. This string should be a comma-separated list of log level names,
7058 and can contain both hard-coded names and dynamic ones. The log level
7059 calculation starts with <literal>YLOG_DEFAULT_LEVEL</literal> and adds a bit
7060 for each word it meets, unless the word starts with a '-', in which case it
7061 clears the bit. If the string <literal>'none'</literal> is found,
7062 all bits are cleared. Typically this string comes from the command-line,
7063 often identified by <literal>-v</literal>. The
7064 <function>yaz_log_mask_str</function> returns a log level that should be
7065 passed to <function>yaz_log_init_level</function> for it to take effect.
7068 Each module should check what log bits should be used, by calling
7069 <function>yaz_log_module_level</function> with a suitable name for the
7070 module. The name is cleared of a preceding path and an extension, if any,
7071 so it is quite possible to use <literal>__FILE__</literal> for it. If the
7072 name has been passed to <function>yaz_log_mask_str</function>, the routine
7073 returns a non-zero bitmask, which should then be used in consequent calls
7074 to yaz_log. (It can also be tested, so as to avoid unnecessary calls to
7075 yaz_log, in time-critical places, or when the log entry would take time
7079 Yaz uses the following dynamic log levels:
7080 <literal>server, session, request, requestdetail</literal> for the server
7082 <literal>zoom</literal> for the zoom client api.
7083 <literal>ztest</literal> for the simple test server.
7084 <literal>malloc, nmem, odr, eventl</literal> for internal
7085 debugging of yaz itself.
7086 Of course, any program using yaz is welcome to define as many new
7090 By default the log is written to stderr, but this can be changed by a call
7091 to <function>yaz_log_init_file</function> or
7092 <function>yaz_log_init</function>. If the log is directed to a file, the
7093 file size is checked at every write, and if it exceeds the limit given in
7094 <function>yaz_log_init_max_size</function>, the log is rotated. The
7095 rotation keeps one old version (with a <literal>.1</literal> appended to
7096 the name). The size defaults to 1GB. Setting it to zero will disable the
7100 A typical yaz-log looks like this
7101 13:23:14-23/11 yaz-ztest(1) [session] Starting session from tcp:127.0.0.1 (pid=30968)
7102 13:23:14-23/11 yaz-ztest(1) [request] Init from 'YAZ' (81) (ver 2.0.28) OK
7103 13:23:17-23/11 yaz-ztest(1) [request] Search Z: @attrset Bib-1 foo OK:7 hits
7104 13:23:22-23/11 yaz-ztest(1) [request] Present: [1] 2+2 OK 2 records returned
7105 13:24:13-23/11 yaz-ztest(1) [request] Close OK
7108 The log entries start with a time stamp. This can be omitted by setting the
7109 <literal>YLOG_NOTIME</literal> bit in the loglevel. This way automatic tests
7110 can be hoped to produce identical log files, that are easy to diff. The
7111 format of the time stamp can be set with
7112 <function>yaz_log_time_format</function>, which takes a format string just
7113 like <function>strftime</function>.
7116 Next in a log line comes the prefix, often the name of the program. For
7117 yaz-based servers, it can also contain the session number. Then
7118 comes one or more logbits in square brackets, depending on the logging
7119 level set by <function>yaz_log_init_level</function> and the loglevel
7120 passed to <function>yaz_log_init_level</function>. Finally comes the format
7121 string and additional values passed to <function>yaz_log</function>
7124 The log level <literal>YLOG_LOGLVL</literal>, enabled by the string
7125 <literal>loglevel</literal>, will log all the log-level affecting
7126 operations. This can come in handy if you need to know what other log
7127 levels would be useful. Grep the logfile for <literal>[loglevel]</literal>.
7130 The log system is almost independent of the rest of &yaz;, the only
7131 important dependence is of <filename>nmem</filename>, and that only for
7132 using the semaphore definition there.
7135 The dynamic log levels and log rotation were introduced in &yaz; 2.0.28. At
7136 the same time, the log bit names were changed from
7137 <literal>LOG_something</literal> to <literal>YLOG_something</literal>,
7138 to avoid collision with <filename>syslog.h</filename>.
7144 YAZ provides a fast utility for working with MARC records.
7145 Early versions of the MARC utility only allowed decoding of ISO2709.
7146 Today the utility may both encode - and decode to a variety of formats.
7149 #include <yaz/marcdisp.h>
7151 /* create handler */
7152 yaz_marc_t yaz_marc_create(void);
7154 void yaz_marc_destroy(yaz_marc_t mt);
7156 /* set XML mode YAZ_MARC_LINE, YAZ_MARC_SIMPLEXML, ... */
7157 void yaz_marc_xml(yaz_marc_t mt, int xmlmode);
7158 #define YAZ_MARC_LINE 0
7159 #define YAZ_MARC_SIMPLEXML 1
7160 #define YAZ_MARC_OAIMARC 2
7161 #define YAZ_MARC_MARCXML 3
7162 #define YAZ_MARC_ISO2709 4
7163 #define YAZ_MARC_XCHANGE 5
7164 #define YAZ_MARC_CHECK 6
7165 #define YAZ_MARC_TURBOMARC 7
7166 #define YAZ_MARC_JSON 8
7168 /* supply iconv handle for character set conversion .. */
7169 void yaz_marc_iconv(yaz_marc_t mt, yaz_iconv_t cd);
7171 /* set debug level, 0=none, 1=more, 2=even more, .. */
7172 void yaz_marc_debug(yaz_marc_t mt, int level);
7174 /* decode MARC in buf of size bsize. Returns >0 on success; <=0 on failure.
7175 On success, result in *result with size *rsize. */
7176 int yaz_marc_decode_buf(yaz_marc_t mt, const char *buf, int bsize,
7177 const char **result, size_t *rsize);
7179 /* decode MARC in buf of size bsize. Returns >0 on success; <=0 on failure.
7180 On success, result in WRBUF */
7181 int yaz_marc_decode_wrbuf(yaz_marc_t mt, const char *buf,
7182 int bsize, WRBUF wrbuf);
7187 The synopsis is just a basic subset of all functionality. Refer
7188 to the actual header file <filename>marcdisp.h</filename> for
7193 A MARC conversion handle must be created by using
7194 <function>yaz_marc_create</function> and destroyed
7195 by calling <function>yaz_marc_destroy</function>.
7198 All other functions operate on a <literal>yaz_marc_t</literal> handle.
7199 The output is specified by a call to <function>yaz_marc_xml</function>.
7200 The <literal>xmlmode</literal> must be one of
7203 <term>YAZ_MARC_LINE</term>
7206 A simple line-by-line format suitable for display but not
7207 recommended for further (machine) processing.
7212 <term>YAZ_MARC_MARCXML</term>
7215 <ulink url="&url.marcxml;">MARCXML</ulink>.
7220 <term>YAZ_MARC_ISO2709</term>
7223 ISO2709 (sometimes just referred to as "MARC").
7228 <term>YAZ_MARC_XCHANGE</term>
7231 <ulink url="&url.marcxchange;">MarcXchange</ulink>.
7236 <term>YAZ_MARC_CHECK</term>
7239 Pseudo format for validation only. Does not generate
7240 any real output except diagnostics.
7245 <term>YAZ_MARC_TURBOMARC</term>
7248 XML format with same semantics as MARCXML but more compact
7249 and geared towards fast processing with XSLT. Refer to
7250 <xref linkend="tools.turbomarc"/> for more information.
7255 <term>YAZ_MARC_JSON</term>
7258 <ulink url="&url.marc_in_json;">MARC-in-JSON</ulink> format.
7265 The actual conversion functions are
7266 <function>yaz_marc_decode_buf</function> and
7267 <function>yaz_marc_decode_wrbuf</function> which decodes and encodes
7268 a MARC record. The former function operates on simple buffers, and
7269 stores the resulting record in a WRBUF handle (WRBUF is a simple string
7272 <example id="example.marc.display">
7273 <title>Display of MARC record</title>
7275 The following program snippet illustrates how the MARC API may
7276 be used to convert a MARC record to the line-by-line format:
7277 <programlisting><![CDATA[
7278 void print_marc(const char *marc_buf, int marc_buf_size)
7280 char *result; /* for result buf */
7281 size_t result_len; /* for size of result */
7282 yaz_marc_t mt = yaz_marc_create();
7283 yaz_marc_xml(mt, YAZ_MARC_LINE);
7284 yaz_marc_decode_buf(mt, marc_buf, marc_buf_size,
7285 &result, &result_len);
7286 fwrite(result, result_len, 1, stdout);
7287 yaz_marc_destroy(mt); /* note that result is now freed... */
7293 <sect2 id="tools.turbomarc">
7294 <title>TurboMARC</title>
7296 TurboMARC is yet another XML encoding of a MARC record. The format
7297 was designed for fast processing with XSLT.
7301 Pazpar2 uses XSLT to convert an XML encoded MARC record to an internal
7302 representation. This conversion mostly checks the tag of a MARC field
7303 to determine the basic rules in the conversion. This check is
7304 costly when that tag is encoded as an attribute in MARCXML.
7305 By having the tag value as the element instead, makes processing
7306 many times faster (at least for Libxslt).
7309 TurboMARC is encoded as follows:
7313 Record elements is part of namespace
7314 "<literal>http://www.indexdata.com/turbomarc</literal>".
7319 A record is enclosed in element <literal>r</literal>.
7324 A collection of records is enclosed in element
7325 <literal>collection</literal>.
7330 The leader is encoded as element <literal>l</literal> with the
7331 leader content as its (text) value.
7336 A control field is encoded as element <literal>c</literal> concatenated
7337 with the tag value of the control field if the tag value
7338 matches the regular expression <literal>[a-zA-Z0-9]*</literal>.
7339 If the tag value does not match the regular expression
7340 <literal>[a-zA-Z0-9]*</literal> the control field is encoded
7341 as element <literal>c</literal> and attribute <literal>code</literal>
7342 will hold the tag value.
7343 This rule ensures that in the rare cases where a tag value might
7344 result in a non-well-formed XML, then YAZ will encode it as a coded attribute
7348 The control field content is the text value of this element.
7349 Indicators are encoded as attribute names
7350 <literal>i1</literal>, <literal>i2</literal>, etc. and
7351 corresponding values for each indicator.
7356 A data field is encoded as element <literal>d</literal> concatenated
7357 with the tag value of the data field or using the attribute
7358 <literal>code</literal> as described in the rules for control fields.
7359 The children of the data field element are subfield elements.
7360 Each subfield element is encoded as <literal>s</literal>
7361 concatenated with the sub field code.
7362 The text of the subfield element is the contents of the subfield.
7363 Indicators are encoded as attributes for the data field element, similar
7364 to the encoding for control fields.
7371 <sect1 id="tools.retrieval">
7372 <title>Retrieval Facility</title>
7374 YAZ version 2.1.20 or later includes a Retrieval facility tool
7375 which allows a SRU/Z39.50 to describe itself and perform record
7376 conversions. The idea is the following:
7380 An SRU/Z39.50 client sends a retrieval request which includes
7381 a combination of the following parameters: syntax (format),
7382 schema (or element set name).
7387 The retrieval facility is invoked with parameters in a
7388 server/proxy. The retrieval facility matches the parameters a set of
7389 "supported" retrieval types.
7390 If there is no match, the retrieval signals an error
7391 (syntax and / or schema not supported).
7396 For a successful match, the backend is invoked with the same
7397 or altered retrieval parameters (syntax, schema). If
7398 a record is received from the backend, it is converted to the
7399 frontend name / syntax.
7404 The resulting record is sent back the client and tagged with
7405 the frontend syntax / schema.
7411 The Retrieval facility is driven by an XML configuration. The
7412 configuration is neither Z39.50 ZeeRex or SRU ZeeRex. But it
7413 should be easy to generate both of them from the XML configuration.
7414 (Unfortunately the two versions
7415 of ZeeRex differ substantially in this regard.)
7417 <sect2 id="tools.retrieval.format">
7418 <title>Retrieval XML format</title>
7420 All elements should be covered by namespace
7421 <literal>http://indexdata.com/yaz</literal> .
7422 The root element node must be <literal>retrievalinfo</literal>.
7425 The <literal>retrievalinfo</literal> must include one or
7426 more <literal>retrieval</literal> elements. Each
7427 <literal>retrieval</literal> defines specific combination of
7428 syntax, name and identifier supported by this retrieval service.
7431 The <literal>retrieval</literal> element may include any of the
7432 following attributes:
7434 <varlistentry><term><literal>syntax</literal> (REQUIRED)</term>
7437 Defines the record syntax. Possible values is any
7438 of the names defined in YAZ' OID database or a raw
7443 <varlistentry><term><literal>name</literal> (OPTIONAL)</term>
7446 Defines the name of the retrieval format. This can be
7447 any string. For SRU, the value is equivalent to schema (short-hand);
7448 for Z39.50 it's equivalent to simple element set name.
7449 For YAZ 3.0.24 and later this name may be specified as a glob
7450 expression with operators
7451 <literal>*</literal> and <literal>?</literal>.
7455 <varlistentry><term><literal>identifier</literal> (OPTIONAL)</term>
7458 Defines the URI schema name of the retrieval format. This can be
7459 any string. For SRU, the value is equivalent to URI schema.
7460 For Z39.50, there is no equivalent.
7467 The <literal>retrieval</literal> may include one
7468 <literal>backend</literal> element. If a <literal>backend</literal>
7469 element is given, it specifies how the records are retrieved by
7470 some backend and how the records are converted from the backend to
7474 The attributes, <literal>name</literal> and <literal>syntax</literal>
7475 may be specified for the <literal>backend</literal> element. The
7476 semantics of these attributes is equivalent to those for the
7477 <literal>retrieval</literal>. However, these values are passed to
7481 The <literal>backend</literal> element may include one or more
7482 conversion instructions (as children elements). The supported
7485 <varlistentry><term><literal>marc</literal></term>
7488 The <literal>marc</literal> element specifies a conversion
7489 to - and from ISO2709 encoded MARC and
7490 <ulink url="&url.marcxml;">&acro.marcxml;</ulink>/MarcXchange.
7491 The following attributes may be specified:
7494 <term><literal>inputformat</literal> (REQUIRED)</term>
7497 Format of input. Supported values are
7498 <literal>marc</literal> (for ISO2709), <literal>xml</literal>
7499 (MARCXML/MarcXchange) and <literal>json</literal>
7500 (<ulink url="&url.marc_in_json;">MARC-in-JSON</ulink>).
7505 <term><literal>outputformat</literal> (REQUIRED)</term>
7508 Format of output. Supported values are
7509 <literal>line</literal> (MARC line format);
7510 <literal>marcxml</literal> (for MARCXML),
7511 <literal>marc</literal> (ISO2709),
7512 <literal>marcxhcange</literal> (for MarcXchange),
7513 or <literal>json</literal>
7514 (<ulink url="&url.marc_in_json;">MARC-in-JSON </ulink>).
7519 <term><literal>inputcharset</literal> (OPTIONAL)</term>
7522 Encoding of input. For XML input formats, this need not
7523 be given, but for ISO2709 based inputformats, this should
7524 be set to the encoding used. For MARC21 records, a common
7525 inputcharset value would be <literal>marc-8</literal>.
7530 <term><literal>outputcharset</literal> (OPTIONAL)</term>
7533 Encoding of output. If outputformat is XML based, it is
7534 strongly recommened to use <literal>utf-8</literal>.
7543 <term><literal>select</literal></term>
7546 The <literal>select</literal> selects one or more text nodes
7547 and decodes them as XML.
7548 The following attributes may be specified:
7550 <varlistentry><term><literal>path</literal> (REQUIRED)</term>
7553 X-Path expression for selecting text nodes.
7560 This conversion is available in YAZ 5.8.0 and later.
7565 <term><literal>solrmarc</literal></term>
7568 The <literal>solrmarc</literal> decodes solrmarc records.
7569 It assumes that the input is pure solrmarc text (no escaping)
7570 and will convert all sequences of the form #XX; to a single
7571 character of the hexadecimal value as given by XX. The output,
7572 presumably, is a valid ISO2709 buffer.
7575 This conversion is available in YAZ 5.0.21 and later.
7580 <term><literal>xslt</literal></term>
7583 The <literal>xslt</literal> element specifies a conversion
7584 via &acro.xslt;. The following attributes may be specified:
7586 <varlistentry><term><literal>stylesheet</literal> (REQUIRED)</term>
7600 <sect2 id="tools.retrieval.examples">
7601 <title>Retrieval Facility Examples</title>
7602 <example id="tools.retrieval.marc21">
7603 <title>MARC21 backend</title>
7605 A typical way to use the retrieval facility is to enable XML
7606 for servers that only supports ISO2709 encoded MARC21 records.
7608 <programlisting><![CDATA[
7610 <retrieval syntax="usmarc" name="F"/>
7611 <retrieval syntax="usmarc" name="B"/>
7612 <retrieval syntax="xml" name="marcxml"
7613 identifier="info:srw/schema/1/marcxml-v1.1">
7614 <backend syntax="usmarc" name="F">
7615 <marc inputformat="marc" outputformat="marcxml"
7616 inputcharset="marc-8"/>
7619 <retrieval syntax="xml" name="dc">
7620 <backend syntax="usmarc" name="F">
7621 <marc inputformat="marc" outputformat="marcxml"
7622 inputcharset="marc-8"/>
7623 <xslt stylesheet="MARC21slim2DC.xsl"/>
7630 This means that our frontend supports:
7634 MARC21 F(ull) records.
7639 MARC21 B(rief) records.
7649 Dublin core records.
7655 <example id="tools.retrieval.marcxml">
7656 <title>MARCXML backend</title>
7658 SRW/SRU and Solr backends return records in XML.
7659 If they return MARCXML or MarcXchange, the retrieval module
7660 can convert those into ISO2709 formats, most commonly USMARC
7662 In this example, the backend returns MARCXML for schema="marcxml".
7664 <programlisting><![CDATA[
7666 <retrieval syntax="usmarc">
7667 <backend syntax="xml" name="marcxml">
7668 <marc inputformat="xml" outputformat="marc"
7669 outputcharset="marc-8"/>
7672 <retrieval syntax="xml" name="marcxml"
7673 identifier="info:srw/schema/1/marcxml-v1.1"/>
7674 <retrieval syntax="xml" name="dc">
7675 <backend syntax="xml" name="marcxml">
7676 <xslt stylesheet="MARC21slim2DC.xsl"/>
7683 This means that our frontend supports:
7687 MARC21 records (any element set name) in MARC-8 encoding.
7692 MARCXML records for element-set=marcxml
7697 Dublin core records for element-set=dc.
7704 <sect2 id="tools.retrieval.api">
7707 It should be easy to use the retrieval systems from applications. Refer
7709 <filename>yaz/retrieval.h</filename> and
7710 <filename>yaz/record_conv.h</filename>.
7714 <sect1 id="sorting">
7715 <title>Sorting</title>
7717 This chapter describes sorting and how it is supported in YAZ.
7718 Sorting applies to a result-set.
7720 <ulink url="http://www.loc.gov/z3950/agency/markup/05.html#3.2.7">
7721 Z39.50 sorting facility
7723 takes one or more input result-sets
7724 and one result-set as output. The most simple case is that
7725 the input-set is the same as the output-set.
7728 Z39.50 sorting has a separate APDU (service) that is, thus, performed
7729 following a search (two phases).
7732 In SRU/Solr, however, the model is different. Here, sorting is specified
7733 during the search operation. Note, however, that SRU might
7734 perform sort as separate search, by referring to an existing result-set
7735 in the query (result-set reference).
7738 <title>Using the Z39.50 sort service</title>
7740 yaz-client and the ZOOM API support the Z39.50 sort facility. In any
7741 case the sort sequence or sort critiera is using a string notation.
7742 This notation is a one-line notation suitable for being manually
7743 entered or generated, and allows for easy logging (one liner).
7744 For the ZOOM API, the sort is specified in the call to ZOOM_query_sortby
7745 function. For yaz-client the sort is performed and specified using
7746 the sort and sort+ commands. For description of the sort criteria notation
7747 refer to the <link linkend="sortspec">sort command</link> in the
7751 The ZOOM API might choose one of several sort strategies for
7752 sorting. Refer to <xref linkend="zoom-sort-strategy"/>.
7756 <title>Type-7 sort</title>
7758 Type-7 sort is an extension to the Bib-1 based RPN query where the
7759 sort specification is embedded as an Attribute-Plus-Term.
7762 The objectives for introducing Type-7 sorting is that it allows
7763 a client to perform sorting even if it does not implement/support
7764 Z39.50 sort. Virtually all Z39.50 client software supports
7765 RPN queries. It also may improve performance because the sort
7766 critieria is specified along with the search query.
7769 The sort is triggered by the presence of type 7, and the value of type 7
7771 <ulink url="http://www.loc.gov/z3950/agency/asn1.html#SortKeySpec">
7774 The value for type 7 is 1 for ascending and 2 for descending.
7776 <ulink url="http://www.loc.gov/z3950/agency/asn1.html#SortElement">
7779 only the generic part is handled. If generic sortKey is of type
7780 sortField, then attribute type 1 is present and the value is
7781 sortField (InternationalString). If generic sortKey is of type
7782 sortAttributes, then the attributes in the list are used. Generic sortKey
7783 of type elementSpec is not supported.
7786 The term in the sorting Attribute-Plus-Term combo should hold
7787 an integer. The value is 0 for primary sorting criteria, 1 for second
7793 <title>Facets</title>
7795 YAZ supports facets in the Solr, SRU 2.0 and Z39.50 protocols.
7798 Like Type-1/RPN, YAZ supports a string notation for specifying
7799 facets. For the API this is performed by
7800 <function>yaz_pqf_parse_facet_list</function>.
7803 For ZOOM C the facets are given by option "facets".
7804 For yaz-client it is used for the 'facets' command.
7807 The grammar of this specification is as follows:
7809 facet-spec ::= facet-list
7811 facet-list ::= facet-list ',' attr-spec | attr-spec
7813 attr-spec ::= attr-spec '@attr' string | '@attr' string
7816 The notation is inspired by PQF. The string following '@attr'
7817 must not include blanks and is of the form
7818 <replaceable>type</replaceable><literal>=</literal><replaceable>value</replaceable>,
7819 where <replaceable>type</replaceable> is an integer and
7820 <replaceable>value</replaceable> is a string or an integer.
7823 The Facets specification is not Bib-1. The following types apply:
7825 <table id="facet.attributes">
7826 <title>Facet attributes</title>
7828 <colspec colwidth="2*" colname="type"></colspec>
7829 <colspec colwidth="9*" colname="description"></colspec>
7833 <entry>Description</entry>
7840 Field-name. This is often a string, e.g. "Author", "Year", etc.
7846 Sort order. Value should be an integer.
7847 Value 0: count descending (frequency). Value 1: alpha ascending.
7853 Number of terms requested.
7868 <title>The ODR Module</title>
7869 <sect1 id="odr.introduction">
7870 <title>Introduction</title>
7872 &odr; is the BER-encoding/decoding subsystem of &yaz;. Care has been taken
7873 to isolate &odr; from the rest of the package - specifically from the
7874 transport interface. &odr; may be used in any context where basic
7875 ASN.1/BER representations are used.
7878 If you are only interested in writing a Z39.50 implementation based on
7879 the PDUs that are already provided with &yaz;, you only need to concern
7880 yourself with the section on managing ODR streams
7881 (<xref linkend="odr.use"/>). Only if you need to
7882 implement ASN.1 beyond that which has been provided, should you
7883 worry about the second half of the documentation
7884 (<xref linkend="odr.programming"/>).
7885 If you use one of the higher-level interfaces, you can skip this
7889 This is important, so we'll repeat it for emphasis: <emphasis>You do
7890 not need to read <xref linkend="odr.programming"/>
7891 to implement Z39.50 with &yaz;.</emphasis>
7894 If you need a part of the protocol that isn't already in &yaz;, you
7895 should contact the authors before going to work on it yourself: We
7896 might already be working on it. Conversely, if you implement a useful
7897 part of the protocol before us, we'd be happy to include it in a
7901 <sect1 id="odr.use">
7902 <title>Using ODR</title>
7903 <sect2 id="odr.streams">
7904 <title>ODR Streams</title>
7906 Conceptually, the ODR stream is the source of encoded data in the
7907 decoding mode; when encoding, it is the receptacle for the encoded
7908 data. Before you can use an ODR stream it must be allocated. This is
7909 done with the function
7912 ODR odr_createmem(int direction);
7915 The <function>odr_createmem()</function> function takes as argument one
7916 of three manifest constants: <literal>ODR_ENCODE</literal>,
7917 <literal>ODR_DECODE</literal>, or <literal>ODR_PRINT</literal>.
7918 An &odr; stream can be in only one mode - it is not possible to change
7919 its mode once it's selected. Typically, your program will allocate
7920 at least two ODR streams - one for decoding, and one for encoding.
7923 When you're done with the stream, you can use
7926 void odr_destroy(ODR o);
7929 to release the resources allocated for the stream.
7932 <sect2 id="odr.memory.management">
7933 <title id="memory">Memory Management</title>
7935 Two forms of memory management take place in the &odr; system. The first
7936 one, which has to do with allocating little bits of memory (sometimes
7937 quite large bits of memory, actually) when a protocol package is
7938 decoded, and turned into a complex of interlinked structures. This
7939 section deals with this system, and how you can use it for your own
7940 purposes. The next section deals with the memory management which is
7941 required when encoding data - to make sure that a large enough buffer is
7942 available to hold the fully encoded PDU.
7945 The &odr; module has its own memory management system, which is
7946 used whenever memory is required. Specifically, it is used to allocate
7947 space for data when decoding incoming PDUs. You can use the memory
7948 system for your own purposes, by using the function
7951 void *odr_malloc(ODR o, size_t size);
7954 You can't use the normal <function>free(2)</function> routine to free
7955 memory allocated by this function, and &odr; doesn't provide a parallel
7956 function. Instead, you can call
7959 void odr_reset(ODR o);
7962 when you are done with the
7963 memory: Everything allocated since the last call to
7964 <function>odr_reset()</function> is released.
7965 The <function>odr_reset()</function> call is also required to clear
7966 up an error condition on a stream.
7972 size_t odr_total(ODR o);
7975 returns the number of bytes allocated on the stream since the last call to
7976 <function>odr_reset()</function>.
7979 The memory subsystem of &odr; is fairly efficient at allocating and
7980 releasing little bits of memory. Rather than managing the individual,
7981 small bits of space, the system maintains a free-list of larger chunks
7982 of memory, which are handed out in small bits. This scheme is
7983 generally known as a <emphasis>nibble-memory</emphasis> system.
7984 It is very useful for maintaining short-lived constructions such
7988 If you want to retain a bit of memory beyond the next call to
7989 <function>odr_reset()</function>, you can use the function
7992 ODR_MEM odr_extract_mem(ODR o);
7995 This function will give you control of the memory recently allocated
7996 on the ODR stream. The memory will live (past calls to
7997 <function>odr_reset()</function>), until you call the function
8000 void odr_release_mem(ODR_MEM p);
8003 The opaque <literal>ODR_MEM</literal> handle has no other purpose than
8004 referencing the memory block for you until you want to release it.
8007 You can use <function>odr_extract_mem()</function> repeatedly between
8008 allocating data, to retain individual control of separate chunks of data.
8011 <sect2 id="odr.encoding.and.decoding">
8012 <title>Encoding and Decoding Data</title>
8014 When encoding data, the ODR stream will write the encoded octet string
8015 in an internal buffer. To retrieve the data, use the function
8018 char *odr_getbuf(ODR o, int *len, int *size);
8021 The integer pointed to by len is set to the length of the encoded
8022 data, and a pointer to that data is returned. <literal>*size</literal>
8023 is set to the size of the buffer (unless <literal>size</literal> is null,
8024 signaling that you are not interested in the size). The next call to
8025 a primitive function using the same &odr; stream will overwrite the
8026 data, unless a different buffer has been supplied using the call
8029 void odr_setbuf(ODR o, char *buf, int len, int can_grow);
8032 which sets the encoding (or decoding) buffer used by
8033 <literal>o</literal> to <literal>buf</literal>, using the length
8034 <literal>len</literal>.
8035 Before a call to an encoding function, you can use
8036 <function>odr_setbuf()</function> to provide the stream with an encoding
8037 buffer of sufficient size (length). The <literal>can_grow</literal>
8038 parameter tells the encoding &odr; stream whether it is allowed to use
8039 <function>realloc(2)</function> to increase the size of the buffer when
8040 necessary. The default condition of a new encoding stream is equivalent
8041 to the results of calling
8044 odr_setbuf(stream, 0, 0, 1);
8047 In this case, the stream will allocate and reallocate memory as
8048 necessary. The stream reallocates memory by repeatedly doubling the
8049 size of the buffer - the result is that the buffer will typically
8050 reach its maximum, working size with only a small number of reallocation
8051 operations. The memory is freed by the stream when the latter is destroyed,
8052 unless it was assigned by the user with the <literal>can_grow</literal>
8053 parameter set to zero (in this case, you are expected to retain
8054 control of the memory yourself).
8057 To assume full control of an encoded buffer, you must first call
8058 <function>odr_getbuf()</function> to fetch the buffer and its length.
8059 Next, you should call <function>odr_setbuf()</function> to provide a
8060 different buffer (or a null pointer) to the stream. In the simplest
8061 case, you will reuse the same buffer over and over again, and you
8062 will just need to call <function>odr_getbuf()</function> after each
8063 encoding operation to get the length and address of the buffer.
8064 Note that the stream may reallocate the buffer during an encoding
8065 operation, so it is necessary to retrieve the correct address after
8066 each encoding operation.
8069 It is important to realize that the ODR stream will not release this
8070 memory when you call <function>odr_reset()</function>: It will
8071 merely update its internal pointers to prepare for the encoding of a
8073 When the stream is released by the <function>odr_destroy()</function>
8074 function, the memory given to it by <function>odr_setbuf</function> will
8075 be released <emphasis>only</emphasis> if the <literal>can_grow</literal>
8076 parameter to <function>odr_setbuf()</function> was nonzero. The
8077 <literal>can_grow</literal> parameter, in other words, is a way of
8078 signaling who is to own the buffer, you or the ODR stream. If you never call
8079 <function>odr_setbuf()</function> on your encoding stream, which is
8080 typically the case, the buffer allocated by the stream will belong to
8081 the stream by default.
8084 When you wish to decode data, you should first call
8085 <function>odr_setbuf()</function>, to tell the decoding stream
8086 where to find the encoded data, and how long the buffer is
8087 (the <literal>can_grow</literal> parameter is ignored by a decoding
8088 stream). After this, you can call the function corresponding to the
8089 data you wish to decode (eg, <function>odr_integer()</function> odr
8090 <function>z_APDU()</function>).
8092 <example id="example.odr.encoding.and.decoding.functions">
8093 <title>Encoding and decoding functions</title>
8095 int odr_integer(ODR o, Odr_int **p, int optional, const char *name);
8097 int z_APDU(ODR o, Z_APDU **p, int optional, const char *name);
8101 If the data is absent (or doesn't match the tag corresponding to
8102 the type), the return value will be either 0 or 1 depending on the
8103 <literal>optional</literal> flag. If <literal>optional</literal>
8104 is 0 and the data is absent, an error flag will be raised in the
8105 stream, and you'll need to call <function>odr_reset()</function> before
8106 you can use the stream again. If <literal>optional</literal> is
8107 nonzero, the pointer <emphasis>pointed</emphasis> to/ by
8108 <literal>p</literal> will be set to the null value, and the function
8110 The <literal>name</literal> argument is used to pretty-print the
8111 tag in question. It may be set to <literal>NULL</literal> if
8112 pretty-printing is not desired.
8115 If the data value is found where it's expected, the pointer
8116 <emphasis>pointed to</emphasis> by the <literal>p</literal> argument
8117 will be set to point to the decoded type.
8118 The space for the type will be allocated and owned by the &odr;
8119 stream, and it will live until you call
8120 <function>odr_reset()</function> on the stream. You cannot use
8121 <function>free(2)</function> to release the memory.
8122 You can decode several data elements (by repeated calls to
8123 <function>odr_setbuf()</function> and your decoding function), and
8124 new memory will be allocated each time. When you do call
8125 <function>odr_reset()</function>, everything decoded since the
8126 last call to <function>odr_reset()</function> will be released.
8128 <example id="example.odr.encoding.of.integer">
8129 <title>Encoding and decoding of an integer</title>
8131 The use of the double indirection can be a little confusing at first
8132 (its purpose will become clear later on, hopefully),
8133 so an example is in order. We'll encode an integer value, and
8134 immediately decode it again using a different stream. A useless, but
8135 informative operation.
8137 <programlisting><![CDATA[
8138 void do_nothing_useful(Odr_int value)
8141 Odr_int *valp, *resvalp;
8145 /* allocate streams */
8146 if (!(encode = odr_createmem(ODR_ENCODE)))
8148 if (!(decode = odr_createmem(ODR_DECODE)))
8152 if (odr_integer(encode, &valp, 0, 0) == 0)
8154 printf("encoding went bad\n");
8157 bufferp = odr_getbuf(encode, &len, 0);
8158 printf("length of encoded data is %d\n", len);
8160 /* now let's decode the thing again */
8161 odr_setbuf(decode, bufferp, len, 0);
8162 if (odr_integer(decode, &resvalp, 0, 0) == 0)
8164 printf("decoding went bad\n");
8167 /* ODR_INT_PRINTF format for printf (such as %d) */
8168 printf("the value is " ODR_INT_PRINTF "\n", *resvalp);
8171 odr_destroy(encode);
8172 odr_destroy(decode);
8177 This looks like a lot of work, offhand. In practice, the &odr; streams
8178 will typically be allocated once, in the beginning of your program
8179 (or at the beginning of a new network session), and the encoding
8180 and decoding will only take place in a few, isolated places in your
8181 program, so the overhead is quite manageable.
8185 <sect2 id="odr.printing">
8186 <title>Printing</title>
8188 When an ODR stream is created of type <literal>ODR_PRINT</literal>
8189 the ODR module will print the contents of a PDU in a readable format.
8190 By default output is written to the <literal>stderr</literal> stream.
8191 This behavior can be changed, however, by calling the function
8193 odr_setprint(ODR o, FILE *file);
8195 before encoders or decoders are being invoked.
8196 It is also possible to direct the output to a buffer (or indeed
8197 another file), by using the more generic mechanism:
8199 void odr_set_stream(ODR o, void *handle,
8200 void (*stream_write)(ODR o, void *handle, int type,
8201 const char *buf, int len),
8202 void (*stream_close)(void *handle));
8204 Here the user provides an opaque handle and two handlers,
8205 <replaceable>stream_write</replaceable> for writing,
8206 and <replaceable>stream_close</replaceable> which is supposed
8207 to close/free resources associated with handle.
8208 The <replaceable>stream_close</replaceable> handler is optional and
8209 if NULL for the function is provided, it will not be invoked.
8210 The <replaceable>stream_write</replaceable> takes the ODR handle
8211 as parameter, the user-defined handle, a type
8212 <literal>ODR_OCTETSTRING</literal>, <literal>ODR_VISIBLESTRING</literal>
8213 which indicates the type of contents being written.
8216 Another utility useful for diagnostics (error handling) or as
8217 part of the printing facilities is:
8219 const char **odr_get_element_path(ODR o);
8221 which returns a list of current elements that ODR deals with at the
8222 moment. For the returned array, say <literal>ar</literal>,
8223 then <literal>ar[0]</literal> is the top level element,
8224 <literal>ar[n]</literal> is the last. The last element has the
8225 property that <literal>ar[n+1] == NULL</literal>.
8227 <example id="example.odr.element.path.record">
8228 <title>Element Path for record</title>
8230 For a database record part of a PresentResponse the
8231 array returned by <function>odr_get_element</function>
8232 is <literal>presentResponse</literal>, <literal>databaseOrSurDiagnostics</literal>, <literal>?</literal>, <literal>record</literal>, <literal>?</literal>, <literal>databaseRecord</literal> . The question mark appears due to
8233 unnamed constructions.
8237 <sect2 id="odr.diagnostics">
8238 <title>Diagnostics</title>
8240 The encoding/decoding functions all return 0 when an error occurs.
8241 Until you call <function>odr_reset()</function>, you cannot use the
8242 stream again, and any function called will immediately return 0.
8245 To provide information to the programmer or administrator, the function
8248 void odr_perror(ODR o, char *message);
8251 is provided, which prints the <literal>message</literal> argument to
8252 <literal>stderr</literal> along with an error message from the stream.
8255 You can also use the function
8258 int odr_geterror(ODR o);
8261 to get the current error number from the screen. The number will be
8262 one of these constants:
8264 <table frame="top" id="odr.error.codes">
8265 <title>ODR Error codes</title>
8270 <entry>Description</entry>
8275 <entry>OMEMORY</entry><entry>Memory allocation failed.</entry>
8278 <entry>OSYSERR</entry><entry>A system- or library call has failed.
8279 The standard diagnostic variable <literal>errno</literal> should be
8280 examined to determine the actual error.</entry>
8283 <entry>OSPACE</entry><entry>No more space for encoding.
8284 This will only occur when the user has explicitly provided a
8285 buffer for an encoding stream without allowing the system to
8286 allocate more space.</entry>
8289 <entry>OREQUIRED</entry><entry>This is a common protocol error; A
8290 required data element was missing during encoding or decoding.</entry>
8293 <entry>OUNEXPECTED</entry><entry>An unexpected data element was
8294 found during decoding.</entry>
8297 <entry>OOTHER</entry><entry>Other error. This is typically an
8298 indication of misuse of the &odr; system by the programmer, and also
8299 that the diagnostic system isn't as good as it should be, yet.</entry>
8305 The character string array
8311 can be indexed by the error code to obtain a human-readable
8312 representation of the problem.
8315 <sect2 id="odr.summary.and.synopsis">
8316 <title>Summary and Synopsis</title>
8318 #include <yaz/odr.h>
8320 ODR odr_createmem(int direction);
8322 void odr_destroy(ODR o);
8324 void odr_reset(ODR o);
8326 char *odr_getbuf(ODR o, int *len, int *size);
8328 void odr_setbuf(ODR o, char *buf, int len, int can_grow);
8330 void *odr_malloc(ODR o, int size);
8332 NMEM odr_extract_mem(ODR o);
8334 int odr_geterror(ODR o);
8336 void odr_perror(ODR o, const char *message);
8338 extern char *odr_errlist[];
8342 <sect1 id="odr.programming">
8343 <title>Programming with ODR</title>
8345 The API of &odr; is designed to reflect the structure of ASN.1, rather
8346 than BER itself. Future releases may be able to represent data in
8347 other external forms.
8351 There is an ASN.1 tutorial available at
8352 <ulink url="&url.asn.1.tutorial;">this site</ulink>.
8353 This site also has standards for ASN.1 (X.680) and BER (X.690)
8354 <ulink url="&url.asn.1.standards;">online</ulink>.
8358 The ODR interface is based loosely on that of the Sun Microsystems
8360 Specifically, each function which corresponds to an ASN.1 primitive
8361 type has a dual function. Depending on the settings of the ODR
8362 stream which is supplied as a parameter, the function may be used
8363 either to encode or decode data. The functions that can be built
8364 using these primitive functions, to represent more complex data types,
8365 share this quality. The result is that you only have to enter the
8366 definition for a type once - and you have the functionality of encoding,
8367 decoding (and pretty-printing) all in one unit.
8368 The resulting C source code is quite compact, and is a pretty
8369 straightforward representation of the source ASN.1 specification.
8372 In many cases, the model of the XDR functions works quite well in this
8374 In others, it is less elegant. Most of the hassle comes from the optional
8375 SEQUENCE members which don't exist in XDR.
8377 <sect2 id="odr.primitive.asn1.types">
8378 <title>The Primitive ASN.1 Types</title>
8380 ASN.1 defines a number of primitive types (many of which correspond
8381 roughly to primitive types in structured programming languages, such as C).
8383 <sect3 id="odr.integer">
8384 <title>INTEGER</title>
8386 The &odr; function for encoding or decoding (or printing) the ASN.1
8387 INTEGER type looks like this:
8390 int odr_integer(ODR o, Odr_int **p, int optional, const char *name);
8393 The <literal>Odr_int</literal> is just a simple integer.
8396 This form is typical of the primitive &odr; functions. They are named
8397 after the type of data that they encode or decode. They take an &odr;
8398 stream, an indirect reference to the type in question, and an
8399 <literal>optional</literal> flag (corresponding to the OPTIONAL keyword
8400 of ASN.1) as parameters. They all return an integer value of either one
8402 When you use the primitive functions to construct encoders for complex
8403 types of your own, you should follow this model as well. This
8404 ensures that your new types can be reused as elements in yet more
8408 The <literal>o</literal> parameter should obviously refer to a properly
8409 initialized &odr; stream of the right type (encoding/decoding/printing)
8410 for the operation that you wish to perform.
8413 When encoding or printing, the function first looks at
8414 <literal>* p</literal>. If <literal>* p</literal> (the pointer pointed
8415 to by <literal>p</literal>) is a null pointer, this is taken to mean that
8416 the data element is absent. If the <literal>optional</literal> parameter
8417 is nonzero, the function will return one (signifying success) without
8418 any further processing. If the <literal>optional</literal> is zero, an
8419 internal error flag is set in the &odr; stream, and the function will
8420 return 0. No further operations can be carried out on the stream without
8421 a call to the function <function>odr_reset()</function>.
8424 If <literal>*p</literal> is not a null pointer, it is expected to
8425 point to an instance of the data type. The data will be subjected to
8426 the encoding rules, and the result will be placed in the buffer held
8427 by the &odr; stream.
8430 The other ASN.1 primitives have similar functions that operate in
8434 <sect3 id="odr.boolean">
8435 <title>BOOLEAN</title>
8437 int odr_bool(ODR o, Odr_bool **p, int optional, const char *name);
8440 <sect3 id="odr.real">
8446 <sect3 id="odr.null">
8449 int odr_null(ODR o, Odr_null **p, int optional, const char *name);
8452 In this case, the value of **p is not important. If <literal>*p</literal>
8453 is different from the null pointer, the null value is present, otherwise
8457 <sect3 id="odr.octet.string">
8458 <title>OCTET STRING</title>
8460 typedef struct odr_oct
8466 int odr_octetstring(ODR o, Odr_oct **p, int optional,
8470 The <literal>buf</literal> field should point to the character array
8471 that holds the octetstring. The <literal>len</literal> field holds the
8473 The character array need not be null-terminated.
8476 To make things a little easier, an alternative is given for string
8477 types that are not expected to contain embedded NULL characters (e.g.
8481 int odr_cstring(ODR o, char **p, int optional, const char *name);
8484 which encodes or decodes between OCTETSTRING representations and
8485 null-terminated C strings.
8488 Functions are provided for the derived string types, e.g.:
8491 int odr_visiblestring(ODR o, char **p, int optional,
8495 <sect3 id="odr.bit.string">
8496 <title>BIT STRING</title>
8498 int odr_bitstring(ODR o, Odr_bitmask **p, int optional,
8502 The opaque type <literal>Odr_bitmask</literal> is only suitable for
8503 holding relatively brief bit strings, e.g. for options fields, etc.
8504 The constant <literal>ODR_BITMASK_SIZE</literal> multiplied by 8
8505 gives the maximum possible number of bits.
8508 A set of macros are provided for manipulating the
8509 <literal>Odr_bitmask</literal> type:
8512 void ODR_MASK_ZERO(Odr_bitmask *b);
8514 void ODR_MASK_SET(Odr_bitmask *b, int bitno);
8516 void ODR_MASK_CLEAR(Odr_bitmask *b, int bitno);
8518 int ODR_MASK_GET(Odr_bitmask *b, int bitno);
8521 The functions are modeled after the manipulation functions that
8522 accompany the <literal>fd_set</literal> type used by the
8523 <function>select(2)</function> call.
8524 <literal>ODR_MASK_ZERO</literal> should always be called first on a
8525 new bitmask, to initialize the bits to zero.
8528 <sect3 id="odr.object.identifier">
8529 <title>OBJECT IDENTIFIER</title>
8531 int odr_oid(ODR o, Odr_oid **p, int optional, const char *name);
8534 The C OID representation is simply an array of integers, terminated by
8535 the value -1 (the <literal>Odr_oid</literal> type is synonymous with
8536 the <literal>short</literal> type).
8537 We suggest that you use the OID database module (see
8538 <xref linkend="tools.oid.database"/>) to handle object identifiers
8539 in your application.
8543 <sect2 id="odr.tagging.primitive.types">
8544 <title>Tagging Primitive Types</title>
8546 The simplest way of tagging a type is to use the
8547 <function>odr_implicit_tag()</function> or
8548 <function>odr_explicit_tag()</function> macros:
8551 int odr_implicit_tag(ODR o, Odr_fun fun, int class, int tag,
8552 int optional, const char *name);
8554 int odr_explicit_tag(ODR o, Odr_fun fun, int class, int tag,
8555 int optional, const char *name);
8558 To create a type derived from the integer type by implicit tagging, you
8562 MyInt ::= [210] IMPLICIT INTEGER
8565 In the &odr; system, this would be written like:
8568 int myInt(ODR o, Odr_int **p, int optional, const char *name)
8570 return odr_implicit_tag(o, odr_integer, p,
8571 ODR_CONTEXT, 210, optional, name);
8575 The function <function>myInt()</function> can then be used like any of
8576 the primitive functions provided by &odr;. Note that the behavior of
8577 <function>odr_explicit_tag()</function>
8578 and <function>odr_implicit_tag()</function> macros
8579 act exactly the same as the functions they are applied to - they
8580 respond to error conditions, etc, in the same manner - they
8581 simply have three extra parameters. The class parameter may
8582 take one of the values: <literal>ODR_CONTEXT</literal>,
8583 <literal>ODR_PRIVATE</literal>, <literal>ODR_UNIVERSAL</literal>, or
8584 <literal>/ODR_APPLICATION</literal>.
8587 <sect2 id="odr.constructed.types">
8588 <title>Constructed Types</title>
8590 Constructed types are created by combining primitive types. The
8591 &odr; system only implements the SEQUENCE and SEQUENCE OF constructions
8592 (although adding the rest of the container types should be simple
8593 enough, if the need arises).
8596 For implementing SEQUENCEs, the functions
8599 int odr_sequence_begin(ODR o, void *p, int size, const char *name);
8600 int odr_sequence_end(ODR o);
8606 The <function>odr_sequence_begin()</function> function should be
8607 called in the beginning of a function that implements a SEQUENCE type.
8608 Its parameters are the &odr; stream, a pointer (to a pointer to the type
8609 you're implementing), and the <literal>size</literal> of the type
8610 (typically a C structure). On encoding, it returns 1 if
8611 <literal>* p</literal> is a null pointer. The <literal>size</literal>
8612 parameter is ignored. On decoding, it returns 1 if the type is found in
8613 the data stream. <literal>size</literal> bytes of memory are allocated,
8614 and <literal>*p</literal> is set to point to this space.
8615 The <function>odr_sequence_end()</function> is called at the end of the
8616 complex function. Assume that a type is defined like this:
8619 MySequence ::= SEQUENCE {
8621 boolval BOOLEAN OPTIONAL
8625 The corresponding &odr; encoder/decoder function and the associated data
8626 structures could be written like this:
8629 typedef struct MySequence
8635 int mySequence(ODR o, MySequence **p, int optional, const char *name)
8637 if (odr_sequence_begin(o, p, sizeof(**p), name) == 0)
8638 return optional && odr_ok(o);
8640 odr_integer(o, &(*p)->intval, 0, "intval") &&
8641 odr_bool(o, &(*p)->boolval, 1, "boolval") &&
8642 odr_sequence_end(o);
8646 Note the 1 in the call to <function>odr_bool()</function>, to mark
8647 that the sequence member is optional.
8648 If either of the member types had been tagged, the macros
8649 <function>odr_implicit_tag()</function> or
8650 <function>odr_explicit_tag()</function>
8651 could have been used.
8652 The new function can be used exactly like the standard functions provided
8653 with &odr;. It will encode, decode or pretty-print a data value of the
8654 <literal>MySequence</literal> type. We like to name types with an
8655 initial capital, as done in ASN.1 definitions, and to name the
8656 corresponding function with the first character of the name in lower case.
8657 You could, of course, name your structures, types, and functions any way
8658 you please - as long as you're consistent, and your code is easily readable.
8659 <literal>odr_ok</literal> is just that - a predicate that returns the
8660 state of the stream. It is used to ensure that the behavior of the new
8661 type is compatible with the interface of the primitive types.
8664 <sect2 id="odr.tagging.constructed.types">
8665 <title>Tagging Constructed Types</title>
8668 See <xref linkend="odr.tagging.primitive.types"/> for information
8669 on how to tag the primitive types, as well as types that are
8673 <sect3 id="odr.implicit.tagging">
8674 <title>Implicit Tagging</title>
8676 Assume the type above had been defined as
8679 MySequence ::= [10] IMPLICIT SEQUENCE {
8681 boolval BOOLEAN OPTIONAL
8685 You would implement this in &odr; by calling the function
8688 int odr_implicit_settag(ODR o, int class, int tag);
8691 which overrides the tag of the type immediately following it. The
8692 macro <function>odr_implicit_tag()</function> works by calling
8693 <function>odr_implicit_settag()</function> immediately
8694 before calling the function pointer argument.
8695 Your type function could look like this:
8698 int mySequence(ODR o, MySequence **p, int optional, const char *name)
8700 if (odr_implicit_settag(o, ODR_CONTEXT, 10) == 0 ||
8701 odr_sequence_begin(o, p, sizeof(**p), name) == 0)
8702 return optional && odr_ok(o);
8704 odr_integer(o, &(*p)->intval, 0, "intval") &&
8705 odr_bool(o, &(*p)->boolval, 1, "boolval") &&
8706 odr_sequence_end(o);
8710 The definition of the structure <literal>MySequence</literal> would be
8714 <sect3 id="odr.explicit.tagging">
8715 <title>Explicit Tagging</title>
8717 Explicit tagging of constructed types is a little more complicated,
8718 since you are in effect adding a level of construction to the data.
8721 Assume the definition:
8724 MySequence ::= [10] IMPLICIT SEQUENCE {
8726 boolval BOOLEAN OPTIONAL
8730 Since the new type has an extra level of construction, two new functions
8731 are needed to encapsulate the base type:
8734 int odr_constructed_begin(ODR o, void *p, int class, int tag,
8737 int odr_constructed_end(ODR o);
8740 Assume that the IMPLICIT in the type definition above were replaced
8741 with EXPLICIT (or that the IMPLICIT keyword was simply deleted, which
8742 would be equivalent). The structure definition would look the same,
8743 but the function would look like this:
8746 int mySequence(ODR o, MySequence **p, int optional, const char *name)
8748 if (odr_constructed_begin(o, p, ODR_CONTEXT, 10, name) == 0)
8749 return optional && odr_ok(o);
8750 if (o->direction == ODR_DECODE)
8751 *p = odr_malloc(o, sizeof(**p));
8752 if (odr_sequence_begin(o, p, sizeof(**p), 0) == 0)
8754 *p = 0; /* this is almost certainly a protocol error */
8758 odr_integer(o, &(*p)->intval, 0, "intval") &&
8759 odr_bool(o, &(*p)->boolval, 1, "boolval") &&
8760 odr_sequence_end(o) &&
8761 odr_constructed_end(o);
8765 Notice that the interface here gets kind of nasty. The reason is
8766 simple: Explicitly tagged, constructed types are fairly rare in
8767 the protocols that we care about, so the
8768 aesthetic annoyance (not to mention the dangers of a cluttered
8769 interface) is less than the time that would be required to develop a
8770 better interface. Nevertheless, it is far from satisfying, and it's a
8771 point that will be worked on in the future. One option for you would
8772 be to simply apply the <function>odr_explicit_tag()</function> macro to
8773 the first function, and not
8774 have to worry about <function>odr_constructed_*</function> yourself.
8775 Incidentally, as you might have guessed, the
8776 <function>odr_sequence_</function> functions are themselves
8777 implemented using the <function>/odr_constructed_</function> functions.
8781 <sect2 id="odr.sequence.of">
8782 <title>SEQUENCE OF</title>
8784 To handle sequences (arrays) of a specific type, the function
8787 int odr_sequence_of(ODR o, int (*fun)(ODR o, void *p, int optional),
8788 void *p, int *num, const char *name);
8791 The <literal>fun</literal> parameter is a pointer to the decoder/encoder
8792 function of the type. <literal>p</literal> is a pointer to an array of
8793 pointers to your type. <literal>num</literal> is the number of elements
8800 MyArray ::= SEQUENCE OF INTEGER
8803 The C representation might be
8806 typedef struct MyArray
8813 And the function might look like
8816 int myArray(ODR o, MyArray **p, int optional, const char *name)
8818 if (o->direction == ODR_DECODE)
8819 *p = odr_malloc(o, sizeof(**p));
8820 if (odr_sequence_of(o, odr_integer, &(*p)->elements,
8821 &(*p)->num_elements, name))
8824 return optional && odr_ok(o);
8828 <sect2 id="odr.choice.types">
8829 <title>CHOICE Types</title>
8831 The choice type is used fairly often in some ASN.1 definitions, so
8832 some work has gone into streamlining its interface.
8835 CHOICE types are handled by the function:
8838 int odr_choice(ODR o, Odr_arm arm[], void *p, void *whichp,
8842 The <literal>arm</literal> array is used to describe each of the possible
8843 types that the CHOICE type may assume. Internally in your application,
8844 the CHOICE type is represented as a discriminated union. That is, a
8845 C union accompanied by an integer (or enum) identifying the active
8847 <literal>whichp</literal> is a pointer to the union discriminator.
8848 When encoding, it is examined to determine the current type.
8849 When decoding, it is set to reference the type that was found in
8853 The Odr_arm type is defined thus:
8856 typedef struct odr_arm
8867 The interpretation of the fields are:
8871 <term>tagmode</term>
8872 <listitem><para>Either <literal>ODR_IMPLICIT</literal>,
8873 <literal>ODR_EXPLICIT</literal>, or <literal>ODR_NONE</literal> (-1)
8874 to mark no tagging.</para></listitem>
8878 <listitem><para>The value of the discriminator that corresponds to
8879 this CHOICE element. Typically, it will be a #defined constant, or
8880 an enum member.</para></listitem>
8884 <listitem><para>A pointer to a function that implements the type of
8885 the CHOICE member. It may be either a standard &odr; type or a type
8886 defined by yourself.</para></listitem>
8890 <listitem><para>Name of tag.</para></listitem>
8894 A handy way to prepare the array for use by the
8895 <function>odr_choice()</function> function is to
8896 define it as a static, initialized array in the beginning of your
8897 decoding/encoding function. Assume the type definition:
8900 MyChoice ::= CHOICE {
8902 tagged [99] IMPLICIT INTEGER,
8907 Your C type might look like
8910 typedef struct MyChoice
8927 And your function could look like this:
8930 int myChoice(ODR o, MyChoice **p, int optional, const char *name)
8932 static Odr_arm arm[] =
8934 {-1, -1, -1, MyChoice_untagged, odr_integer, "untagged"},
8935 {ODR_IMPLICIT, ODR_CONTEXT, 99, MyChoice_tagged, odr_integer,
8937 {-1, -1, -1, MyChoice_other, odr_boolean, "other"},
8941 if (o->direction == ODR_DECODE)
8942 *p = odr_malloc(o, sizeof(**p);
8944 return optional && odr_ok(o);
8946 if (odr_choice(o, arm, &(*p)->u, &(*p)->which), name)
8949 return optional && odr_ok(o);
8953 In some cases (say, a non-optional choice which is a member of a
8954 sequence), you can "embed" the union and its discriminator in the
8955 structure belonging to the enclosing type, and you won't need to
8956 fiddle with memory allocation to create a separate structure to
8957 wrap the discriminator and union.
8960 The corresponding function is somewhat nicer in the Sun XDR interface.
8961 Most of the complexity of this interface comes from the possibility of
8962 declaring sequence elements (including CHOICEs) optional.
8965 The ASN.1 specifications naturally require that each member of a
8966 CHOICE have a distinct tag, so they can be told apart on decoding.
8967 Sometimes it can be useful to define a CHOICE that has multiple types
8968 that share the same tag. You'll need some other mechanism, perhaps
8969 keyed to the context of the CHOICE type. In effect, we would like to
8970 introduce a level of context-sensitiveness to our ASN.1 specification.
8971 When encoding an internal representation, we have no problem, as long
8972 as each CHOICE member has a distinct discriminator value. For
8973 decoding, we need a way to tell the choice function to look for a
8974 specific arm of the table. The function
8977 void odr_choice_bias(ODR o, int what);
8980 provides this functionality. When called, it leaves a notice for the next
8981 call to <function>odr_choice()</function> to be called on the decoding
8982 stream <literal>o</literal>, that only the <literal>arm</literal> entry with
8983 a <literal>which</literal> field equal to <literal>what</literal>
8987 The most important application (perhaps the only one, really) is in
8988 the definition of application-specific EXTERNAL encoders/decoders
8989 which will automatically decode an ANY member given the direct or
8994 <sect1 id="odr.debugging">
8995 <title>Debugging</title>
8997 The protocol modules are suffering somewhat from a lack of diagnostic
8998 tools at the moment. Specifically ways to pretty-print PDUs that
8999 aren't recognized by the system. We'll include something to this end
9000 in a not-too-distant release. In the meantime, what we do when we get
9001 packages we don't understand is to compile the ODR module with
9002 <literal>ODR_DEBUG</literal> defined. This causes the module to dump tracing
9003 information as it processes data units. With this output and the
9004 protocol specification (Z39.50), it is generally fairly easy to see
9009 <chapter id="comstack">
9010 <title>The COMSTACK Module</title>
9011 <sect1 id="comstack.synopsis">
9012 <title>Synopsis (blocking mode)</title>
9013 <programlisting><![CDATA[
9016 int size = 0, length_incoming;
9017 char server_address_str[] = "localhost:9999";
9018 void *server_address_ip;
9021 char *protocol_package = "GET / HTTP/1.0\r\n\r\n";
9022 int protocol_package_length = strlen(protocol_package);
9024 stack = cs_create(tcpip_type, 1, PROTO_HTTP);
9026 perror("cs_create"); /* use perror() here since we have no stack yet */
9030 server_address_ip = cs_straddr(stack, server_address_str);
9031 if (!server_address_ip) {
9032 fprintf(stderr, "cs_straddr: address could not be resolved\n");
9036 status = cs_connect(stack, server_address_ip);
9038 fprintf(stderr, "cs_connect: %s\n", cs_strerror(stack));
9042 status = cs_rcvconnect(stack);
9044 fprintf(stderr, "cs_rcvconnect: %s\n", cs_strerror(stack));
9048 status = cs_put(stack, protocol_package, protocol_package_length);
9050 fprintf(stderr, "cs_put: %s\n", cs_strerror(stack));
9054 /* Now get a response */
9055 length_incoming = cs_get(stack, &buf, &size);
9056 if (!length_incoming) {
9057 fprintf(stderr, "Connection closed\n");
9059 } else if (length_incoming < 0) {
9060 fprintf(stderr, "cs_get: %s\n", cs_strerror(stack));
9065 fwrite(buf, length_incoming, 1, stdout);
9076 <sect1 id="comstack.introduction">
9077 <title>Introduction</title>
9080 subsystem provides a transparent interface to different types of transport
9081 stacks for the exchange of BER-encoded data and HTTP packets.
9082 At present, the RFC1729 method (BER over TCP/IP), local UNIX socket and an
9083 experimental SSL stack are supported, but others may be added in time.
9084 The philosophy of the
9085 module is to provide a simple interface by hiding unused options and
9086 facilities of the underlying libraries. This is always done at the risk
9087 of losing generality, and it may prove that the interface will need
9092 There hasn't been interest in the XTImOSI stack for some years.
9093 Therefore, it is no longer supported.
9097 The interface is implemented in such a fashion that only the
9098 sub-layers constructed to the transport methods that you wish to
9099 use in your application are linked in.
9102 You will note that even though simplicity was a goal in the design,
9103 the interface is still orders of magnitudes more complex than the
9104 transport systems found in many other packages. One reason is that
9105 the interface needs to support the somewhat different requirements of
9106 the different lower-layer communications stacks; another important
9107 reason is that the interface seeks to provide a more or less
9108 industrial-strength approach to asynchronous event-handling.
9109 When no function is allowed to block, things get more complex -
9110 particularly on the server side.
9111 We urge you to have a look at the demonstration client and server
9112 provided with the package. They are meant to be easily readable and
9113 instructive, while still being at least moderately useful.
9116 <sect1 id="comstack.common">
9117 <title>Common Functions</title>
9118 <sect2 id="comstack.managing.endpoints">
9119 <title>Managing Endpoints</title>
9121 COMSTACK cs_create(CS_TYPE type, int blocking, int protocol);
9124 Creates an instance of the protocol stack - a communications endpoint.
9125 The <literal>type</literal> parameter determines the mode
9126 of communication. At present the following values are supported:
9130 <term><literal>tcpip_type</literal></term>
9131 <listitem><para>TCP/IP (BER over TCP/IP or HTTP over TCP/IP)
9135 <term><literal>ssl_type</literal></term>
9136 <listitem><para>Secure Socket Layer (SSL). This COMSTACK
9137 is experimental and is not fully implemented. If
9138 HTTP is used, this effectively is HTTPS.
9142 <term><literal>unix_type</literal></term>
9143 <listitem><para>Unix socket (unix only). Local Transfer via
9144 file socket. See <citerefentry><refentrytitle>unix</refentrytitle>
9145 <manvolnum>7</manvolnum></citerefentry>.
9150 The <function>cs_create</function> function returns a null-pointer
9151 if a system error occurs.
9152 The <literal>blocking</literal> parameter should be '1' if
9153 you wish the association to operate in blocking mode, and '0' otherwise.
9154 The <literal>protocol</literal> field should be
9155 <literal>PROTO_Z3950</literal> or <literal>PROTO_HTTP</literal>.
9156 Protocol <literal>PROTO_SR</literal> is no longer supported.
9159 void cs_close(COMSTACK handle);
9162 Closes the connection (as elegantly as the lower layers will permit),
9163 and releases the resources pointed to by the
9164 <literal>handle</literal>
9166 <literal>handle</literal>
9167 should not be referenced again after this call.
9171 We really need a soft disconnect, don't we?
9175 <sect2 id="comstack.data.exchange">
9176 <title>Data Exchange</title>
9178 int cs_put(COMSTACK handle, char *buf, int len);
9181 Sends <literal>buf</literal> down the wire.
9182 In blocking mode, this function will return only when a full buffer has
9183 been written, or an error has occurred. In nonblocking mode, it's
9184 possible that the function will be unable to send the full buffer
9185 at once, which will be indicated by a return value of 1.
9186 The function will keep track of the number of octets already written; you
9187 should call it repeatedly with the same values of <literal>buf</literal>
9188 and <literal>len</literal>, until the buffer has been transmitted.
9189 When a full buffer has been sent, the function will return 0 for
9190 success. The return value -1 indicates an error condition (see below).
9193 int cs_get(COMSTACK handle, char **buf, int *size);
9196 Receives a PDU or HTTP Response from the peer. Returns the number of
9198 In nonblocking mode, it is possible that not all of the packet can be
9199 read at once. In this case, the function returns 1. To simplify the
9200 interface, the function is
9201 responsible for managing the size of the buffer. It will be reallocated
9202 if necessary to contain large packages, and will sometimes be moved
9203 around internally by the subsystem when partial packages are read. Before
9205 <function>cs_get</function>
9206 for the first time, the buffer can be initialized to the null pointer,
9207 and the length should also be set to 0 (cs_get will perform a
9208 <function>malloc(2)</function>
9209 on the buffer for you). When a full buffer has been read, the size of
9210 the package is returned (which will always be greater than 1).
9211 The return value -1 indicates an error condition.
9214 See also the <function>cs_more()</function> function below.
9217 int cs_more(COMSTACK handle);
9220 The <function>cs_more()</function> function should be used in conjunction
9221 with <function>cs_get</function> and
9222 <function>select(2)</function>.
9223 The <function>cs_get()</function> function will sometimes
9224 (notably in the TCP/IP mode) read more than a single protocol package
9225 off the network. When this happens, the extra package is stored
9226 by the subsystem. After calling <function>cs_get()</function>, and before
9227 waiting for more input, You should always call
9228 <function>cs_more()</function>
9229 to check if there's a full protocol package already read. If
9230 <function>cs_more()</function>
9232 <function>cs_get()</function>
9233 can be used to immediately fetch the new package. For the
9235 subsystem, the function should always return 0, but if you want your
9236 stuff to be protocol independent, you should use it.
9240 The <function>cs_more()</function>
9241 function is required because the RFC1729-method
9242 does not provide a way of separating individual PDUs, short of
9243 partially decoding the BER. Some other implementations will carefully
9244 nibble at the packet by calling
9245 <function>read(2)</function>
9246 several times. This was felt to be too inefficient (or at least
9247 clumsy) - hence the call for this extra function.
9251 int cs_look(COMSTACK handle);
9254 This function is useful when you're operating in nonblocking
9256 <function>select(2)</function>
9257 tells you there's something happening on the line. It returns one of
9258 the following values:
9262 <term>CS_NONE</term>
9264 No event is pending. The data found on the line was not a
9269 <term>CS_CONNECT</term>
9271 A response to your connect request has been received. Call
9272 <function>cs_rcvconnect</function>
9273 to process the event and to finalize the connection establishment.
9277 <term>CS_DISCON</term>
9279 The other side has closed the connection (or maybe sent a disconnect
9280 request - but do we care? Maybe later). Call
9281 <function>cs_close</function> to close your end of the association
9286 <term>CS_LISTEN</term>
9288 A connect request has been received.
9289 Call <function>cs_listen</function> to process the event.
9293 <term>CS_DATA</term>
9295 There's data to be found on the line.
9296 Call <function>cs_get</function> to get it.
9302 You should be aware that even if
9303 <function>cs_look()</function>
9304 tells you that there's an event event pending, the corresponding
9305 function may still return and tell you there was nothing to be found.
9306 This means that only part of a package was available for reading. The
9307 same event will show up again, when more data has arrived.
9311 int cs_fileno(COMSTACK h);
9314 returns the file descriptor of the association. Use this when
9315 file-level operations on the endpoint are required
9316 (<function>select(2)</function> operations, specifically).
9320 <sect1 id="comstack.client">
9321 <title>Client Side</title>
9323 int cs_connect(COMSTACK handle, void *address);
9326 Initiate a connection with the target at <literal>address</literal>
9327 (more on addresses below). The function will return 0 on success, and 1 if
9328 the operation does not complete immediately (this will only
9329 happen on a nonblocking endpoint). In this case, use
9330 <function>cs_rcvconnect</function> to complete the operation,
9331 when <function>select(2)</function> or <function>poll(2)</function>
9332 reports input pending on the association.
9335 int cs_rcvconnect(COMSTACK handle);
9338 Complete a connect operation initiated by <function>cs_connect()</function>.
9339 It will return 0 on success; 1 if the operation has not yet completed (in
9340 this case, call the function again later); -1 if an error has occurred.
9343 <sect1 id="comstack.server">
9344 <title>Server Side</title>
9346 To establish a server under the <application>inetd</application>
9350 COMSTACK cs_createbysocket(int socket, CS_TYPE type, int blocking,
9354 The <literal>socket</literal> parameter is an established socket (when
9355 your application is invoked from <application>inetd</application>, the
9356 socket will typically be 0.
9357 The following parameters are identical to the ones for
9358 <function>cs_create</function>.
9361 int cs_bind(COMSTACK handle, void *address, int mode)
9364 Binds a local address to the endpoint. Read about addresses below. The
9365 <literal>mode</literal> parameter should be either
9366 <literal>CS_CLIENT</literal> or <literal>CS_SERVER</literal>.
9369 int cs_listen(COMSTACK handle, char *addr, int *addrlen);
9372 Call this to process incoming events on an endpoint that has been
9373 bound in listening mode. It will return 0 to indicate that the connect
9374 request has been received, 1 to signal a partial reception, and -1 to
9375 indicate an error condition.
9378 COMSTACK cs_accept(COMSTACK handle);
9381 This finalizes the server-side association establishment, after
9382 cs_listen has completed successfully. It returns a new connection
9383 endpoint, which represents the new association. The application will
9384 typically wish to fork off a process to handle the association at this
9385 point, and continue listen for new connections on the old
9386 <literal>handle</literal>.
9389 You can use the call
9392 const char *cs_addrstr(COMSTACK);
9395 on an established connection to retrieve the host-name of the remote host.
9399 You may need to use this function with some care if your
9400 name server service is slow or unreliable.
9404 <sect1 id="comstack.addresses">
9405 <title>Addresses</title>
9407 The low-level format of the addresses are different depending on the
9408 mode of communication you have chosen. A function is provided by each
9409 of the lower layers to map a user-friendly string-form address to the
9410 binary form required by the lower layers.
9413 void *cs_straddr(COMSTACK handle, const char *str);
9416 The format for TCP/IP and SSL addresses is:
9419 <host> [ ':' <portnum> ]
9422 The <literal>hostname</literal> can be either a domain name or an
9423 IP address. The port number, if omitted, defaults to 210.
9426 For TCP/IP and SSL, the special hostnames <literal>@</literal>,
9427 maps to <literal>IN6ADDR_ANY_INIT</literal> with
9428 IPV4 binding as well (bindv6only=0),
9429 The special hostname <literal>@4</literal> binds to
9430 <literal>INADDR_ANY</literal> (IPV4 only listener).
9431 The special hostname <literal>@6</literal> binds to
9432 <literal>IN6ADDR_ANY_INIT</literal> with bindv6only=1 (IPV6 only listener).
9435 For UNIX sockets, the format of an address is the socket filename.
9438 When a connection has been established, you can use
9441 const char *cs_addrstr(COMSTACK h);
9444 to retrieve the host name of the peer system. The function returns
9445 a pointer to a static area, which is overwritten on the next call
9449 A fairly recent addition to the &comstack; module is the utility
9453 COMSTACK cs_create_host (const char *str, int blocking, void **vp);
9456 which is just a wrapper for <function>cs_create</function> and
9457 <function>cs_straddr</function>. The <parameter>str</parameter>
9458 is similar to that described for <function>cs_straddr</function>
9459 but with a prefix denoting the &comstack; type. Prefixes supported
9460 are <literal>tcp:</literal> and <literal>unix:</literal> and
9461 <literal>ssl:</literal> for TCP/IP and UNIX and SSL respectively.
9462 If no prefix is given, then TCP/IP is used.
9463 The <parameter>blocking</parameter> is passed to
9464 function <function>cs_create</function>. The third parameter
9465 <parameter>vp</parameter> is a pointer to &comstack; stack type
9467 Parameter <parameter>vp</parameter> is reserved for future use.
9468 Set it to <literal>NULL</literal>.
9471 <sect1 id="comstack.ssl">
9475 void *cs_get_ssl(COMSTACK cs);
9477 Returns the SSL handle, <literal>SSL *</literal> for comstack. If comstack
9478 is not of type SSL, then NULL is returned.
9482 int cs_set_ssl_ctx(COMSTACK cs, void *ctx);
9484 Sets SSL context for comstack. The parameter is expected to be of type
9485 <literal>SSL_CTX *</literal>. This function should be called just
9486 after comstack has been created (before connect, bind, etc).
9487 This function returns 1 for success; 0 for failure.
9491 int cs_set_ssl_certificate_file(COMSTACK cs, const char *fname);
9493 Sets SSL certificate for comstack as a PEM file. This function
9494 returns 1 for success; 0 for failure.
9498 int cs_get_ssl_peer_certificate_x509(COMSTACK cs, char **buf, int *len);
9500 This function returns the peer certificate. If successful,
9501 <literal>*buf</literal> and <literal>*len</literal> holds
9502 X509 buffer and length respectively. Buffer should be freed
9503 with <literal>xfree</literal>. This function returns 1 for success;
9507 <sect1 id="comstack.diagnostics">
9508 <title>Diagnostics</title>
9510 All functions return -1 if an error occurs. Typically, the functions
9511 will return 0 on success, but the data exchange functions
9512 (<function>cs_get</function>, <function>cs_put</function>,
9513 <function>cs_more</function>) follow special rules. Consult their
9517 The error code for the COMSTACK can be retrieved using C macro
9518 <function>cs_errno</function> which will return one
9519 of the error codes <literal>CSYSERR</literal>,
9520 <literal>CSOUTSTATE</literal>,
9521 <literal>CSNODATA</literal>, ...
9524 int cs_errno(COMSTACK handle);
9527 You can the textual representation of the error code
9528 by using <function>cs_errmsg</function>, which
9529 works like <function>strerror(3)</function>.
9532 const char *cs_errmsg(int n);
9535 It is also possible to get straight to the textual representation
9536 without the error code, by using
9537 <function>cs_strerror</function>.
9540 const char *cs_strerror(COMSTACK h);
9543 <sect1 id="comstack.summary">
9544 <title>Summary and Synopsis</title>
9546 #include <yaz/comstack.h>
9548 #include <yaz/tcpip.h> /* this is for TCP/IP and SSL support */
9549 #include <yaz/unix.h> /* this is for UNIX socket support */
9551 COMSTACK cs_create(CS_TYPE type, int blocking, int protocol);
9553 COMSTACK cs_createbysocket(int s, CS_TYPE type, int blocking,
9555 COMSTACK cs_create_host(const char *str, int blocking,
9558 int cs_bind(COMSTACK handle, int mode);
9560 int cs_connect(COMSTACK handle, void *address);
9562 int cs_rcvconnect(COMSTACK handle);
9564 int cs_listen(COMSTACK handle);
9566 COMSTACK cs_accept(COMSTACK handle);
9568 int cs_put(COMSTACK handle, char *buf, int len);
9570 int cs_get(COMSTACK handle, char **buf, int *size);
9572 int cs_more(COMSTACK handle);
9574 void cs_close(COMSTACK handle);
9576 int cs_look(COMSTACK handle);
9578 void *cs_straddr(COMSTACK handle, const char *str);
9580 const char *cs_addrstr(COMSTACK h);
9585 <chapter id="future">
9586 <title>Future Directions</title>
9588 We have a new and better version of the front-end server on the drawing
9589 board. Resources and external commitments will govern when we'll be
9590 able to do something real with it. Features should include greater
9591 flexibility, greater support for access/resource control, and easy
9592 support for Explain (possibly with Zebra as an extra database engine).
9595 &yaz; is a BER toolkit and as such should support all protocols
9596 out there based on that. We'd like to see running ILL applications.
9597 It shouldn't be that hard. Another thing that would be interesting is
9598 LDAP. Maybe a generic framework for doing IR using both LDAP and
9599 Z39.50 transparently.
9602 The SOAP implementation is incomplete. In the future we hope
9603 to add more features to it. Perhaps make a WSDL/XML Schema compiler.
9604 The authors of libxml2 are already working on XML Schema / RelaxNG
9605 compilers so this may not be too hard.
9608 It would be neat to have a proper module mechanism for the Generic
9609 Frontend Server so that backend would be dynamically
9610 loaded (as shared objects / DLLs).
9613 Other than that, &yaz; generally moves in the directions which appear to
9614 make the most people happy (including ourselves, as prime users of the
9615 software). If there's something you'd like to see in here, then drop
9616 us a note and let's see what we can come up with.
9619 <reference id="reference">
9620 <title>Reference</title>
9621 <partintro id="reference-introduction">
9623 The material in this chapter is drawn directly from the individual
9629 <appendix id="list-oids">
9630 <title>List of Object Identifiers</title>
9632 These is a list of object identifiers that are built into YAZ.
9636 <appendix id="bib1-diagnostics">
9637 <title>Bib-1 diagnostics</title>
9639 List of Bib-1 diagnostics that are known to YAZ.
9643 <appendix id="sru-diagnostics">
9644 <title>SRU diagnostics</title>
9646 List of SRU diagnostics that are known to YAZ.
9650 <appendix id="license">
9651 <title>License</title>
9652 <sect1 id="license.indexdata">
9653 <title>Index Data Copyright</title>
9655 Copyright © ©right-year; Index Data.
9658 All rights reserved.
9661 Redistribution and use in source and binary forms, with or without
9662 modification, are permitted provided that the following conditions are met:
9667 Redistributions of source code must retain the above copyright
9668 notice, this list of conditions and the following disclaimer.
9673 Redistributions in binary form must reproduce the above copyright
9674 notice, this list of conditions and the following disclaimer in the
9675 documentation and/or other materials provided with the distribution.
9680 Neither the name of Index Data nor the names of its contributors
9681 may be used to endorse or promote products derived from this
9682 software without specific prior written permission.
9687 THIS SOFTWARE IS PROVIDED BY INDEX DATA ``AS IS'' AND ANY
9688 EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
9689 WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
9690 DISCLAIMED. IN NO EVENT SHALL INDEX DATA BE LIABLE FOR
9691 ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
9692 DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
9693 SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
9694 CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
9695 LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
9696 OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
9701 <appendix id="indexdata">
9702 <title>About Index Data</title>
9704 Index Data is a consulting and software-development enterprise that
9705 specializes in library and information management systems. Our
9706 interests and expertise span a broad range of related fields, and one
9707 of our primary, long-term objectives is the development of a powerful
9708 information management
9709 system with open network interfaces and hyper-media capabilities.
9711 We make this software available free of charge, on a fairly unrestrictive
9712 license; as a service to the networking community, and to further the
9713 development of quality software for open network communication.
9715 We'll be happy to answer questions about the software, and about ourselves
9721 <street>Amagerfælledvej 56</street>
9722 <postcode>2300 Copenhagen S</postcode>
9723 <country>Denmark</country>
9724 Email <email>info@indexdata.dk</email>
9728 The Hacker's Jargon File has the following to say about the
9730 prefix "YA" in the name of a software product.
9734 Yet Another. adj. 1. Of your own work: A
9735 humorous allusion often used in titles to acknowledge that the
9736 topic is not original, though the content is. As in "Yet Another
9737 AI Group" or "Yet Another Simulated Annealing Algorithm".
9739 others' work: Describes something of which there are already far
9744 <appendix id="credits">
9745 <title>Credits</title>
9747 This appendix lists individuals that have contributed in the development
9748 of &yaz;. Some have contributed with code, while others have provided bug
9749 fixes or suggestions. If we're missing somebody, of if you, for
9750 whatever reason, don't like to be listed here, let us know.
9760 Morten Bøgeskov
9781 Mads Bondo Dydensborg
9790 Morten Garkier Hendriksen
9847 Tom André Øverland
9853 <!-- Keep this comment at the end of the file
9856 nxml-child-indent: 1