1 <?xml version="1.0" standalone="no"?>
2 <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1//EN"
3 "http://www.oasis-open.org/docbook/xml/4.1/docbookx.dtd"
5 <!ENTITY % local SYSTEM "local.ent">
7 <!ENTITY manref SYSTEM "manref.xml">
8 <!ENTITY progref SYSTEM "progref.xml">
9 <!ENTITY % idcommon SYSTEM "common/common.ent">
11 <!-- Next line allows imagedata/@format="PDF" and is taken from
12 http://lists.oasis-open.org/archives/docbook/200303/msg00163.html
14 <!ENTITY % local.notation.class "| PDF">
15 <!-- Next line is necessary for some XML parsers, for reasons I
16 don't understand. I got this from
17 http://lists.oasis-open.org/archives/docbook/200303/msg00180.html
19 <!NOTATION PDF SYSTEM "PDF">
21 <!-- $Id: book.xml,v 1.61 2007-09-18 18:43:17 adam Exp $ -->
24 <title>Metaproxy - User's Guide and Reference</title>
27 <firstname>Adam</firstname><surname>Dickmeiss</surname>
30 <firstname>Marc</firstname><surname>Cromme</surname>
33 <firstname>Mike</firstname><surname>Taylor</surname>
36 <releaseinfo>&version;</releaseinfo>
38 <year>2005-2007</year>
39 <holder>Index Data ApS</holder>
43 This manual is part of Metaproxy version &version;.
46 Metaproxy is a universal router, proxy and encapsulated
47 metasearcher for information retrieval protocols. It accepts,
48 processes, interprets and redirects requests from IR clients using
49 standard protocols such as the binary
50 <ulink url="&url.z39.50;">ANSI/NISO Z39.50</ulink>
51 and the information search and retireval
52 web services <ulink url="&url.sru;">SRU</ulink>
53 and <ulink url="&url.srw;">SRW</ulink>, as
54 well as functioning as a limited
55 <ulink url="&url.http;">HTTP</ulink> server.
58 Metaproxy is configured by an XML file which
59 specifies how the software should function in terms of routes that
60 the request packets can take through the proxy, each step on a
61 route being an instantiation of a filter. Filters come in many
62 types, one for each operation: accepting Z39.50 packets, logging,
63 query transformation, multiplexing, etc. Further filter-types can
64 be added as loadable modules to extend Metaproxy functionality,
68 Metaproxy is covered by the GNU General Public License version 2.
73 <imagedata fileref="common/id.png" format="PNG"/>
76 <imagedata fileref="common/id.eps" format="EPS"/>
83 <chapter id="introduction">
84 <title>Introduction</title>
87 <ulink url="&url.metaproxy;">Metaproxy</ulink>
88 is a stand alone program that acts as a universal router, proxy and
89 encapsulated metasearcher for information retrieval protocols such
90 as <ulink url="&url.z39.50;">Z39.50</ulink>, and in the future
91 <ulink url="&url.sru;">SRU</ulink> and <ulink url="&url.srw;">SRW</ulink>.
92 To clients, it acts as a server of these protocols: it can be searched,
93 records can be retrieved from it, etc.
94 To servers, it acts as a client: it searches in them,
95 retrieves records from them, etc. it satisfies its clients'
96 requests by transforming them, multiplexing them, forwarding them
97 on to zero or more servers, merging the results, transforming
98 them, and delivering them back to the client. In addition, it
99 acts as a simple <ulink url="&url.http;">HTTP</ulink> server; support
100 for further protocols can be added in a modular fashion, through the
101 creation of new filters.
106 Fish, bananas, cold pyjamas,
107 Mutton, beef and trout!
108 - attributed to Cole Porter.
111 Metaproxy is a more capable alternative to
112 <ulink url="&url.yazproxy;">YAZ Proxy</ulink>,
113 being more powerful, flexible, configurable and extensible. Among
114 its many advantages over the older, more pedestrian work are
115 support for multiplexing (encapsulated metasearching), routing by
116 database name, authentication and authorization and serving local
117 files via HTTP. Equally significant, its modular architecture
118 facilitites the creation of pluggable modules implementing further
122 This manual will describe how to install Metaproxy
123 before giving an overview of its architecture, then discussing the
124 key concept of a filter in some depth and giving an overview of
125 the various filter types, then discussing the configuration file
126 format. After this come several optional chapters which may be
127 freely skipped: a detailed discussion of virtual databases and
128 multi-database searching, some notes on writing extensions
129 (additional filter types) and a high-level description of the
130 source code. Finally comes the reference guide, which contains
131 instructions for invoking the <command>metaproxy</command>
132 program, and detailed information on each type of filter,
137 <chapter id="installation">
138 <title>Installation</title>
140 Metaproxy depends on the following tools/libraries:
142 <varlistentry><term><ulink url="&url.yazplusplus;">YAZ++</ulink></term>
145 This is a C++ library based on <ulink url="&url.yaz;">YAZ</ulink>.
149 <varlistentry><term><ulink url="&url.libxslt;">Libxslt</ulink></term>
151 <para>This is an XSLT processor - based on
152 <ulink url="&url.libxml2;">Libxml2</ulink>. Both Libxml2 and
153 Libxslt must be installed with the development components
154 (header files, etc.) as well as the run-time libraries.
158 <varlistentry><term><ulink url="&url.boost;">Boost</ulink></term>
161 The popular C++ library. Initial versions of Metaproxy
162 was built with 1.33.0. Version 1.33.1 works too. Metaproxy
163 also compiles with Boost 1.34.1 but the tests do
164 <emphasis>not</emphasis> compile, because the Test API
165 of Boost has changed.
172 In order to compile Metaproxy a modern C++ compiler is
173 required. Boost, in particular, requires the C++ compiler
174 to facilitate the newest features. Refer to Boost
175 <ulink url="&url.boost.compilers.status;">Compiler Status</ulink>
176 for more information.
179 We have successfully built Metaproxy using the compilers
180 <ulink url="&url.gcc;">GCC</ulink> version 4.0 and
181 <ulink url="&url.vstudio;">Microsoft Visual Studio</ulink> 2003/2005.
184 <section id="installation.unix">
185 <title>Installation on Unix (from Source)</title>
187 Here is a quick step-by-step guide on how to compile all the
188 tools that Metaproxy uses. Only few systems have none of the required
189 tools binary packages. If, for example, Libxml2/libxslt are already
190 installed as development packages use those (and omit compilation).
197 gunzip -c libxml2-version.tar.gz|tar xf -
205 gunzip -c libxslt-version.tar.gz|tar xf -
216 gunzip -c yaz-version.tar.gz|tar xf -
224 gunzip -c yazpp-version.tar.gz|tar xf -
235 gunzip -c boost-version.tar.gz|tar xf -
246 gunzip -c metaproxy-version.tar.gz|tar xf -
255 <section id="installation.debian">
256 <title>Installation on Debian GNU/Linux</title>
258 All dependencies for Metaproxy are available as
259 <ulink url="&url.debian;">Debian</ulink>
260 packages for the sarge (stable in 2005) and etch (testing in 2005)
264 The procedures for Debian based systems, such as
265 <ulink url="&url.ubuntu;">Ubuntu</ulink> is probably similar
268 There is currently no official Debian package for YAZ++.
269 And the official Debian package for YAZ is probably too old.
270 But Index Data bulds "new" versions of those for Debian (i386 only).
273 Update the <filename>/etc/apt/sources.list</filename>
274 to include the Index Data repository.
275 See YAZ' <ulink url="&url.yaz.download.debian;">Download Debian</ulink>
276 for more information.
279 apt-get install libxslt1-dev
280 apt-get install libyazpp2-dev
281 apt-get install libboost-dev
282 apt-get install libboost-thread-dev
283 apt-get install libboost-date-time-dev
284 apt-get install libboost-program-options-dev
285 apt-get install libboost-test-dev
288 With these packages installed, the usual configure + make
289 procedure can be used for Metaproxy as outlined in
290 <xref linkend="installation.unix"/>.
294 <section id="installation.rpm">
295 <title>Installation on RPM based Linux Systems</title>
297 All external dependencies for Metaproxy are available as
298 RPM packages, either from your distribution site, or from the
299 <ulink url="http://fr.rpmfind.net/">RPMfind</ulink> site.
302 For example, an installation of the requires Boost C++ development
303 libraries on RedHat Fedora C4 and C5 can be done like this:
305 wget ftp://fr.rpmfind.net/wlinux/fedora/core/updates/testing/4/SRPMS/boost-1.33.0-3.fc4.src.rpm
306 sudo rpmbuild --buildroot src/ --rebuild -p fc4/boost-1.33.0-3.fc4.src.rpm
307 sudo rpm -U /usr/src/redhat/RPMS/i386/boost-*rpm
311 The <ulink url="&url.yaz;">YAZ</ulink> library is needed to
312 compile &metaproxy;, see there
313 for more information on available RPM packages.
316 There is currently no official RPM package for YAZ++.
317 See the <ulink url="&url.yazplusplus;">YAZ++</ulink> pages
318 for more information on a Unix tarball install.
321 With these packages installed, the usual configure + make
322 procedure can be used for Metaproxy as outlined in
323 <xref linkend="installation.unix"/>.
327 <section id="installation.windows">
328 <title>Installation on Windows</title>
330 Metaproxy can be compiled with Microsoft
331 <ulink url="&url.vstudio;">Visual Studio</ulink>.
332 Version 2003 (C 7.1) and 2005 (C 8.0) is known to work.
334 <section id="installation.windows.boost">
337 For Windows, it's easiest to get the precompiled Boost
338 package from <ulink url="&url.boost.windows.download;">here</ulink>.
339 Several versions of the Boost libraries may be selected when
340 installing Boost for windows. Please choose at least the
341 <emphasis>multithreaded</emphasis> (non-DLL) version because
342 the Metaproxy makefile uses that.
345 For more information about installing Boost refer to the
346 <ulink url="&url.boost.getting.started;">getting started</ulink>
351 <section id="installation.windows.libxslt">
352 <title>Libxslt</title>
354 <ulink url="&url.libxslt;">Libxslt</ulink> can be downloaded
356 <ulink url="&url.libxml2.download.win32;">here</ulink>.
359 Libxslt has other dependencies, but these can all be downloaded
360 from the same site. Get the following package:
361 iconv, zlib, libxml2, libxslt.
365 <section id="installation.windows.yaz">
368 <ulink url="&url.yaz;">YAZ</ulink> can be downloaded
370 <ulink url="&url.yaz.download.win32;">here</ulink>.
374 <section id="installation.windows.yazplusplus">
377 Get <ulink url="&url.yazplusplus;">YAZ++</ulink> as well.
378 Version 1.0.3 or later is required.
381 YAZ++ includes NMAKE makefiles, similar to those found in the
386 <section id="installation.windows.metaproxy">
387 <title>Metaproxy</title>
389 Metaproxy is shipped with NMAKE makefiles as well - similar
390 to those found in the YAZ++/YAZ packages. Adjust this Makefile
391 to point to the proper locations of Boost, Libxslt, Libxml2,
392 zlib, iconv, yaz and yazpp.
396 <varlistentry><term><literal>DEBUG</literal></term>
398 If set to 1, the software is
399 compiled with debugging libraries (code generation is
400 multi-threaded debug DLL).
401 If set to 0, the software is compiled with release libraries
402 (code generation is multi-threaded DLL).
407 <term><literal>BOOST</literal></term>
410 Boost install location
416 <term><literal>BOOST_VERSION</literal></term>
419 Boost version (replace . with _).
425 <term><literal>BOOST_TOOLSET</literal></term>
434 <term><literal>LIBXSLT_DIR</literal>,
435 <literal>LIBXML2_DIR</literal> ..</term>
438 Specify the locations of Libxslt, libiconv, libxml2 and
447 After successful compilation you'll find
448 <literal>metaproxy.exe</literal> in the
449 <literal>bin</literal> directory.
457 <chapter id="yazproxy-comparison">
458 <title>YAZ Proxy Comparison</title>
460 The table below lists facilities either supported by either
461 <ulink url="&url.yazproxy;">YAZ Proxy</ulink> or Metaproxy.
463 <table id="yazproxy-comparison-table">
464 <title>Metaproxy / YAZ Proxy comparison</title>
468 <entry>Facility</entry>
469 <entry>Metaproxy</entry>
470 <entry>YAZ Proxy</entry>
475 <entry>Z39.50 server</entry>
476 <entry>Using filter <literal>frontend_net</literal></entry>
477 <entry>Supported</entry>
480 <entry>SRU server</entry>
481 <entry>Supported with filter <literal>sru_z3950</literal></entry>
482 <entry>Supported</entry>
485 <entry>Z39.50 client</entry>
486 <entry>Supported with filter <literal>z3950_client</literal></entry>
487 <entry>Supported</entry>
490 <entry>SRU client</entry>
491 <entry>Unsupported</entry>
492 <entry>Unsupported</entry>
495 <entry>Connection reuse</entry>
496 <entry>Supported with filter <literal>session_shared</literal></entry>
497 <entry>Supported</entry>
500 <entry>Connection share</entry>
501 <entry>Supported with filter <literal>session_shared</literal></entry>
502 <entry>Unsupported</entry>
505 <entry>Result set reuse</entry>
506 <entry>Supported with filter <literal>session_shared</literal></entry>
507 <entry>Within one Z39.50 session / HTTP keep-alive</entry>
510 <entry>Record cache</entry>
511 <entry>Unsupported</entry>
512 <entry>Supported for last result set within one Z39.50/HTTP-keep alive session</entry>
515 <entry>Z39.50 Virtual database, i.e. select any Z39.50 target for database</entry>
516 <entry>Supported with filter <literal>virt_db</literal></entry>
517 <entry>Unsupported</entry>
520 <entry>SRU Virtual database, i.e. select any Z39.50 target for path</entry>
521 <entry>Supported with filter <literal>virt_db</literal>,
522 <literal>sru_z3950</literal></entry>
523 <entry>Supported</entry>
526 <entry>Multi target search</entry>
527 <entry>Supported with filter <literal>multi</literal> (round-robin)</entry>
528 <entry>Unsupported</entry>
531 <entry>Retrieval and search limits</entry>
532 <entry>Unsupported</entry>
533 <entry>Supported</entry>
536 <entry>Bandwidth limits</entry>
537 <entry>Unsupported</entry>
538 <entry>Supported</entry>
541 <entry>Connect limits</entry>
542 <entry>Unsupported</entry>
543 <entry>Supported</entry>
546 <entry>Retrieval sanity check and conversions</entry>
547 <entry>Supported using filter <literal>record_transform</literal></entry>
548 <entry>Supported</entry>
551 <entry>Query check</entry>
553 Supported in a limited way using <literal>query_rewrite</literal>
555 <entry>Supported</entry>
558 <entry>Query rewrite</entry>
559 <entry>Supported with <literal>query_rewrite</literal></entry>
560 <entry>Unsupported</entry>
563 <entry>Session invalidate for -1 hits</entry>
564 <entry>Unsupported</entry>
565 <entry>Supported</entry>
568 <entry>Architecture</entry>
569 <entry>Multi-threaded + select for networked modules such as
570 <literal>frontend_net</literal>)</entry>
571 <entry>Single-threaded using select</entry>
575 <entry>Extensability</entry>
576 <entry>Most functionality implemented as loadable modules</entry>
577 <entry>Unsupported and experimental</entry>
581 <entry><ulink url="&url.usemarcon;">USEMARCON</ulink></entry>
582 <entry>Unsupported</entry>
583 <entry>Supported</entry>
587 <entry>Portability</entry>
589 Requires YAZ, YAZ++ and modern C++ compiler supporting
590 <ulink url="&url.boost;">Boost</ulink>.
593 Requires YAZ and YAZ++.
594 STL is not required so pretty much any C++ compiler out there should work.
603 <chapter id="architecture">
604 <title>The Metaproxy Architecture</title>
606 The Metaproxy architecture is based on three concepts:
607 the <emphasis>package</emphasis>,
608 the <emphasis>route</emphasis>
609 and the <emphasis>filter</emphasis>.
613 <term>Packages</term>
616 A package is request or response, encoded in some protocol,
617 issued by a client, making its way through Metaproxy, send to or
618 received from a server, or sent back to the client.
621 The core of a package is the protocol unit - for example, a
622 Z39.50 Init Request or Search Response, or an SRU searchRetrieve
623 URL or Explain Response. In addition to this core, a package
624 also carries some extra information added and used by Metaproxy
628 In general, packages are doctored as they pass through
629 Metaproxy. For example, when the proxy performs authentication
630 and authorization on a Z39.50 Init request, it removes the
631 authentication credentials from the package so that they are not
632 passed onto the back-end server; and when search-response
633 packages are obtained from multiple servers, they are merged
634 into a single unified package that makes its way back to the
643 Packages make their way through routes, which can be thought of
644 as programs that operate on the package data-type. Each
645 incoming package initially makes its way through a default
646 route, but may be switched to a different route based on various
647 considerations. Routes are made up of sequences of filters (see
656 Filters provide the individual instructions within a route, and
657 effect the necessary transformations on packages. A particular
658 configuration of Metaproxy is essentially a set of filters,
659 described by configuration details and arranged in order in one
660 or more routes. There are many kinds of filter - about a dozen
661 at the time of writing with more appearing all the time - each
662 performing a specific function and configured by different
666 The word ``filter'' is sometimes used rather loosely, in two
667 different ways: it may be used to mean a particular
668 <emphasis>type</emphasis> of filter, as when we speak of ``the
669 auth_simple filter'' or ``the multi filter''; or it may be used
670 to be a specific <emphasis>instance</emphasis> of a filter
671 within a Metaproxy configuration. For example, a single
672 configuration will often contain multiple instances of the
673 <literal>z3950_client</literal> filter. In
674 operational terms, of these is a separate filter. In practice,
675 context always make it clear which sense of the word ``filter''
679 Extensibility of Metaproxy is primarily through the creation of
680 plugins that provide new filters. The filter API is small and
681 conceptually simple, but there are many details to master. See
683 <link linkend="filters">Filters</link>.
689 Since packages are created and handled by the system itself, and
690 routes are conceptually simple, most of the remainder of this
691 document concentrates on filters. After a brief overview of the
692 filter types follows, along with some thoughts on possible future
699 <chapter id="filters">
700 <title>Filters</title>
703 <section id="filters-introductory-notes">
704 <title>Introductory notes</title>
706 It's useful to think of Metaproxy as an interpreter providing a small
707 number of primitives and operations, but operating on a very
708 complex data type, namely the ``package''.
711 A package represents a Z39.50 or SRU/W request (whether for Init,
712 Search, Scan, etc.) together with information about where it came
713 from. Packages are created by front-end filters such as
714 <literal>frontend_net</literal> (see below), which reads them from
715 the network; other front-end filters are possible. They then pass
716 along a route consisting of a sequence of filters, each of which
717 transforms the package and may also have side-effects such as
718 generating logging. Eventually, the route will yield a response,
719 which is sent back to the origin.
722 There are many kinds of filter: some that are defined statically
723 as part of Metaproxy, and others may be provided by third parties
724 and dynamically loaded. They all conform to the same simple API
725 of essentially two methods: <function>configure()</function> is
726 called at startup time, and is passed an XML DOM tree representing that
727 part of the configuration file that pertains to this filter
728 instance: it is expected to walk that tree extracting relevant
729 information; and <function>process()</function> is called every
730 time the filter has to processes a package.
733 While all filters provide the same API, there are different modes
734 of functionality. Some filters are sources: they create
736 (<literal>frontend_net</literal>);
737 others are sinks: they consume packages and return a result
738 (<literal>backend_test</literal>,
739 <literal>bounce</literal>,
740 <literal>http_file</literal>,
741 <literal>z3950_client</literal>);
742 the others are true filters, that read, process and pass on the
743 packages they are fed
744 (<literal>auth_simple</literal>,
745 <literal>log</literal>,
746 <literal>multi</literal>,
747 <literal>query_rewrite</literal>,
748 <literal>record_transform</literal>,
749 <literal>session_shared</literal>,
750 <literal>sru_z3950</literal>,
751 <literal>template</literal>,
752 <literal>virt_db</literal>).
757 <section id="overview.filter.types">
758 <title>Overview of filter types</title>
760 We now briefly consider each of the types of filter supported by
761 the core Metaproxy binary. This overview is intended to give a
762 flavor of the available functionality; more detailed information
763 about each type of filter is included below in
764 <xref linkend="reference"/>.
767 The filters are here named by the string that is used as the
768 <literal>type</literal> attribute of a
769 <literal><filter></literal> element in the configuration
770 file to request them, with the name of the class that implements
771 them in parentheses. (The classname is not needed for normal
772 configuration and use of Metaproxy; it is useful only to
776 The filters are here listed in alphabetical order:
783 New virt_db-alike that does inteligent peer choice, explain merging,
784 adds FD&N to explain. Keeps init responses (like "virt_db Classic"),
785 makes routing choices based on local explain knowledge. Ref IDDI
788 Filter to convert Explain Classic to ZeeRex.
790 CQL2PQF (which needs augmented ZeeRex) - MARC for Talis.
794 Figure out what additional information we need in:
795 ZeeRex (check against D3.1)
796 Init request (e.g. loop detection)
797 Query package (e.g. number of hops)
798 Query response (e.g. record source)
802 <section id="auth_simple">
803 <title><literal>auth_simple</literal>
804 (mp::filter::AuthSimple)</title>
806 Simple authentication and authorization. The configuration
807 specifies the name of a file that is the user register, which
808 lists <varname>username</varname>:<varname>password</varname>
809 pairs, one per line, colon separated. When a session begins, it
810 is rejected unless username and passsword are supplied, and match
811 a pair in the register. The configuration file may also specific
812 the name of another file that is the target register: this lists
813 lists <varname>username</varname>:<varname>dbname</varname>,<varname>dbname</varname>...
814 sets, one per line, with multiple database names separated by
815 commas. When a search is processed, it is rejected unless the
816 database to be searched is one of those listed as available to
821 <section id="backend_test">
822 <title><literal>backend_test</literal>
823 (mp::filter::Backend_test)</title>
825 A partial sink that provides dummy responses in the manner of the
826 <literal>yaz-ztest</literal> Z39.50 server. This is useful only
827 for testing. Seriously, you don't need this. Pretend you didn't
828 even read this section.
832 <section id="bounce">
833 <title><literal>bounce</literal>
834 (mp::filter::Bounce)</title>
836 A sink that swallows <emphasis>all packages</emphasis>,
837 and returns them almost unprocessed.
838 It never sends any package of any type further down the row, but
839 sets Z39.50 packages to Z_Close, and HTTP_Request packages to
840 HTTP_Response err code 400 packages, and adds a suitable bounce
842 The bounce filter is usually added at end of each filter chain route
843 to prevent infinite hanging of for example HTTP
844 requests packages when only the Z39.50 client partial sink
845 filter is found in the
850 <section id="cql_rpn">
851 <title><literal>cql_rpn</literal>
852 (mp::filter::CQLtoRPN)</title>
854 A query language transforming filter which catches Z39.50
855 <literal>searchRequest</literal>
856 packages containing <literal>CQL</literal> queries, transforms
857 those to <literal>RPN</literal> queries,
858 and sends the <literal>searchRequests</literal> on to the next
859 filters. It is among other things useful in a SRU context.
863 <section id="frontend_net">
864 <title><literal>frontend_net</literal>
865 (mp::filter::FrontendNet)</title>
867 A source that accepts Z39.50 connections from a port
868 specified in the configuration, reads protocol units, and
869 feeds them into the next filter in the route. When the result is
870 received, it is returned to the original origin.
874 <section id="http_file">
875 <title><literal>http_file</literal>
876 (mp::filter::HttpFile)</title>
878 A partial sink which swallows only
879 <literal>HTTP_Request</literal> packages, and
880 returns the contents of files from the local
881 filesystem in response to HTTP requests.
882 It lets Z39.50 packages and all other forthcoming package types
885 does mean that Metaproxy is also a Web-server in its spare time. So
886 far it does not contain either an email-reader or a Lisp
887 interpreter, but that day is surely coming.)
891 <section id="load_balance">
892 <title><literal>load_balance</literal>
893 (mp::filter::LoadBalance)</title>
895 Performs load balancing for incoming Z39.50 init requests.
896 It is used together with the <literal>virt_db</literal> filter,
897 but unlike the <literal>multi</literal> filter it does send an
898 entire session to only one of the virtual backends. The
899 <literal>load_balance</literal> filter is assuming that
900 all backend targets have equal content, and chooses the backend
901 with least load cost for a new session.
904 This filter is experimental and yet not mature for heavy load
912 <title><literal>log</literal>
913 (mp::filter::Log)</title>
915 Writes logging information to standard output, and passes on
916 the package unchanged. A log file name can be specified, as well
917 as multiple different logging formats.
922 <title><literal>multi</literal>
923 (mp::filter::Multi)</title>
925 Performs multi-database searching.
927 <link linkend="multidb">the extended discussion</link>
928 of virtual databases and multi-database searching below.
932 <section id="query_rewrite">
933 <title><literal>query_rewrite</literal>
934 (mp::filter::QueryRewrite)</title>
936 Rewrites Z39.50 <literal>Type-1</literal>
937 and <literal>Type-101</literal> (``<literal>RPN</literal>'')
939 three-step process: the query is transliterated from Z39.50
940 packet structures into an XML representation; that XML
941 representation is transformed by an XSLT stylesheet; and the
942 resulting XML is transliterated back into the Z39.50 packet
948 <section id="record_transform">
949 <title><literal>record_transform</literal>
950 (mp::filter::RecordTransform)</title>
952 This filter acts only on Z3950 present requests, and let all
953 other types of packages and requests pass untouched. It's use is
954 twofold: blocking Z3950 present requests, which the backend
955 server does not understand and can not honor, and transforming
956 the present syntax and elementset name according to the rules
957 specified, to fetch only existing record formats, and transform
958 them on the fly to requested record syntaxes.
962 <section id="session_shared">
963 <title><literal>session_shared</literal>
964 (mp::filter::SessionShared)</title>
966 This filter implements global sharing of
967 result sets (i.e. between threads and therefore between
968 clients), yielding performance improvements by clever resource
973 <section id="sru_z3950">
974 <title><literal>sru_z3950</literal>
975 (mp::filter::SRUtoZ3950)</title>
977 This filter transforms valid
978 SRU GET/POST/SOAP searchRetrieve requests to Z3950 init, search,
979 and present requests, and wraps the
980 received hit counts and XML records into suitable SRU response
982 The <literal>sru_z3950</literal> filter processes also SRU
983 GET/POST/SOAP explain requests, returning
984 either the absolute minimum required by the standard, or a full
985 pre-defined ZeeReX explain record.
987 <ulink url="&url.zeerex.explain;">ZeeReX Explain</ulink>
988 standard pages and the
989 <ulink url="&url.sru.explain;">SRU Explain</ulink> pages
990 for more information on the correct explain syntax.
991 SRU scan requests are not supported yet.
995 <section id="template">
996 <title><literal>template</literal>
997 (mp::filter::Template)</title>
999 Does nothing at all, merely passing the packet on. (Maybe it
1000 should be called <literal>nop</literal> or
1001 <literal>passthrough</literal>?) This exists not to be used, but
1002 to be copied - to become the skeleton of new filters as they are
1003 written. As with <literal>backend_test</literal>, this is not
1004 intended for civilians.
1008 <section id="virt_db">
1009 <title><literal>virt_db</literal>
1010 (mp::filter::VirtualDB)</title>
1012 Performs virtual database selection: based on the name of the
1013 database in the search request, a server is selected, and its
1014 address added to the request in a <literal>VAL_PROXY</literal>
1015 otherInfo packet. It will subsequently be used by a
1016 <literal>z3950_client</literal> filter.
1018 <link linkend="multidb">the extended discussion</link>
1019 of virtual databases and multi-database searching below.
1023 <section id="z3950_client">
1024 <title><literal>z3950_client</literal>
1025 (mp::filter::Z3950Client)</title>
1027 A partial sink which swallows only Z39.50 packages.
1028 It performs Z39.50 searching and retrieval by proxying the
1029 packages that are passed to it. Init requests are sent to the
1030 address specified in the <literal>VAL_PROXY</literal> otherInfo
1031 attached to the request: this may have been specified by client,
1032 or generated by a <literal>virt_db</literal> filter earlier in
1033 the route. Subsequent requests are sent to the same address,
1034 which is remembered at Init time in a Session object.
1035 HTTP_Request packages and all other forthcoming package types
1036 are passed untouched.
1041 <section id="zeerex_explain">
1042 <title><literal>zeerex_explain</literal>
1043 (mp::filter::ZeerexExplain)</title>
1045 This filter acts as a sink for
1046 Z39.50 explain requests, returning a static ZeeReX
1047 Explain XML record from the config section. All other packages
1050 <ulink url="&url.zeerex.explain;">ZeeReX Explain</ulink>
1052 for more information on the correct explain syntax.
1056 This filter is not yet completed.
1065 <section id="future.directions">
1066 <title>Future directions</title>
1068 Some other filters that do not yet exist, but which would be
1069 useful, are briefly described. These may be added in future
1070 releases (or may be created by third parties, as loadable
1076 <term><literal>frontend_cli</literal> (source)</term>
1079 Command-line interface for generating requests.
1084 <term><literal>sru_client</literal> (sink)</term>
1087 SRU/GET and SRU/SOAP searching and retrieval.
1092 <term><literal>opensearch_client</literal> (sink)</term>
1095 A9 OpenSearch searching and retrieval.
1105 <chapter id="configuration">
1106 <title>Configuration: the Metaproxy configuration file format</title>
1109 <section id="configuration-introductory-notes">
1110 <title>Introductory notes</title>
1112 If Metaproxy is an interpreter providing operations on packages, then
1113 its configuration file can be thought of as a program for that
1114 interpreter. Configuration is by means of a single XML file, the name
1115 of which is supplied as the sole command-line argument to the
1116 <command>metaproxy</command> program. (See
1117 <xref linkend="reference"/> below for more information on invoking
1122 <section id="overview.xml.structure">
1123 <title>Overview of the config file XML structure</title>
1125 All elements and attributes are in the namespace
1126 <ulink url="http://indexdata.com/metaproxy"/>.
1127 This is most easily achieved by setting the default namespace on
1128 the top-level element, as here:
1131 <metaproxy xmlns="http://indexdata.com/metaproxy" version="1.0">
1134 The top-level element is <metaproxy>. This contains a
1135 <start> element, a <filters> element and a
1136 <routes> element, in that order. <filters> is
1137 optional; the other two are mandatory. All three are
1141 The <start> element is empty, but carries a
1142 <literal>route</literal> attribute, whose value is the name of
1143 route at which to start running - analogous to the name of the
1144 start production in a formal grammar.
1147 If present, <filters> contains zero or more <filter>
1148 elements. Each filter carries a <literal>type</literal> attribute
1149 which specifies what kind of filter is being defined
1150 (<literal>frontend_net</literal>, <literal>log</literal>, etc.)
1151 and contain various elements that provide suitable configuration
1152 for a filter of its type. The filter-specific elements are
1154 <xref linkend="reference"/>.
1155 Filters defined in this part of the file must carry an
1156 <literal>id</literal> attribute so that they can be referenced
1160 <routes> contains one or more <route> elements, each
1161 of which must carry an <literal>id</literal> element. One of the
1162 routes must have the ID value that was specified as the start
1163 route in the <start> element's <literal>route</literal>
1164 attribute. Each route contains zero or more <filter>
1165 elements. These are of two types. They may be empty, but carry a
1166 <literal>refid</literal> attribute whose value is the same as the
1167 <literal>id</literal> of a filter previously defined in the
1168 <filters> section. Alternatively, a route within a filter
1169 may omit the <literal>refid</literal> attribute, but contain
1170 configuration elements similar to those used for filters defined
1171 in the <filters> section. (In other words, each filter in a
1172 route may be included either by reference or by physical
1178 <section id="example.configuration">
1179 <title>An example configuration</title>
1181 The following is a small, but complete, Metaproxy configuration
1182 file (included in the distribution as
1183 <literal>metaproxy/etc/config1.xml</literal>).
1184 This file defines a very simple configuration that simply proxies
1185 to whatever back-end server the client requests, but logs each
1186 request and response. This can be useful for debugging complex
1187 client-server dialogues.
1189 <screen><![CDATA[<?xml version="1.0"?>
1190 <metaproxy xmlns="http://indexdata.com/metaproxy" version="1.0">
1191 <start route="start"/>
1193 <filter id="frontend" type="frontend_net">
1196 <filter id="backend" type="z3950_client">
1201 <filter refid="frontend"/>
1202 <filter type="log"/>
1203 <filter refid="backend"/>
1204 <filter type="bounce"/>
1210 It works by defining a single route, called
1211 <literal>start</literal>, which consists of a sequence of four
1212 filters. The first and last of these are included by reference:
1213 their <literal><filter></literal> elements have
1214 <literal>refid</literal> attributes that refer to filters defined
1215 within the prior <literal><filters></literal> section. The
1216 middle filter is included inline in the route.
1219 The four filters in the route are as follows: first, a
1220 <literal>frontend_net</literal> filter accepts Z39.50 requests
1221 from any host on port 9000; then these requests are passed through
1222 a <literal>log</literal> filter that emits a message for each
1223 request; they are then fed into a <literal>z3950_client</literal>
1224 filter, which forwards all Z39.50 requests to the client-specified
1225 back-end Z39.509 server. Those Z39.50 packages are returned by the
1226 <literal>z3950_client</literal> filter, with the response data
1227 filled by the external Z39.50 server targeted.
1228 All non-Z39.50 packages are passed through to the
1229 <literal>bounce</literal> filter, which definitely bounces
1230 everything, including fish, bananas, cold pyjamas,
1231 mutton, beef and trout packages.
1232 When the response arrives, it is handed
1233 back to the <literal>log</literal> filter, which emits another
1234 message; and then to the <literal>frontend_net</literal> filter,
1235 which returns the response to the client.
1239 <section id="config-file-modularity">
1240 <title>Config file modularity</title>
1242 Metaproxy XML configuration snippets can be reused by other
1243 filters using the <literal>XInclude</literal> standard, as seen in
1244 the <literal>/etc/config-sru-to-z3950.xml</literal> example SRU
1247 <filter id="sru" type="sru_z3950">
1248 <database name="Default">
1249 <xi:include xmlns:xi="http://www.w3.org/2001/XInclude"
1250 href="explain.xml"/>
1257 <section id="config-file-syntax-check">
1258 <title>Config file syntax checking</title>
1260 The distribution contains RelaxNG Compact and XML syntax checking
1261 files, as well as XML Schema files. These are found in the
1264 xml/schema/metaproxy.rnc
1265 xml/schema/metaproxy.rng
1266 xml/schema/metaproxy.xsd
1268 and can be used to verify or debug the XML structure of
1269 configuration files. For example, using the utility
1270 <filename>xmllint</filename>, syntax checking is done like this:
1272 xmllint --noout --schema xml/schema/metaproxy.xsd etc/config-local.xml
1273 xmllint --noout --relaxng xml/schema/metaproxy.rng etc/config-local.xml
1275 (A recent version of <literal>libxml2</literal> is required, as
1276 support for XML Schemas is a relatively recent addition.)
1279 You can of course use any other RelaxNG or XML Schema compliant tool
1287 <chapter id="multidb">
1288 <title>Virtual databases and multi-database searching</title>
1291 <section id="multidb-introductory-notes">
1292 <title>Introductory notes</title>
1294 Two of Metaproxy's filters are concerned with multiple-database
1295 operations. Of these, <literal>virt_db</literal> can work alone
1296 to control the routing of searches to one of a number of servers,
1297 while <literal>multi</literal> can work together with
1298 <literal>virt_db</literal> to perform multi-database searching, merging
1299 the results into a unified result-set - ``metasearch in a box''.
1302 The interaction between
1303 these two filters is necessarily complex: it reflects the real,
1304 irreducible complexity of multi-database searching in a protocol such
1305 as Z39.50 that separates initialization from searching, and in
1306 which the database to be searched is not known at initialization
1310 It's possible to use these filters without understanding the
1311 details of their functioning and the interaction between them; the
1312 next two sections of this chapter are ``HOW-TO'' guides for doing
1313 just that. However, debugging complex configurations will require
1314 a deeper understanding, which the last two sections of this
1315 chapters attempt to provide.
1320 <section id="multidb.virt_db">
1321 <title>Virtual databases with the <literal>virt_db</literal> filter</title>
1323 Working alone, the purpose of the
1324 <literal>virt_db</literal>
1325 filter is to route search requests to one of a selection of
1326 back-end databases. In this way, a single Z39.50 endpoint
1327 (running Metaproxy) can provide access to several different
1328 underlying services, including those that would otherwise be
1329 inaccessible due to firewalls. In many useful configurations, the
1330 back-end databases are local to the Metaproxy installation, but
1331 the software does not enforce this, and any valid Z39.50 servers
1332 may be used as back-ends.
1335 For example, a <literal>virt_db</literal>
1336 filter could be set up so that searches in the virtual database
1337 ``lc'' are forwarded to the Library of Congress bibliographic
1338 catalogue server, and searches in the virtual database ``marc''
1339 are forwarded to the toy database of MARC records that Index Data
1340 hosts for testing purposes. A <literal>virt_db</literal>
1341 configuration to make this switch would look like this:
1343 <screen><![CDATA[<filter type="virt_db">
1345 <database>lc</database>
1346 <target>z3950.loc.gov:7090/voyager</target>
1349 <database>marc</database>
1350 <target>indexdata.com/marc</target>
1352 </filter>]]></screen>
1354 As well as being useful in it own right, this filter also provides
1355 the foundation for multi-database searching.
1360 <section id="multidb.multi">
1361 <title>Multi-database search with the <literal>multi</literal> filter</title>
1363 To arrange for Metaproxy to broadcast searches to multiple back-end
1364 servers, the configuration needs to include two components: a
1365 <literal>virt_db</literal>
1366 filter that specifies multiple
1367 <literal><target></literal>
1368 elements, and a subsequent
1369 <literal>multi</literal>
1370 filter. Here, for example, is a complete configuration that
1371 broadcasts searches to both the Library of Congress catalogue and
1372 Index Data's tiny testing database of MARC records:
1374 <screen><![CDATA[<?xml version="1.0"?>
1375 <metaproxy xmlns="http://indexdata.com/metaproxy" version="1.0">
1376 <start route="start"/>
1379 <filter type="frontend_net">
1380 <threads>10</threads>
1383 <filter type="virt_db">
1385 <database>lc</database>
1386 <target>z3950.loc.gov:7090/voyager</target>
1389 <database>marc</database>
1390 <target>indexdata.com/marc</target>
1393 <database>all</database>
1394 <target>z3950.loc.gov:7090/voyager</target>
1395 <target>indexdata.com/marc</target>
1398 <filter type="multi"/>
1399 <filter type="z3950_client">
1400 <timeout>30</timeout>
1402 <filter type="bounce"/>
1405 </metaproxy>]]></screen>
1408 <literal>virt_db</literal>
1409 filter that specifies multiple
1410 <literal><target></literal>
1411 elements but without a subsequent
1412 <literal>multi</literal>
1413 filter yields surprising and undesirable results, as will be
1414 described below. Don't do that.)
1417 Metaproxy can be invoked with this configuration as follows:
1419 <screen>../src/metaproxy --config config-simple-multi.xml</screen>
1421 And thereafter, Z39.50 clients can connect to the running server
1422 (on port 9000, as specified in the configuration) and search in
1423 any of the databases
1424 <literal>lc</literal> (the Library of Congress catalogue),
1425 <literal>marc</literal> (Index Data's test database of MARC records)
1427 <literal>all</literal> (both of these). As an example, a session
1428 using the YAZ command-line client <literal>yaz-client</literal> is
1429 here included (edited for brevity and clarity):
1431 <screen><![CDATA[$ yaz-client @:9000
1435 Search was a success.
1436 Number of hits: 10000, setno 1
1440 Search was a success.
1441 Number of hits: 10, setno 3
1445 Search was a success.
1446 Number of hits: 10010, setno 4
1449 [marc]Record type: USmarc
1452 005 00000000000000.0
1453 008 910710c19910701nju 00010 eng
1457 100 10 $a Jack Collins
1458 245 10 $a How to program a computer
1464 [VOYAGER]Record type: USmarc
1466 005 20041229102447.0
1467 008 030910s2004 caua 000 0 eng
1468 035 $a (DLC) 2003112666
1469 906 $a 7 $b cbc $c orignew $d 4 $e epcn $f 20 $g y-gencatlg
1470 925 0 $a acquire $b 1 shelf copy $x policy default
1471 955 $a pc10 2003-09-10 $a pv12 2004-06-23 to SSCD; $h sj05 2004-11-30 $e sj05 2004-11-30 to Shelf.
1474 040 $a DLC $c DLC $d DLC
1475 050 00 $a MLCM 2004/03312 (G)
1476 245 10 $a 007, everything or nothing : $b Prima's official strategy guide / $c created by Kaizen Media Group.
1477 246 3 $a Double-O-seven, everything or nothing
1478 246 30 $a Prima's official strategy guide
1479 260 $a Roseville, CA : $b Prima Games, $c c2004.
1480 300 $a 161 p. : $b col. ill. ; $c 28 cm.
1481 500 $a "Platforms: Nintendo GameCube, Macintosh, PC, PlayStation 2 computer entertainment system, Xbox"--P. [4] of cover.
1482 650 0 $a Video games.
1483 710 2 $a Kaizen Media Group.
1484 856 42 $3 Publisher description $u http://www.loc.gov/catdir/description/random052/2003112666.html
1489 As can be seen, the first record in the result set is from the
1490 Index Data test database, and the second from the Library of
1491 Congress database. The result-set continues alternating records
1492 round-robin style until the point where one of the databases'
1493 records are exhausted.
1496 This example uses only two back-end databases; more may be used.
1497 There is no limitation imposed on the number of databases that may
1498 be metasearched in this way: issues of resource usage and
1499 administrative complexity dictate the practical limits.
1502 What happens when one of the databases doesn't respond? By default,
1503 the entire multi-database search fails, and the appropriate
1504 diagnostic is returned to the client. This is usually appropriate
1505 during development, when technicians need maximum information, but
1506 can be inconvenient in deployment, when users typically don't want
1507 to be bothered with problems of this kind and prefer just to get
1508 the records from the databases that are available. To obtain this
1509 latter behavior add an empty
1510 <literal><hideunavailable></literal>
1512 <literal>multi</literal> filter:
1514 <screen><![CDATA[ <filter type="multi">
1516 </filter>]]></screen>
1518 Under this regime, an error is reported to the client only if
1519 <emphasis>all</emphasis> the databases in a multi-database search
1525 <section id="multidb.what">
1526 <title>What's going on?</title>
1528 <title>Lark's vomit</title>
1530 This section goes into a level of technical detail that is
1531 probably not necessary in order to configure and use Metaproxy.
1532 It is provided only for those who like to know how things work.
1533 You should feel free to skip on to the next section if this one
1534 doesn't seem like fun.
1538 Hold on tight - this may get a little hairy.
1541 In the general course of things, a Z39.50 Init request may carry
1542 with it an otherInfo packet of type <literal>VAL_PROXY</literal>,
1543 whose value indicates the address of a Z39.50 server to which the
1544 ultimate connection is to be made. (This otherInfo packet is
1545 supported by YAZ-based Z39.50 clients and servers, but has not yet
1546 been ratified by the Maintenance Agency and so is not widely used
1547 in non-Index Data software. We're working on it.)
1548 The <literal>VAL_PROXY</literal> packet functions
1549 analogously to the absoluteURI-style Request-URI used with the GET
1550 method when a web browser asks a proxy to forward its request: see
1552 <ulink url="http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html#sec5.1.2"
1553 >Request-URI</ulink>
1555 <ulink url="http://www.w3.org/Protocols/rfc2616/rfc2616.html"
1556 >the HTTP 1.1 specification</ulink>.
1559 Within Metaproxy, Search requests that are part of the same
1560 session as an Init request that carries a
1561 <literal>VAL_PROXY</literal> otherInfo are also annotated with the
1562 same information. The role of the <literal>virt_db</literal>
1563 filter is to rewrite this otherInfo packet dependent on the
1564 virtual database that the client wants to search.
1567 When Metaproxy receives a Z39.50 Init request from a client, it
1568 doesn't immediately forward that request to the back-end server.
1569 Why not? Because it doesn't know <emphasis>which</emphasis>
1570 back-end server to forward it to until the client sends a Search
1571 request that specifies the database that it wants to search in.
1572 Instead, it just treasures the Init request up in its heart; and,
1573 later, the first time the client does a search on one of the
1574 specified virtual databases, a connection is forged to the
1575 appropriate server and the Init request is forwarded to it. If,
1576 later in the session, the same client searches in a different
1577 virtual database, then a connection is forged to the server that
1578 hosts it, and the same cached Init request is forwarded there,
1582 All of this clever Init-delaying is done by the
1583 <literal>frontend_net</literal> filter. The
1584 <literal>virt_db</literal> filter knows nothing about it; in
1585 fact, because the Init request that is received from the client
1586 doesn't get forwarded until a Search request is received, the
1587 <literal>virt_db</literal> filter (and the
1588 <literal>z3950_client</literal> filter behind it) doesn't even get
1589 invoked at Init time. The <emphasis>only</emphasis> thing that a
1590 <literal>virt_db</literal> filter ever does is rewrite the
1591 <literal>VAL_PROXY</literal> otherInfo in the requests that pass
1595 It is possible for a <literal>virt_db</literal> filter to contain
1597 <literal><target></literal>
1598 elements. What does this mean? Only that the filter will add
1599 multiple <literal>VAL_PROXY</literal> otherInfo packets to the
1600 Search requests that pass through it. That's because the virtual
1601 DB filter is dumb, and does exactly what it's told - no more, no
1603 If a Search request with multiple <literal>VAL_PROXY</literal>
1604 otherInfo packets reaches a <literal>z3950_client</literal>
1605 filter, this is an error. That filter doesn't know how to deal
1606 with multiple targets, so it will either just pick one and search
1607 in it, or (better) fail with an error message.
1610 The <literal>multi</literal> filter comes to the rescue! This is
1611 the only filter that knows how to deal with multiple
1612 <literal>VAL_PROXY</literal> otherInfo packets, and it does so by
1613 making multiple copies of the entire Search request: one for each
1614 <literal>VAL_PROXY</literal>. Each of these new copies is then
1615 passed down through the remaining filters in the route. (The
1616 copies are handled in parallel though the
1617 spawning of new threads.) Since the copies each have only one
1618 <literal>VAL_PROXY</literal> otherInfo, they can be handled by the
1619 <literal>z3950_client</literal> filter, which happily deals with
1620 each one individually. When the results of the individual
1621 searches come back up to the <literal>multi</literal> filter, it
1622 merges them into a single Search response, which is what
1623 eventually makes it back to the client.
1628 <section id="multidb.picture">
1629 <title>A picture is worth a thousand words (but only five hundred on 64-bit architectures)</title>
1633 <imagedata fileref="multi.pdf" format="PDF" scale="50"/>
1636 <imagedata fileref="multi.png" format="PNG"/>
1639 <!-- Fall back if none of the images can be used -->
1641 [Here there should be a diagram showing the progress of
1642 packages through the filters during a simple virtual-database
1643 search and a multi-database search, but is seems that your
1644 tool chain has not been able to include the diagram in this
1648 <!-- ### This used to work with an older version of DocBook
1650 <para>Caption: progress of packages through filters.</para>
1653 </inlinemediaobject>
1659 <chapter id="sru-server">
1660 <title>Combined SRU webservice and Z39.50 server configuration</title>
1662 Metaproxy can act as
1663 <ulink url="&url.sru;">SRU</ulink> and
1664 <ulink url="&url.srw;">SRW</ulink>
1665 web service server, which translates web service requests to
1666 <ulink url="&url.z39.50;">ANSI/NISO Z39.50</ulink> packages and
1667 sends them off to common available targets.
1670 A typical setup for this operation needs a filter route including the
1674 <table id="sru-server-table-config" frame="top">
1675 <title>SRU/Z39.50 Server Filter Route Configuration</title>
1679 <entry>Filter</entry>
1680 <entry>Importance</entry>
1681 <entry>Purpose</entry>
1687 <entry><literal>frontend_net</literal></entry>
1688 <entry>required</entry>
1689 <entry>Accepting HTTP connections and passing them to following
1690 filters. Since this filter also accepts Z39.50 connections, the
1691 server works as SRU and Z39.50 server on the same port.</entry>
1694 <entry><literal>sru_z3950</literal></entry>
1695 <entry>required</entry>
1696 <entry>Accepting SRU GET/POST/SOAP explain and
1697 searchRetrieve requests for the the configured databases.
1698 Explain requests are directly served from the static XML configuration.
1699 SearchRetrieve requests are
1700 transformed to Z39.50 search and present packages.
1701 All other HTTP and Z39.50 packages are passed unaltered.</entry>
1704 <entry><literal>http_file</literal></entry>
1705 <entry>optional</entry>
1706 <entry>Serving HTTP requests from the filesystem. This is only
1707 needed if the server should serve XSLT stylesheets, static HTML
1708 files or Java Script for thin browser based clients.
1709 Z39.50 packages are passed unaltered.</entry>
1712 <entry><literal>cql_rpn</literal></entry>
1713 <entry>required</entry>
1714 <entry>Usually, Z39.50 servers do not talk CQL, hence the
1715 translation of the CQL query language to RPN is mandatory in
1716 most cases. Affects only Z39.50 search packages.</entry>
1719 <entry><literal>record_transform</literal></entry>
1720 <entry>optional</entry>
1721 <entry>Some Z39.50 backend targets can not present XML record
1722 syntaxes in common wanted element sets. using this filter, one
1723 can transform binary MARC records to MARCXML records, and
1724 further transform those to any needed XML schema/format by XSLT
1725 transformations. Changes only Z39.50 present packages.</entry>
1728 <entry><literal>session_shared</literal></entry>
1729 <entry>optional</entry>
1730 <entry>The stateless nature of web services requires frequent
1731 re-searching of the same targets for display of paged result set
1732 records. This might be an unacceptable burden for the accessed
1733 backend Z39.50 targets, and this mosule can be added for
1734 efficient backend target resource pooling.</entry>
1737 <entry><literal>z3950_client</literal></entry>
1738 <entry>required</entry>
1739 <entry>Finally, a Z39.50 package sink is needed in the filter
1740 chain to provide the response packages. The Z39.50 client module
1741 is used to access external targets over the network, but any
1742 coming local Z39.50 package sink could be used instead of.</entry>
1745 <entry><literal>bounce</literal></entry>
1746 <entry>required</entry>
1747 <entry>Any Metaproxy package arriving here did not do so by
1748 purpose, and is bounced back with connection closure. this
1749 prevents inifinite package hanging inside the SRU server.</entry>
1755 A typical minimal example <ulink url="&url.sru;">SRU</ulink> and
1756 <ulink url="&url.srw;">SRW</ulink> server configuration file is found
1757 in the tarball distribution at
1758 <literal>etc/config-sru-to-z3950.xml</literal>.
1761 Off course, any other metaproxy modules can be integrated into a
1762 SRU server solution, including, but not limited to, load balancing,
1763 multiple target querying
1764 (see <xref linkend="multidb"/>), and complex RPN query rewrites.
1771 <chapter id="extensions">
1772 <title>Writing extensions for Metaproxy</title>
1773 <para>### To be written</para>
1779 <chapter id="classes">
1780 <title>Classes in the Metaproxy source code</title>
1783 <section id="classes-introductory-notes">
1784 <title>Introductory notes</title>
1786 <emphasis>Stop! Do not read this!</emphasis>
1787 You won't enjoy it at all. You should just skip ahead to
1788 <xref linkend="reference"/>,
1790 <!-- The remainder of this paragraph is lifted verbatim from
1791 Douglas Adams' _Hitch Hiker's Guide to the Galaxy_, chapter 8 -->
1792 you things you really need to know, like the fact that the
1793 fabulously beautiful planet Bethselamin is now so worried about
1794 the cumulative erosion by ten billion visiting tourists a year
1795 that any net imbalance between the amount you eat and the amount
1796 you excrete whilst on the planet is surgically removed from your
1797 bodyweight when you leave: so every time you go to the lavatory it
1798 is vitally important to get a receipt.
1801 This chapter contains documentation of the Metaproxy source code, and is
1802 of interest only to maintainers and developers. If you need to
1803 change Metaproxy's behavior or write a new filter, then you will most
1804 likely find this chapter helpful. Otherwise it's a waste of your
1805 good time. Seriously: go and watch a film or something.
1806 <citetitle>This is Spinal Tap</citetitle> is particularly good.
1809 Still here? OK, let's continue.
1812 In general, classes seem to be named big-endianly, so that
1813 <literal>FactoryFilter</literal> is not a filter that filters
1814 factories, but a factory that produces filters; and
1815 <literal>FactoryStatic</literal> is a factory for the statically
1816 registered filters (as opposed to those that are dynamically
1821 <section id="individual.classes">
1822 <title>Individual classes</title>
1824 The classes making up the Metaproxy application are here listed by
1825 class-name, with the names of the source files that define them in
1829 <section id="class-FactoryFilter">
1830 <title><literal>mp::FactoryFilter</literal>
1831 (<filename>factory_filter.cpp</filename>)</title>
1833 A factory class that exists primarily to provide the
1834 <literal>create()</literal> method, which takes the name of a
1835 filter class as its argument and returns a new filter of that
1836 type. To enable this, the factory must first be populated by
1837 calling <literal>add_creator()</literal> for static filters (this
1838 is done by the <literal>FactoryStatic</literal> class, see below)
1839 and <literal>add_creator_dyn()</literal> for filters loaded
1844 <section id="class-FactoryStatic">
1845 <title><literal>mp::FactoryStatic</literal>
1846 (<filename>factory_static.cpp</filename>)</title>
1848 A subclass of <literal>FactoryFilter</literal> which is
1849 responsible for registering all the statically defined filter
1850 types. It does this by knowing about all those filters'
1851 structures, which are listed in its constructor. Merely
1852 instantiating this class registers all the static classes. It is
1853 for the benefit of this class that <literal>struct
1854 metaproxy_1_filter_struct</literal> exists, and that all the filter
1855 classes provide a static object of that type.
1859 <section id="class-filter-Base">
1860 <title><literal>mp::filter::Base</literal>
1861 (<filename>filter.cpp</filename>)</title>
1863 The virtual base class of all filters. The filter API is, on the
1864 surface at least, extremely simple: two methods.
1865 <literal>configure()</literal> is passed an XML DOM tree representing
1866 that part of the configuration file that pertains to this filter
1867 instance, and is expected to walk that tree extracting relevant
1868 information. And <literal>process()</literal> processes a
1869 package (see below). That surface simplicity is a bit
1870 misleading, as <literal>process()</literal> needs to know a lot
1871 about the <literal>Package</literal> class in order to do
1876 <section id="class-AuthSimple">
1877 <title><literal>mp::filter::AuthSimple</literal>,
1878 <literal>Backend_test</literal>, etc.
1879 (<filename>filter_auth_simple.cpp</filename>,
1880 <filename>filter_backend_test.cpp</filename>, etc.)</title>
1882 Individual filters. Each of these is implemented by a header and
1883 a source file, named <filename>filter_*.hpp</filename> and
1884 <filename>filter_*.cpp</filename> respectively. All the header
1885 files should be pretty much identical, in that they declare the
1886 class, including a private <literal>Rep</literal> class and a
1887 member pointer to it, and the two public methods.
1890 The source file for each filter needs to supply:
1895 A definition of the private <literal>Rep</literal> class.
1900 Some boilerplate constructors and destructors.
1905 A <literal>configure()</literal> method that uses the
1906 appropriate XML fragment.
1911 Most important, the <literal>process()</literal> method that
1912 does all the actual work.
1918 <section id="class-Package">
1919 <title><literal>mp::Package</literal>
1920 (<filename>package.cpp</filename>)</title>
1922 Represents a package on its way through the series of filters
1923 that make up a route. This is essentially a Z39.50 or SRU APDU
1924 together with information about where it came from, which is
1925 modified as it passes through the various filters.
1929 <section id="class-Pipe">
1930 <title><literal>mp::Pipe</literal>
1931 (<filename>pipe.cpp</filename>)</title>
1933 This class provides a compatibility layer so that we have an IPC
1934 mechanism that works the same under Unix and Windows. It's not
1935 particularly exciting.
1939 <section id="class-RouterChain">
1940 <title><literal>mp::RouterChain</literal>
1941 (<filename>router_chain.cpp</filename>)</title>
1947 <section id="class-RouterFleXML">
1948 <title><literal>mp::RouterFleXML</literal>
1949 (<filename>router_flexml.cpp</filename>)</title>
1955 <section id="class-Session">
1956 <title><literal>mp::Session</literal>
1957 (<filename>session.cpp</filename>)</title>
1963 <section id="class-ThreadPoolSocketObserver">
1964 <title><literal>mp::ThreadPoolSocketObserver</literal>
1965 (<filename>thread_pool_observer.cpp</filename>)</title>
1971 <section id="class-util">
1972 <title><literal>mp::util</literal>
1973 (<filename>util.cpp</filename>)</title>
1975 A namespace of various small utility functions and classes,
1976 collected together for convenience. Most importantly, includes
1977 the <literal>mp::util::odr</literal> class, a wrapper for YAZ's
1982 <section id="class-xml">
1983 <title><literal>mp::xml</literal>
1984 (<filename>xmlutil.cpp</filename>)</title>
1986 A namespace of various XML utility functions and classes,
1987 collected together for convenience.
1993 <section id="other.source.files">
1994 <title>Other Source Files</title>
1996 In addition to the Metaproxy source files that define the classes
1997 described above, there are a few additional files which are
1998 briefly described here:
2002 <term><literal>metaproxy_prog.cpp</literal></term>
2005 The main function of the <command>metaproxy</command> program.
2010 <term><literal>ex_router_flexml.cpp</literal></term>
2013 Identical to <literal>metaproxy_prog.cpp</literal>: it's not clear why.
2018 <term><literal>test_*.cpp</literal></term>
2021 Unit-tests for various modules.
2027 ### Still to be described:
2028 <literal>ex_filter_frontend_net.cpp</literal>,
2029 <literal>filter_dl.cpp</literal>,
2030 <literal>plainfile.cpp</literal>,
2031 <literal>tstdl.cpp</literal>.
2037 <reference id="reference">
2038 <title>Reference</title>
2039 <partintro id="reference-introduction">
2041 The material in this chapter is drawn directly from the individual
2042 manual entries. In particular, the Metaproxy invocation section is
2043 available using <command>man metaproxy</command>, and the section
2044 on each individual filter is available using the name of the filter
2045 as the argument to the <command>man</command> command.
2051 <appendix id="license">
2052 <title>License</title>
2055 Metaproxy, Copyright © 1995-2007 Index Data ApS.
2059 Metaproxy is free software; you can redistribute it and/or modify it under
2060 the terms of the GNU General Public License as published by the Free
2061 Software Foundation; either version 2, or (at your option) any later
2066 Metaproxy is distributed in the hope that it will be useful, but WITHOUT ANY
2067 WARRANTY; without even the implied warranty of MERCHANTABILITY or
2068 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
2073 You should have received a copy of the GNU General Public License
2074 along with Metaproxy; see the file LICENSE. If not, write to the
2075 Free Software Foundation,
2076 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
2080 <title>GNU General Public License</title>
2082 GNU GENERAL PUBLIC LICENSE
2083 Version 2, June 1991
2085 Copyright (C) 1989, 1991 Free Software Foundation, Inc.
2086 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
2087 w Everyone is permitted to copy and distribute verbatim copies
2088 of this license document, but changing it is not allowed.
2092 The licenses for most software are designed to take away your
2093 freedom to share and change it. By contrast, the GNU General Public
2094 License is intended to guarantee your freedom to share and change free
2095 software--to make sure the software is free for all its users. This
2096 General Public License applies to most of the Free Software
2097 Foundation's software and to any other program whose authors commit to
2098 using it. (Some other Free Software Foundation software is covered by
2099 the GNU Library General Public License instead.) You can apply it to
2102 When we speak of free software, we are referring to freedom, not
2103 price. Our General Public Licenses are designed to make sure that you
2104 have the freedom to distribute copies of free software (and charge for
2105 this service if you wish), that you receive source code or can get it
2106 if you want it, that you can change the software or use pieces of it
2107 in new free programs; and that you know you can do these things.
2109 To protect your rights, we need to make restrictions that forbid
2110 anyone to deny you these rights or to ask you to surrender the rights.
2111 These restrictions translate to certain responsibilities for you if you
2112 distribute copies of the software, or if you modify it.
2114 For example, if you distribute copies of such a program, whether
2115 gratis or for a fee, you must give the recipients all the rights that
2116 you have. You must make sure that they, too, receive or can get the
2117 source code. And you must show them these terms so they know their
2120 We protect your rights with two steps: (1) copyright the software, and
2121 (2) offer you this license which gives you legal permission to copy,
2122 distribute and/or modify the software.
2124 Also, for each author's protection and ours, we want to make certain
2125 that everyone understands that there is no warranty for this free
2126 software. If the software is modified by someone else and passed on, we
2127 want its recipients to know that what they have is not the original, so
2128 that any problems introduced by others will not reflect on the original
2129 authors' reputations.
2131 Finally, any free program is threatened constantly by software
2132 patents. We wish to avoid the danger that redistributors of a free
2133 program will individually obtain patent licenses, in effect making the
2134 program proprietary. To prevent this, we have made it clear that any
2135 patent must be licensed for everyone's free use or not licensed at all.
2137 The precise terms and conditions for copying, distribution and
2138 modification follow.
2140 GNU GENERAL PUBLIC LICENSE
2141 TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
2143 0. This License applies to any program or other work which contains
2144 a notice placed by the copyright holder saying it may be distributed
2145 under the terms of this General Public License. The "Program", below,
2146 refers to any such program or work, and a "work based on the Program"
2147 means either the Program or any derivative work under copyright law:
2148 that is to say, a work containing the Program or a portion of it,
2149 either verbatim or with modifications and/or translated into another
2150 language. (Hereinafter, translation is included without limitation in
2151 the term "modification".) Each licensee is addressed as "you".
2153 Activities other than copying, distribution and modification are not
2154 covered by this License; they are outside its scope. The act of
2155 running the Program is not restricted, and the output from the Program
2156 is covered only if its contents constitute a work based on the
2157 Program (independent of having been made by running the Program).
2158 Whether that is true depends on what the Program does.
2160 1. You may copy and distribute verbatim copies of the Program's
2161 source code as you receive it, in any medium, provided that you
2162 conspicuously and appropriately publish on each copy an appropriate
2163 copyright notice and disclaimer of warranty; keep intact all the
2164 notices that refer to this License and to the absence of any warranty;
2165 and give any other recipients of the Program a copy of this License
2166 along with the Program.
2168 You may charge a fee for the physical act of transferring a copy, and
2169 you may at your option offer warranty protection in exchange for a fee.
2171 2. You may modify your copy or copies of the Program or any portion
2172 of it, thus forming a work based on the Program, and copy and
2173 distribute such modifications or work under the terms of Section 1
2174 above, provided that you also meet all of these conditions:
2176 a) You must cause the modified files to carry prominent notices
2177 stating that you changed the files and the date of any change.
2179 b) You must cause any work that you distribute or publish, that in
2180 whole or in part contains or is derived from the Program or any
2181 part thereof, to be licensed as a whole at no charge to all third
2182 parties under the terms of this License.
2184 c) If the modified program normally reads commands interactively
2185 when run, you must cause it, when started running for such
2186 interactive use in the most ordinary way, to print or display an
2187 announcement including an appropriate copyright notice and a
2188 notice that there is no warranty (or else, saying that you provide
2189 a warranty) and that users may redistribute the program under
2190 these conditions, and telling the user how to view a copy of this
2191 License. (Exception: if the Program itself is interactive but
2192 does not normally print such an announcement, your work based on
2193 the Program is not required to print an announcement.)
2195 These requirements apply to the modified work as a whole. If
2196 identifiable sections of that work are not derived from the Program,
2197 and can be reasonably considered independent and separate works in
2198 themselves, then this License, and its terms, do not apply to those
2199 sections when you distribute them as separate works. But when you
2200 distribute the same sections as part of a whole which is a work based
2201 on the Program, the distribution of the whole must be on the terms of
2202 this License, whose permissions for other licensees extend to the
2203 entire whole, and thus to each and every part regardless of who wrote it.
2205 Thus, it is not the intent of this section to claim rights or contest
2206 your rights to work written entirely by you; rather, the intent is to
2207 exercise the right to control the distribution of derivative or
2208 collective works based on the Program.
2210 In addition, mere aggregation of another work not based on the Program
2211 with the Program (or with a work based on the Program) on a volume of
2212 a storage or distribution medium does not bring the other work under
2213 the scope of this License.
2215 3. You may copy and distribute the Program (or a work based on it,
2216 under Section 2) in object code or executable form under the terms of
2217 Sections 1 and 2 above provided that you also do one of the following:
2219 a) Accompany it with the complete corresponding machine-readable
2220 source code, which must be distributed under the terms of Sections
2221 1 and 2 above on a medium customarily used for software interchange; or,
2223 b) Accompany it with a written offer, valid for at least three
2224 years, to give any third party, for a charge no more than your
2225 cost of physically performing source distribution, a complete
2226 machine-readable copy of the corresponding source code, to be
2227 distributed under the terms of Sections 1 and 2 above on a medium
2228 customarily used for software interchange; or,
2230 c) Accompany it with the information you received as to the offer
2231 to distribute corresponding source code. (This alternative is
2232 allowed only for noncommercial distribution and only if you
2233 received the program in object code or executable form with such
2234 an offer, in accord with Subsection b above.)
2236 The source code for a work means the preferred form of the work for
2237 making modifications to it. For an executable work, complete source
2238 code means all the source code for all modules it contains, plus any
2239 associated interface definition files, plus the scripts used to
2240 control compilation and installation of the executable. However, as a
2241 special exception, the source code distributed need not include
2242 anything that is normally distributed (in either source or binary
2243 form) with the major components (compiler, kernel, and so on) of the
2244 operating system on which the executable runs, unless that component
2245 itself accompanies the executable.
2247 If distribution of executable or object code is made by offering
2248 access to copy from a designated place, then offering equivalent
2249 access to copy the source code from the same place counts as
2250 distribution of the source code, even though third parties are not
2251 compelled to copy the source along with the object code.
2253 4. You may not copy, modify, sublicense, or distribute the Program
2254 except as expressly provided under this License. Any attempt
2255 otherwise to copy, modify, sublicense or distribute the Program is
2256 void, and will automatically terminate your rights under this License.
2257 However, parties who have received copies, or rights, from you under
2258 this License will not have their licenses terminated so long as such
2259 parties remain in full compliance.
2261 5. You are not required to accept this License, since you have not
2262 signed it. However, nothing else grants you permission to modify or
2263 distribute the Program or its derivative works. These actions are
2264 prohibited by law if you do not accept this License. Therefore, by
2265 modifying or distributing the Program (or any work based on the
2266 Program), you indicate your acceptance of this License to do so, and
2267 all its terms and conditions for copying, distributing or modifying
2268 the Program or works based on it.
2270 6. Each time you redistribute the Program (or any work based on the
2271 Program), the recipient automatically receives a license from the
2272 original licensor to copy, distribute or modify the Program subject to
2273 these terms and conditions. You may not impose any further
2274 restrictions on the recipients' exercise of the rights granted herein.
2275 You are not responsible for enforcing compliance by third parties to
2278 7. If, as a consequence of a court judgment or allegation of patent
2279 infringement or for any other reason (not limited to patent issues),
2280 conditions are imposed on you (whether by court order, agreement or
2281 otherwise) that contradict the conditions of this License, they do not
2282 excuse you from the conditions of this License. If you cannot
2283 distribute so as to satisfy simultaneously your obligations under this
2284 License and any other pertinent obligations, then as a consequence you
2285 may not distribute the Program at all. For example, if a patent
2286 license would not permit royalty-free redistribution of the Program by
2287 all those who receive copies directly or indirectly through you, then
2288 the only way you could satisfy both it and this License would be to
2289 refrain entirely from distribution of the Program.
2291 If any portion of this section is held invalid or unenforceable under
2292 any particular circumstance, the balance of the section is intended to
2293 apply and the section as a whole is intended to apply in other
2296 It is not the purpose of this section to induce you to infringe any
2297 patents or other property right claims or to contest validity of any
2298 such claims; this section has the sole purpose of protecting the
2299 integrity of the free software distribution system, which is
2300 implemented by public license practices. Many people have made
2301 generous contributions to the wide range of software distributed
2302 through that system in reliance on consistent application of that
2303 system; it is up to the author/donor to decide if he or she is willing
2304 to distribute software through any other system and a licensee cannot
2307 This section is intended to make thoroughly clear what is believed to
2308 be a consequence of the rest of this License.
2310 8. If the distribution and/or use of the Program is restricted in
2311 certain countries either by patents or by copyrighted interfaces, the
2312 original copyright holder who places the Program under this License
2313 may add an explicit geographical distribution limitation excluding
2314 those countries, so that distribution is permitted only in or among
2315 countries not thus excluded. In such case, this License incorporates
2316 the limitation as if written in the body of this License.
2318 9. The Free Software Foundation may publish revised and/or new versions
2319 of the General Public License from time to time. Such new versions will
2320 be similar in spirit to the present version, but may differ in detail to
2321 address new problems or concerns.
2323 Each version is given a distinguishing version number. If the Program
2324 specifies a version number of this License which applies to it and "any
2325 later version", you have the option of following the terms and conditions
2326 either of that version or of any later version published by the Free
2327 Software Foundation. If the Program does not specify a version number of
2328 this License, you may choose any version ever published by the Free Software
2331 10. If you wish to incorporate parts of the Program into other free
2332 programs whose distribution conditions are different, write to the author
2333 to ask for permission. For software which is copyrighted by the Free
2334 Software Foundation, write to the Free Software Foundation; we sometimes
2335 make exceptions for this. Our decision will be guided by the two goals
2336 of preserving the free status of all derivatives of our free software and
2337 of promoting the sharing and reuse of software generally.
2341 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
2342 FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
2343 OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
2344 PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
2345 OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
2346 MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
2347 TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
2348 PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
2349 REPAIR OR CORRECTION.
2351 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
2352 WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
2353 REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
2354 INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
2355 OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
2356 TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
2357 YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
2358 PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
2359 POSSIBILITY OF SUCH DAMAGES.
2361 END OF TERMS AND CONDITIONS
2368 <!-- Keep this comment at the end of the file
2373 sgml-minimize-attributes:nil
2374 sgml-always-quote-attributes:t
2377 sgml-parent-document: nil
2378 sgml-local-catalogs: nil
2379 sgml-namecase-general:t