1 <?xml version="1.0" standalone="no"?>
2 <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
3 "http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd"
5 <!ENTITY copyright SYSTEM "copyright.xml">
6 <!ENTITY % local SYSTEM "local.ent">
8 <!ENTITY manref SYSTEM "manref.xml">
9 <!ENTITY gpl2 SYSTEM "gpl-2.0.xml">
10 <!ENTITY % idcommon SYSTEM "common/common.ent">
15 <title>Metaproxy - User's Guide and Reference</title>
18 <firstname>Adam</firstname><surname>Dickmeiss</surname>
21 <firstname>Marc</firstname><surname>Cromme</surname>
24 <firstname>Mike</firstname><surname>Taylor</surname>
27 <releaseinfo>&version;</releaseinfo>
29 <year>2005-2014</year>
30 <holder>Index Data</holder>
34 This manual is part of Metaproxy version &version;.
37 Metaproxy is a universal router, proxy and encapsulated
38 metasearcher for information retrieval protocols. It accepts,
39 processes, interprets and redirects requests from IR clients using
40 standard protocols such as the binary
41 <ulink url="&url.z39.50;">ANSI/NISO Z39.50</ulink>
42 and the information search and retrieval
43 web service <ulink url="&url.sru;">SRU</ulink>
44 as well as functioning as a limited
45 <ulink url="&url.http;">HTTP</ulink> server.
48 Metaproxy is configured by an XML file which
49 specifies how the software should function in terms of routes that
50 the request packets can take through the proxy, each step on a
51 route being an instantiation of a filter. Filters come in many
52 types, one for each operation: accepting Z39.50 packets, logging,
53 query transformation, multiplexing, etc. Further filter-types can
54 be added as loadable modules to extend Metaproxy functionality,
58 Metaproxy is covered by the GNU General Public License version 2.
63 <imagedata fileref="common/id.png" format="PNG"/>
66 <imagedata fileref="common/id.eps" format="EPS"/>
73 <chapter id="introduction">
74 <title>Introduction</title>
77 <ulink url="&url.metaproxy;">Metaproxy</ulink>
78 is a stand alone program that acts as a universal router, proxy and
79 encapsulated metasearcher for information retrieval protocols such
80 as <ulink url="&url.z39.50;">Z39.50</ulink> and
81 <ulink url="&url.sru;">SRU</ulink>.
82 To clients, it acts as a server of these protocols: it can be searched,
83 records can be retrieved from it, etc.
84 To servers, it acts as a client: it searches in them,
85 retrieves records from them, etc. it satisfies its clients'
86 requests by transforming them, multiplexing them, forwarding them
87 on to zero or more servers, merging the results, transforming
88 them, and delivering them back to the client. In addition, it
89 acts as a simple <ulink url="&url.http;">HTTP</ulink> server; support
90 for further protocols can be added in a modular fashion, through the
91 creation of new filters.
96 Fish, bananas, cold pyjamas,
97 Mutton, beef and trout!
98 - attributed to Cole Porter.
101 Metaproxy is a more capable alternative to
102 <ulink url="&url.yazproxy;">YAZ Proxy</ulink>,
103 being more powerful, flexible, configurable and extensible. Among
104 its many advantages over the older, more pedestrian work are
105 support for multiplexing (encapsulated metasearching), routing by
106 database name, authentication and authorization and serving local
107 files via HTTP. Equally significant, its modular architecture
108 facilitites the creation of pluggable modules implementing further
112 This manual will describe how to install Metaproxy
113 before giving an overview of its architecture, then discussing the
114 key concept of a filter in some depth and giving an overview of
115 the various filter types, then discussing the configuration file
116 format. After this come several optional chapters which may be
117 freely skipped: a detailed discussion of virtual databases and
118 multi-database searching, some notes on writing extensions
119 (additional filter types) and a high-level description of the
120 source code. Finally comes the reference guide, which contains
121 instructions for invoking the <command>metaproxy</command>
122 program, and detailed information on each type of filter,
127 <chapter id="installation">
128 <title>Installation</title>
130 Metaproxy depends on the following tools/libraries:
132 <varlistentry><term><ulink url="&url.yazplusplus;">YAZ++</ulink></term>
135 This is a C++ library based on <ulink url="&url.yaz;">YAZ</ulink>.
139 <varlistentry><term><ulink url="&url.libxslt;">Libxslt</ulink></term>
141 <para>This is an XSLT processor - based on
142 <ulink url="&url.libxml2;">Libxml2</ulink>. Both Libxml2 and
143 Libxslt must be installed with the development components
144 (header files, etc.) as well as the run-time libraries.
148 <varlistentry><term><ulink url="&url.boost;">Boost</ulink></term>
151 The popular C++ library. Initial versions of Metaproxy
152 was built with 1.32 but this is no longer supported.
153 Metaproxy is known to work with Boost version 1.33 through 1.55.
160 In order to compile Metaproxy a modern C++ compiler is
161 required. Boost, in particular, requires the C++ compiler
162 to facilitate the newest features. Refer to Boost
163 <ulink url="&url.boost.compilers.status;">Compiler Status</ulink>
164 for more information.
167 We have successfully built Metaproxy using the compilers
168 <ulink url="&url.gcc;">GCC</ulink> version 4.0 and
169 <ulink url="&url.vstudio;">Microsoft Visual Studio</ulink> 2003/2005/2008.
173 As an option, Metaproxy may also be compiled with
174 <ulink url="&url.usemarcon;">USEMARCON</ulink> support which allows for
175 MARC conversions for the <xref linkend="ref-record_transform"/> filter.
177 <section id="installation.unix">
178 <title>Installation on Unix (from Source)</title>
180 Here is a quick step-by-step guide on how to compile all the
181 tools that Metaproxy uses. Only few systems have none of the required
182 tools binary packages. If, for example, Libxml2/libxslt are already
183 installed as development packages use those (and omit compilation).
188 <ulink url="&url.usemarcon;">USEMARCON</ulink> is not available
189 as a package at the moment, so Metaproxy must be built from source
190 if that is to be used.
194 <section id="libxml2.fromsource">
195 <title>Libxml2/libxslt</title>
200 gunzip -c libxml2-version.tar.gz|tar xf -
208 gunzip -c libxslt-version.tar.gz|tar xf -
216 <section id="usemarcon">
217 <title>USEMARCON (optional)</title>
219 gunzip -c usemarcon317.tar.gz|tar xf -
228 <section id="yaz.fromsource">
229 <title>YAZ/YAZ++</title>
231 gunzip -c yaz-version.tar.gz|tar xf -
239 gunzip -c yazpp-version.tar.gz|tar xf -
248 <title id="boost.fromsource">Boost</title>
250 Metaproxy needs components thread and test from
254 gunzip -c boost-version.tar.gz|tar xf -
256 ./configure --with-libraries=thread,test,regex --with-toolset=gcc
262 However, under the hood bjam is used. You can invoke that with
265 ./bjam --toolset=gcc --with-thread --with-test --with-regex stage
268 Replace <literal>stage</literal> with <literal>clean</literal> /
269 <literal>install</literal> to perform clean and install respectively.
272 Add <literal>--prefix=DIR</literal> to install Boost in other
273 prefix than <literal>/usr/local</literal>.
276 <section id="metaproxy.fromsource">
277 <title>Metaproxy</title>
279 gunzip -c metaproxy-version.tar.gz|tar xf -
287 You may have to tell configure where Boost is installed by supplying
288 options <literal>--with-boost</literal> and <literal>--with-boost-toolset</literal>.
289 The former sets the PREFIX for Boost (same as --prefix for Boost above).
290 The latter the compiler toolset (eg. gcc34).
293 Pass <literal>--help</literal> to configure to get a list of
299 <section id="installation.debian">
300 <title>Installation on Debian GNU/Linux</title>
302 All dependencies for Metaproxy are available as
303 <ulink url="&url.debian;">Debian</ulink> packages.
306 The procedures for Debian based systems, such as
307 <ulink url="&url.ubuntu;">Ubuntu</ulink> is probably similar
310 There is currently no official Debian package for YAZ++.
311 And the official Debian package for YAZ is probably too old.
312 But Index Data builds "new" versions of those for Debian (i386, amd64 only).
315 Update the <filename>/etc/apt/sources.list</filename>
316 to include the Index Data repository.
317 See YAZ' <ulink url="&url.yaz.download.debian;">Download Debian</ulink>
318 for more information.
321 apt-get install libxslt1-dev
322 apt-get install libyazpp6-dev
323 apt-get install libboost-dev
324 apt-get install libboost-system-dev
325 apt-get install libboost-thread-dev
326 apt-get install libboost-test-dev
327 apt-get install libboost-regex-dev
330 With these packages installed, the usual configure + make
331 procedure can be used for Metaproxy as outlined in
332 <xref linkend="installation.unix"/>.
336 <section id="installation.rpm">
337 <title>Installation on RPM based Linux Systems</title>
339 All external dependencies for Metaproxy are available as
340 RPM packages, either from your distribution site, or from the
341 <ulink url="http://fr.rpmfind.net/">RPMfind</ulink> site.
344 For example, an installation of the requires Boost C++ development
345 libraries on RedHat Fedora C4 and C5 can be done like this:
347 wget ftp://fr.rpmfind.net/wlinux/fedora/core/updates/testing/4/SRPMS/boost-1.33.0-3.fc4.src.rpm
348 sudo rpmbuild --buildroot src/ --rebuild -p fc4/boost-1.33.0-3.fc4.src.rpm
349 sudo rpm -U /usr/src/redhat/RPMS/i386/boost-*rpm
353 The <ulink url="&url.yaz;">YAZ</ulink> library is needed to
354 compile &metaproxy;, see there
355 for more information on available RPM packages.
358 There is currently no official RPM package for YAZ++.
359 See the <ulink url="&url.yazplusplus;">YAZ++</ulink> pages
360 for more information on a Unix tarball install.
363 With these packages installed, the usual configure + make
364 procedure can be used for Metaproxy as outlined in
365 <xref linkend="installation.unix"/>.
369 <section id="installation.windows">
370 <title>Installation on Windows</title>
372 Metaproxy can be compiled with Microsoft
373 <ulink url="&url.vstudio;">Visual Studio</ulink>.
374 Versions 2003 (C 7.1), 2005 (C 8.0) and 2008 (C 9.0) are known to work.
376 <section id="installation.windows.boost">
379 For Windows, it's easiest to get the precompiled Boost
380 package from <ulink url="&url.boost.windows.download;">here</ulink>.
381 Several versions of the Boost libraries may be selected when
382 installing Boost for windows. Please choose at least the
383 <emphasis>multithreaded</emphasis> (non-DLL) version because
384 the Metaproxy makefile uses that.
387 For more information about installing Boost refer to the
388 <ulink url="&url.boost.getting.started;">getting started</ulink>
393 <section id="installation.windows.libxslt">
394 <title>Libxslt</title>
396 <ulink url="&url.libxslt;">Libxslt</ulink> can be downloaded
398 <ulink url="&url.libxml2.download.win32;">here</ulink>.
401 Libxslt has other dependencies, but these can all be downloaded
402 from the same site. Get the following package:
403 iconv, zlib, libxml2, libxslt.
407 <section id="installation.windows.yaz">
410 <ulink url="&url.yaz;">YAZ</ulink> can be downloaded
412 <ulink url="&url.yaz.download.win32;">here</ulink>.
416 <section id="installation.windows.yazplusplus">
419 Get <ulink url="&url.yazplusplus;">YAZ++</ulink> as well.
420 Version 1.5.2 or later is required.
423 YAZ++ includes NMAKE makefiles, similar to those found in the
428 <section id="installation.windows.metaproxy">
429 <title>Metaproxy</title>
431 Metaproxy is shipped with NMAKE makefiles as well - similar
432 to those found in the YAZ++/YAZ packages. Adjust this Makefile
433 to point to the proper locations of Boost, Libxslt, Libxml2,
434 zlib, iconv, yaz and yazpp.
438 <varlistentry><term><literal>DEBUG</literal></term>
440 If set to 1, the software is
441 compiled with debugging libraries (code generation is
442 multi-threaded debug DLL).
443 If set to 0, the software is compiled with release libraries
444 (code generation is multi-threaded DLL).
449 <term><literal>BOOST</literal></term>
452 Boost install location
458 <term><literal>BOOST_VERSION</literal></term>
461 Boost version (replace . with _).
467 <term><literal>BOOST_TOOLSET</literal></term>
476 <term><literal>LIBXSLT_DIR</literal>,
477 <literal>LIBXML2_DIR</literal> ..</term>
480 Specify the locations of Libxslt, libiconv, libxml2 and
489 After successful compilation you'll find
490 <literal>metaproxy.exe</literal> in the
491 <literal>bin</literal> directory.
499 <chapter id="yazproxy-comparison">
500 <title>YAZ Proxy Comparison</title>
502 The table below lists facilities either supported by either
503 <ulink url="&url.yazproxy;">YAZ Proxy</ulink> or Metaproxy.
505 <table id="yazproxy-comparison-table">
506 <title>Metaproxy / YAZ Proxy comparison</title>
510 <entry>Facility</entry>
511 <entry>Metaproxy</entry>
512 <entry>YAZ Proxy</entry>
517 <entry>Z39.50 server</entry>
518 <entry>Using filter <xref linkend="ref-frontend_net"/></entry>
519 <entry>Supported</entry>
522 <entry>SRU server</entry>
523 <entry>Supported with filter <xref linkend="ref-sru_z3950"/></entry>
524 <entry>Supported</entry>
527 <entry>Z39.50 client</entry>
528 <entry>Supported with filter <xref linkend="ref-z3950_client"/></entry>
529 <entry>Supported</entry>
532 <entry>SRU client</entry>
533 <entry>Supported with filter <xref linkend="ref-zoom"/></entry>
534 <entry>Unsupported</entry>
537 <entry>Connection reuse</entry>
538 <entry>Supported with filter <literal>session_shared</literal></entry>
539 <entry>Supported</entry>
542 <entry>Connection share</entry>
543 <entry>Supported with filter <literal>session_shared</literal></entry>
544 <entry>Unsupported</entry>
547 <entry>Result set reuse</entry>
548 <entry>Supported with filter <literal>session_shared</literal></entry>
549 <entry>Within one Z39.50 session / HTTP keep-alive</entry>
552 <entry>Record cache</entry>
553 <entry>Supported by filter <literal>session_shared</literal></entry>
554 <entry>Supported for last result set within one Z39.50/HTTP-keep alive session</entry>
557 <entry>Z39.50 Virtual database, i.e. select any Z39.50 target for database</entry>
558 <entry>Supported with filter <literal>virt_db</literal></entry>
559 <entry>Unsupported</entry>
562 <entry>SRU Virtual database, i.e. select any Z39.50 target for path</entry>
563 <entry>Supported with filter <literal>virt_db</literal>,
564 <literal>sru_z3950</literal></entry>
565 <entry>Supported</entry>
568 <entry>Multi target search</entry>
569 <entry>Supported with filter <literal>multi</literal> (round-robin)</entry>
570 <entry>Unsupported</entry>
573 <entry>Retrieval and search limits</entry>
574 <entry>Supported using filter <literal>limit</literal></entry>
575 <entry>Supported</entry>
578 <entry>Bandwidth limits</entry>
579 <entry>Supported using filter <literal>limit</literal></entry>
580 <entry>Supported</entry>
583 <entry>Connect limits</entry>
584 <entry>Supported by filter <literal>frontend_net</literal> (connect-max)</entry>
585 <entry>Supported</entry>
588 <entry>Retrieval sanity check and conversions</entry>
589 <entry>Supported using filter <literal>record_transform</literal></entry>
590 <entry>Supported</entry>
593 <entry>Query check</entry>
595 Supported by <literal>query_rewrite</literal> which may be check
596 a query and throw diagnostics (errors)
598 <entry>Supported</entry>
601 <entry>Query rewrite</entry>
602 <entry>Supported with <literal>query_rewrite</literal></entry>
603 <entry>Unsupported</entry>
606 <entry>Session invalidate for -1 hits</entry>
607 <entry>Unsupported</entry>
608 <entry>Supported</entry>
611 <entry>Architecture</entry>
612 <entry>Multi-threaded + select for networked modules such as
613 <literal>frontend_net</literal>)</entry>
614 <entry>Single-threaded using select</entry>
618 <entry>Extensability</entry>
619 <entry>Most functionality implemented as loadable modules</entry>
620 <entry>Unsupported and experimental</entry>
624 <entry><ulink url="&url.usemarcon;">USEMARCON</ulink></entry>
625 <entry>Supported with <literal>record_transform</literal></entry>
626 <entry>Supported</entry>
630 <entry>Portability</entry>
632 Requires YAZ, YAZ++ and modern C++ compiler supporting
633 <ulink url="&url.boost;">Boost</ulink>.
636 Requires YAZ and YAZ++.
637 STL is not required so pretty much any C++ compiler out there should work.
646 <chapter id="architecture">
647 <title>The Metaproxy Architecture</title>
649 The Metaproxy architecture is based on three concepts:
650 the <emphasis>package</emphasis>,
651 the <emphasis>route</emphasis>
652 and the <emphasis>filter</emphasis>.
656 <term>Packages</term>
659 A package is request or response, encoded in some protocol,
660 issued by a client, making its way through Metaproxy, send to or
661 received from a server, or sent back to the client.
664 The core of a package is the protocol unit - for example, a
665 Z39.50 Init Request or Search Response, or an SRU searchRetrieve
666 URL or Explain Response. In addition to this core, a package
667 also carries some extra information added and used by Metaproxy
671 In general, packages are doctored as they pass through
672 Metaproxy. For example, when the proxy performs authentication
673 and authorization on a Z39.50 Init request, it removes the
674 authentication credentials from the package so that they are not
675 passed onto the back-end server; and when search-response
676 packages are obtained from multiple servers, they are merged
677 into a single unified package that makes its way back to the
686 Packages make their way through routes, which can be thought of
687 as programs that operate on the package data-type. Each
688 incoming package initially makes its way through a default
689 route, but may be switched to a different route based on various
690 considerations. Routes are made up of sequences of filters (see
699 Filters provide the individual instructions within a route, and
700 effect the necessary transformations on packages. A particular
701 configuration of Metaproxy is essentially a set of filters,
702 described by configuration details and arranged in order in one
703 or more routes. There are many kinds of filter - about a dozen
704 at the time of writing with more appearing all the time - each
705 performing a specific function and configured by different
709 The word ``filter'' is sometimes used rather loosely, in two
710 different ways: it may be used to mean a particular
711 <emphasis>type</emphasis> of filter, as when we speak of ``the
712 auth_simple filter'' or ``the multi filter''; or it may be used
713 to be a specific <emphasis>instance</emphasis> of a filter
714 within a Metaproxy configuration. For example, a single
715 configuration will often contain multiple instances of the
716 <literal>z3950_client</literal> filter. In
717 operational terms, of these is a separate filter. In practice,
718 context always make it clear which sense of the word ``filter''
722 Extensibility of Metaproxy is primarily through the creation of
723 plugins that provide new filters. The filter API is small and
724 conceptually simple, but there are many details to master. See
726 <link linkend="filters">Filters</link>.
732 Since packages are created and handled by the system itself, and
733 routes are conceptually simple, most of the remainder of this
734 document concentrates on filters. After a brief overview of the
735 filter types follows, along with some thoughts on possible future
742 <chapter id="filters">
743 <title>Filters</title>
746 <section id="filters-introductory-notes">
747 <title>Introductory notes</title>
749 It's useful to think of Metaproxy as an interpreter providing a small
750 number of primitives and operations, but operating on a very
751 complex data type, namely the ``package''.
754 A package represents a Z39.50 or SRU/W request (whether for Init,
755 Search, Scan, etc.) together with information about where it came
756 from. Packages are created by front-end filters such as
757 <literal>frontend_net</literal> (see below), which reads them from
758 the network; other front-end filters are possible. They then pass
759 along a route consisting of a sequence of filters, each of which
760 transforms the package and may also have side-effects such as
761 generating logging. Eventually, the route will yield a response,
762 which is sent back to the origin.
765 There are many kinds of filter: some that are defined statically
766 as part of Metaproxy, and others may be provided by third parties
767 and dynamically loaded. They all conform to the same simple API
768 of essentially two methods: <function>configure()</function> is
769 called at startup time, and is passed an XML DOM tree representing that
770 part of the configuration file that pertains to this filter
771 instance: it is expected to walk that tree extracting relevant
772 information; and <function>process()</function> is called every
773 time the filter has to processes a package.
776 While all filters provide the same API, there are different modes
777 of functionality. Some filters are sources: they create
779 (<literal>frontend_net</literal>);
780 others are sinks: they consume packages and return a result
781 (<literal>backend_test</literal>,
782 <literal>bounce</literal>,
783 <literal>http_file</literal>,
784 <literal>z3950_client</literal>);
785 the others are true filters, that read, process and pass on the
786 packages they are fed
787 (<literal>auth_simple</literal>,
788 <literal>log</literal>,
789 <literal>multi</literal>,
790 <literal>query_rewrite</literal>,
791 <literal>record_transform</literal>,
792 <literal>session_shared</literal>,
793 <literal>sru_z3950</literal>,
794 <literal>template</literal>,
795 <literal>virt_db</literal>).
800 <section id="overview.filter.types">
801 <title>Overview of filter types</title>
803 We now briefly consider each of the types of filter supported by
804 the core Metaproxy binary. This overview is intended to give a
805 flavor of the available functionality; more detailed information
806 about each type of filter is included below in
807 <xref linkend="reference"/>.
810 The filters are here named by the string that is used as the
811 <literal>type</literal> attribute of a
812 <literal><filter></literal> element in the configuration
813 file to request them, with the name of the class that implements
814 them in parentheses. (The classname is not needed for normal
815 configuration and use of Metaproxy; it is useful only to
819 The filters are here listed in alphabetical order:
826 New virt_db-alike that does inteligent peer choice, explain merging,
827 adds FD&N to explain. Keeps init responses (like "virt_db Classic"),
828 makes routing choices based on local explain knowledge. Ref IDDI
831 Filter to convert Explain Classic to ZeeRex.
833 CQL2PQF (which needs augmented ZeeRex) - MARC for Talis.
837 Figure out what additional information we need in:
838 ZeeRex (check against D3.1)
839 Init request (e.g. loop detection)
840 Query package (e.g. number of hops)
841 Query response (e.g. record source)
845 <section id="auth_simple">
846 <title><literal>auth_simple</literal>
847 (mp::filter::AuthSimple)</title>
849 Simple authentication and authorization. The configuration
850 specifies the name of a file that is the user register, which
851 lists <varname>username</varname>:<varname>password</varname>
852 pairs, one per line, colon separated. When a session begins, it
853 is rejected unless username and passsword are supplied, and match
854 a pair in the register. The configuration file may also specific
855 the name of another file that is the target register: this lists
856 lists <varname>username</varname>:<varname>dbname</varname>,<varname>dbname</varname>...
857 sets, one per line, with multiple database names separated by
858 commas. When a search is processed, it is rejected unless the
859 database to be searched is one of those listed as available to
864 <section id="backend_test">
865 <title><literal>backend_test</literal>
866 (mp::filter::Backend_test)</title>
868 A partial sink that provides dummy responses in the manner of the
869 <literal>yaz-ztest</literal> Z39.50 server. This is useful only
870 for testing. Seriously, you don't need this. Pretend you didn't
871 even read this section.
875 <section id="bounce">
876 <title><literal>bounce</literal>
877 (mp::filter::Bounce)</title>
879 A sink that swallows <emphasis>all packages</emphasis>,
880 and returns them almost unprocessed.
881 It never sends any package of any type further down the row, but
882 sets Z39.50 packages to Z_Close, and HTTP_Request packages to
883 HTTP_Response err code 400 packages, and adds a suitable bounce
885 The bounce filter is usually added at end of each filter chain route
886 to prevent infinite hanging of for example HTTP
887 requests packages when only the Z39.50 client partial sink
888 filter is found in the
893 <section id="cql_rpn">
894 <title><literal>cql_rpn</literal>
895 (mp::filter::CQLtoRPN)</title>
897 A query language transforming filter which catches Z39.50
898 <literal>searchRequest</literal>
899 packages containing <literal>CQL</literal> queries, transforms
900 those to <literal>RPN</literal> queries,
901 and sends the <literal>searchRequests</literal> on to the next
902 filters. It is among other things useful in a SRU context.
906 <section id="frontend_net">
907 <title><literal>frontend_net</literal>
908 (mp::filter::FrontendNet)</title>
910 A source that accepts Z39.50 connections from a port
911 specified in the configuration, reads protocol units, and
912 feeds them into the next filter in the route. When the result is
913 received, it is returned to the original origin.
917 <section id="http_file">
918 <title><literal>http_file</literal>
919 (mp::filter::HttpFile)</title>
921 A partial sink which swallows only
922 <literal>HTTP_Request</literal> packages, and
923 returns the contents of files from the local
924 filesystem in response to HTTP requests.
925 It lets Z39.50 packages and all other forthcoming package types
928 does mean that Metaproxy is also a Web-server in its spare time. So
929 far it does not contain either an email-reader or a Lisp
930 interpreter, but that day is surely coming.)
934 <section id="load_balance">
935 <title><literal>load_balance</literal>
936 (mp::filter::LoadBalance)</title>
938 Performs load balancing for incoming Z39.50 init requests.
939 It is used together with the <literal>virt_db</literal> filter,
940 but unlike the <literal>multi</literal> filter it does send an
941 entire session to only one of the virtual backends. The
942 <literal>load_balance</literal> filter is assuming that
943 all backend targets have equal content, and chooses the backend
944 with least load cost for a new session.
947 This filter is experimental and yet not mature for heavy load
955 <title><literal>log</literal>
956 (mp::filter::Log)</title>
958 Writes logging information to standard output, and passes on
959 the package unchanged. A log file name can be specified, as well
960 as multiple different logging formats.
965 <title><literal>multi</literal>
966 (mp::filter::Multi)</title>
968 Performs multi-database searching.
970 <link linkend="multidb">the extended discussion</link>
971 of virtual databases and multi-database searching below.
975 <section id="query_rewrite">
976 <title><literal>query_rewrite</literal>
977 (mp::filter::QueryRewrite)</title>
979 Rewrites Z39.50 <literal>Type-1</literal>
980 and <literal>Type-101</literal> (``<literal>RPN</literal>'')
982 three-step process: the query is transliterated from Z39.50
983 packet structures into an XML representation; that XML
984 representation is transformed by an XSLT stylesheet; and the
985 resulting XML is transliterated back into the Z39.50 packet
991 <section id="record_transform">
992 <title><literal>record_transform</literal>
993 (mp::filter::RecordTransform)</title>
995 This filter acts only on Z3950 present requests, and let all
996 other types of packages and requests pass untouched. It's use is
997 twofold: blocking Z3950 present requests, which the backend
998 server does not understand and can not honor, and transforming
999 the present syntax and elementset name according to the rules
1000 specified, to fetch only existing record formats, and transform
1001 them on the fly to requested record syntaxes.
1005 <section id="session_shared">
1006 <title><literal>session_shared</literal>
1007 (mp::filter::SessionShared)</title>
1009 This filter implements global sharing of
1010 result sets (i.e. between threads and therefore between
1011 clients), yielding performance improvements by clever resource
1016 <section id="sru_z3950">
1017 <title><literal>sru_z3950</literal>
1018 (mp::filter::SRUtoZ3950)</title>
1020 This filter transforms valid
1021 SRU GET/POST/SOAP searchRetrieve requests to Z3950 init, search,
1022 and present requests, and wraps the
1023 received hit counts and XML records into suitable SRU response
1025 The <literal>sru_z3950</literal> filter processes also SRU
1026 GET/POST/SOAP explain requests, returning
1027 either the absolute minimum required by the standard, or a full
1028 pre-defined ZeeReX explain record.
1030 <ulink url="&url.zeerex.explain;">ZeeReX Explain</ulink>
1031 standard pages and the
1032 <ulink url="&url.sru.explain;">SRU Explain</ulink> pages
1033 for more information on the correct explain syntax.
1034 SRU scan requests are not supported yet.
1038 <section id="template">
1039 <title><literal>template</literal>
1040 (mp::filter::Template)</title>
1042 Does nothing at all, merely passing the packet on. (Maybe it
1043 should be called <literal>nop</literal> or
1044 <literal>passthrough</literal>?) This exists not to be used, but
1045 to be copied - to become the skeleton of new filters as they are
1046 written. As with <literal>backend_test</literal>, this is not
1047 intended for civilians.
1051 <section id="virt_db">
1052 <title><literal>virt_db</literal>
1053 (mp::filter::VirtualDB)</title>
1055 Performs virtual database selection: based on the name of the
1056 database in the search request, a server is selected, and its
1057 address added to the request in a <literal>VAL_PROXY</literal>
1058 otherInfo packet. It will subsequently be used by a
1059 <literal>z3950_client</literal> filter.
1061 <link linkend="multidb">the extended discussion</link>
1062 of virtual databases and multi-database searching below.
1066 <section id="z3950_client">
1067 <title><literal>z3950_client</literal>
1068 (mp::filter::Z3950Client)</title>
1070 A partial sink which swallows only Z39.50 packages.
1071 It performs Z39.50 searching and retrieval by proxying the
1072 packages that are passed to it. Init requests are sent to the
1073 address specified in the <literal>VAL_PROXY</literal> otherInfo
1074 attached to the request: this may have been specified by client,
1075 or generated by a <literal>virt_db</literal> filter earlier in
1076 the route. Subsequent requests are sent to the same address,
1077 which is remembered at Init time in a Session object.
1078 HTTP_Request packages and all other forthcoming package types
1079 are passed untouched.
1084 <section id="zeerex_explain">
1085 <title><literal>zeerex_explain</literal>
1086 (mp::filter::ZeerexExplain)</title>
1088 This filter acts as a sink for
1089 Z39.50 explain requests, returning a static ZeeReX
1090 Explain XML record from the config section. All other packages
1093 <ulink url="&url.zeerex.explain;">ZeeReX Explain</ulink>
1095 for more information on the correct explain syntax.
1099 This filter is not yet completed.
1108 <section id="future.directions">
1109 <title>Future directions</title>
1111 Some other filters that do not yet exist, but which would be
1112 useful, are briefly described. These may be added in future
1113 releases (or may be created by third parties, as loadable
1119 <term><literal>frontend_cli</literal> (source)</term>
1122 Command-line interface for generating requests.
1127 <term><literal>sru_client</literal> (sink)</term>
1130 SRU/GET and SRU/SOAP searching and retrieval.
1135 <term><literal>opensearch_client</literal> (sink)</term>
1138 A9 OpenSearch searching and retrieval.
1148 <chapter id="configuration">
1149 <title>Configuration: the Metaproxy configuration file format</title>
1152 <section id="configuration-introductory-notes">
1153 <title>Introductory notes</title>
1155 If Metaproxy is an interpreter providing operations on packages, then
1156 its configuration file can be thought of as a program for that
1157 interpreter. Configuration is by means of a single XML file, the name
1158 of which is supplied as the sole command-line argument to the
1159 <command>metaproxy</command> program. (See
1160 <xref linkend="reference"/> below for more information on invoking
1165 <section id="overview.xml.structure">
1166 <title>Overview of the config file XML structure</title>
1168 All elements and attributes are in the namespace
1169 <ulink url="http://indexdata.com/metaproxy"/>.
1170 This is most easily achieved by setting the default namespace on
1171 the top-level element, as here:
1174 <metaproxy xmlns="http://indexdata.com/metaproxy" version="1.0">
1177 The top-level element is <metaproxy>. This contains
1178 a <dlpath> element,
1179 a <start> element,
1180 a <filters> element and
1181 a <routes> element, in that order. <dlpath> and
1182 <filters> are optional; the other two are mandatory.
1183 All four are non-repeatable.
1186 The <dlpath;> element contains a text element which
1187 specifies the location of filter modules. This is only needed
1188 if Metaproxy must load 3rd party filters (most filters with Metaproxy
1189 are built into the Metaproxy application).
1192 The <start> element is empty, but carries a
1193 <literal>route</literal> attribute, whose value is the name of
1194 route at which to start running - analogous to the name of the
1195 start production in a formal grammar.
1198 If present, <filters> contains zero or more <filter>
1199 elements. Each filter carries a <literal>type</literal> attribute
1200 which specifies what kind of filter is being defined
1201 (<literal>frontend_net</literal>, <literal>log</literal>, etc.)
1202 and contain various elements that provide suitable configuration
1203 for a filter of its type. The filter-specific elements are
1205 <xref linkend="reference"/>.
1206 Filters defined in this part of the file must carry an
1207 <literal>id</literal> attribute so that they can be referenced
1211 <routes> contains one or more <route> elements, each
1212 of which must carry an <literal>id</literal> element. One of the
1213 routes must have the ID value that was specified as the start
1214 route in the <start> element's <literal>route</literal>
1215 attribute. Each route contains zero or more <filter>
1216 elements. These are of two types. They may be empty, but carry a
1217 <literal>refid</literal> attribute whose value is the same as the
1218 <literal>id</literal> of a filter previously defined in the
1219 <filters> section. Alternatively, a route within a filter
1220 may omit the <literal>refid</literal> attribute, but contain
1221 configuration elements similar to those used for filters defined
1222 in the <filters> section. (In other words, each filter in a
1223 route may be included either by reference or by physical
1229 <section id="example.configuration">
1230 <title>An example configuration</title>
1232 The following is a small, but complete, Metaproxy configuration
1233 file (included in the distribution as
1234 <literal>metaproxy/etc/config1.xml</literal>).
1235 This file defines a very simple configuration that simply proxies
1236 to whatever back-end server the client requests, but logs each
1237 request and response. This can be useful for debugging complex
1238 client-server dialogues.
1240 <screen><![CDATA[<?xml version="1.0"?>
1241 <metaproxy xmlns="http://indexdata.com/metaproxy" version="1.0">
1242 <dlpath>/usr/lib/metaproxy/modules</dlpath>
1243 <start route="start"/>
1245 <filter id="frontend" type="frontend_net">
1248 <filter id="backend" type="z3950_client">
1253 <filter refid="frontend"/>
1254 <filter type="log"/>
1255 <filter refid="backend"/>
1256 <filter type="bounce"/>
1262 It works by defining a single route, called
1263 <literal>start</literal>, which consists of a sequence of four
1264 filters. The first and last of these are included by reference:
1265 their <literal><filter></literal> elements have
1266 <literal>refid</literal> attributes that refer to filters defined
1267 within the prior <literal><filters></literal> section. The
1268 middle filter is included inline in the route.
1271 The four filters in the route are as follows: first, a
1272 <literal>frontend_net</literal> filter accepts Z39.50 requests
1273 from any host on port 9000; then these requests are passed through
1274 a <literal>log</literal> filter that emits a message for each
1275 request; they are then fed into a <literal>z3950_client</literal>
1276 filter, which forwards all Z39.50 requests to the client-specified
1277 back-end Z39.509 server. Those Z39.50 packages are returned by the
1278 <literal>z3950_client</literal> filter, with the response data
1279 filled by the external Z39.50 server targeted.
1280 All non-Z39.50 packages are passed through to the
1281 <literal>bounce</literal> filter, which definitely bounces
1282 everything, including fish, bananas, cold pyjamas,
1283 mutton, beef and trout packages.
1284 When the response arrives, it is handed
1285 back to the <literal>log</literal> filter, which emits another
1286 message; and then to the <literal>frontend_net</literal> filter,
1287 which returns the response to the client.
1291 <section id="config-file-modularity">
1292 <title>Config file modularity</title>
1294 Metaproxy XML configuration snippets can be reused by other
1295 filters using the <literal>XInclude</literal> standard, as seen in
1296 the <literal>/etc/config-sru-to-z3950.xml</literal> example SRU
1299 <filter id="sru" type="sru_z3950">
1300 <database name="Default">
1301 <xi:include xmlns:xi="http://www.w3.org/2001/XInclude"
1302 href="explain.xml"/>
1309 <section id="config-file-syntax-check">
1310 <title>Config file syntax checking</title>
1312 The distribution contains RelaxNG Compact and XML syntax checking
1313 files, as well as XML Schema files. These are found in the
1316 xml/schema/metaproxy.rnc
1317 xml/schema/metaproxy.rng
1318 xml/schema/metaproxy.xsd
1320 and can be used to verify or debug the XML structure of
1321 configuration files. For example, using the utility
1322 <filename>xmllint</filename>, syntax checking is done like this:
1324 xmllint --noout --schema xml/schema/metaproxy.xsd etc/config-local.xml
1325 xmllint --noout --relaxng xml/schema/metaproxy.rng etc/config-local.xml
1327 (A recent version of <literal>libxml2</literal> is required, as
1328 support for XML Schemas is a relatively recent addition.)
1331 You can of course use any other RelaxNG or XML Schema compliant tool
1339 <chapter id="multidb">
1340 <title>Virtual databases and multi-database searching</title>
1343 <section id="multidb-introductory-notes">
1344 <title>Introductory notes</title>
1346 Two of Metaproxy's filters are concerned with multiple-database
1347 operations. Of these, <literal>virt_db</literal> can work alone
1348 to control the routing of searches to one of a number of servers,
1349 while <literal>multi</literal> can work together with
1350 <literal>virt_db</literal> to perform multi-database searching, merging
1351 the results into a unified result-set - ``metasearch in a box''.
1354 The interaction between
1355 these two filters is necessarily complex: it reflects the real,
1356 irreducible complexity of multi-database searching in a protocol such
1357 as Z39.50 that separates initialization from searching, and in
1358 which the database to be searched is not known at initialization
1362 It's possible to use these filters without understanding the
1363 details of their functioning and the interaction between them; the
1364 next two sections of this chapter are ``HOW-TO'' guides for doing
1365 just that. However, debugging complex configurations will require
1366 a deeper understanding, which the last two sections of this
1367 chapters attempt to provide.
1372 <section id="multidb.virt_db">
1373 <title>Virtual databases with the <literal>virt_db</literal> filter</title>
1375 Working alone, the purpose of the
1376 <literal>virt_db</literal>
1377 filter is to route search requests to one of a selection of
1378 back-end databases. In this way, a single Z39.50 endpoint
1379 (running Metaproxy) can provide access to several different
1380 underlying services, including those that would otherwise be
1381 inaccessible due to firewalls. In many useful configurations, the
1382 back-end databases are local to the Metaproxy installation, but
1383 the software does not enforce this, and any valid Z39.50 servers
1384 may be used as back-ends.
1387 For example, a <literal>virt_db</literal>
1388 filter could be set up so that searches in the virtual database
1389 ``lc'' are forwarded to the Library of Congress bibliographic
1390 catalogue server, and searches in the virtual database ``marc''
1391 are forwarded to the toy database of MARC records that Index Data
1392 hosts for testing purposes. A <literal>virt_db</literal>
1393 configuration to make this switch would look like this:
1395 <screen><![CDATA[<filter type="virt_db">
1397 <database>lc</database>
1398 <target>z3950.loc.gov:7090/voyager</target>
1401 <database>marc</database>
1402 <target>indexdata.com/marc</target>
1404 </filter>]]></screen>
1406 As well as being useful in it own right, this filter also provides
1407 the foundation for multi-database searching.
1412 <section id="multidb.multi">
1413 <title>Multi-database search with the <literal>multi</literal> filter</title>
1415 To arrange for Metaproxy to broadcast searches to multiple back-end
1416 servers, the configuration needs to include two components: a
1417 <literal>virt_db</literal>
1418 filter that specifies multiple
1419 <literal><target></literal>
1420 elements, and a subsequent
1421 <literal>multi</literal>
1422 filter. Here, for example, is a complete configuration that
1423 broadcasts searches to both the Library of Congress catalogue and
1424 Index Data's tiny testing database of MARC records:
1426 <screen><![CDATA[<?xml version="1.0"?>
1427 <metaproxy xmlns="http://indexdata.com/metaproxy" version="1.0">
1428 <start route="start"/>
1431 <filter type="frontend_net">
1432 <threads>10</threads>
1435 <filter type="virt_db">
1437 <database>lc</database>
1438 <target>z3950.loc.gov:7090/voyager</target>
1441 <database>marc</database>
1442 <target>indexdata.com/marc</target>
1445 <database>all</database>
1446 <target>z3950.loc.gov:7090/voyager</target>
1447 <target>indexdata.com/marc</target>
1450 <filter type="multi"/>
1451 <filter type="z3950_client">
1452 <timeout>30</timeout>
1454 <filter type="bounce"/>
1457 </metaproxy>]]></screen>
1460 <literal>virt_db</literal>
1461 filter that specifies multiple
1462 <literal><target></literal>
1463 elements but without a subsequent
1464 <literal>multi</literal>
1465 filter yields surprising and undesirable results, as will be
1466 described below. Don't do that.)
1469 Metaproxy can be invoked with this configuration as follows:
1471 <screen>../src/metaproxy --config config-simple-multi.xml</screen>
1473 And thereafter, Z39.50 clients can connect to the running server
1474 (on port 9000, as specified in the configuration) and search in
1475 any of the databases
1476 <literal>lc</literal> (the Library of Congress catalogue),
1477 <literal>marc</literal> (Index Data's test database of MARC records)
1479 <literal>all</literal> (both of these). As an example, a session
1480 using the YAZ command-line client <literal>yaz-client</literal> is
1481 here included (edited for brevity and clarity):
1483 <screen><![CDATA[$ yaz-client @:9000
1487 Search was a success.
1488 Number of hits: 10000, setno 1
1492 Search was a success.
1493 Number of hits: 10, setno 3
1497 Search was a success.
1498 Number of hits: 10010, setno 4
1501 [marc]Record type: USmarc
1504 005 00000000000000.0
1505 008 910710c19910701nju 00010 eng
1509 100 10 $a Jack Collins
1510 245 10 $a How to program a computer
1516 [VOYAGER]Record type: USmarc
1518 005 20041229102447.0
1519 008 030910s2004 caua 000 0 eng
1520 035 $a (DLC) 2003112666
1521 906 $a 7 $b cbc $c orignew $d 4 $e epcn $f 20 $g y-gencatlg
1522 925 0 $a acquire $b 1 shelf copy $x policy default
1523 955 $a pc10 2003-09-10 $a pv12 2004-06-23 to SSCD; $h sj05 2004-11-30 $e sj05 2004-11-30 to Shelf.
1526 040 $a DLC $c DLC $d DLC
1527 050 00 $a MLCM 2004/03312 (G)
1528 245 10 $a 007, everything or nothing : $b Prima's official strategy guide / $c created by Kaizen Media Group.
1529 246 3 $a Double-O-seven, everything or nothing
1530 246 30 $a Prima's official strategy guide
1531 260 $a Roseville, CA : $b Prima Games, $c c2004.
1532 300 $a 161 p. : $b col. ill. ; $c 28 cm.
1533 500 $a "Platforms: Nintendo GameCube, Macintosh, PC, PlayStation 2 computer entertainment system, Xbox"--P. [4] of cover.
1534 650 0 $a Video games.
1535 710 2 $a Kaizen Media Group.
1536 856 42 $3 Publisher description $u http://www.loc.gov/catdir/description/random052/2003112666.html
1541 As can be seen, the first record in the result set is from the
1542 Index Data test database, and the second from the Library of
1543 Congress database. The result-set continues alternating records
1544 round-robin style until the point where one of the databases'
1545 records are exhausted.
1548 This example uses only two back-end databases; more may be used.
1549 There is no limitation imposed on the number of databases that may
1550 be metasearched in this way: issues of resource usage and
1551 administrative complexity dictate the practical limits.
1554 What happens when one of the databases doesn't respond? By default,
1555 the entire multi-database search fails, and the appropriate
1556 diagnostic is returned to the client. This is usually appropriate
1557 during development, when technicians need maximum information, but
1558 can be inconvenient in deployment, when users typically don't want
1559 to be bothered with problems of this kind and prefer just to get
1560 the records from the databases that are available. To obtain this
1561 latter behavior add an empty
1562 <literal><hideunavailable></literal>
1564 <literal>multi</literal> filter:
1566 <screen><![CDATA[ <filter type="multi">
1568 </filter>]]></screen>
1570 Under this regime, an error is reported to the client only if
1571 <emphasis>all</emphasis> the databases in a multi-database search
1577 <section id="multidb.what">
1578 <title>What's going on?</title>
1580 <title>Lark's vomit</title>
1582 This section goes into a level of technical detail that is
1583 probably not necessary in order to configure and use Metaproxy.
1584 It is provided only for those who like to know how things work.
1585 You should feel free to skip on to the next section if this one
1586 doesn't seem like fun.
1590 Hold on tight - this may get a little hairy.
1593 In the general course of things, a Z39.50 Init request may carry
1594 with it an otherInfo packet of type <literal>VAL_PROXY</literal>,
1595 whose value indicates the address of a Z39.50 server to which the
1596 ultimate connection is to be made. (This otherInfo packet is
1597 supported by YAZ-based Z39.50 clients and servers, but has not yet
1598 been ratified by the Maintenance Agency and so is not widely used
1599 in non-Index Data software. We're working on it.)
1600 The <literal>VAL_PROXY</literal> packet functions
1601 analogously to the absoluteURI-style Request-URI used with the GET
1602 method when a web browser asks a proxy to forward its request: see
1604 <ulink url="http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html#sec5.1.2"
1605 >Request-URI</ulink>
1607 <ulink url="http://www.w3.org/Protocols/rfc2616/rfc2616.html"
1608 >the HTTP 1.1 specification</ulink>.
1611 Within Metaproxy, Search requests that are part of the same
1612 session as an Init request that carries a
1613 <literal>VAL_PROXY</literal> otherInfo are also annotated with the
1614 same information. The role of the <literal>virt_db</literal>
1615 filter is to rewrite this otherInfo packet dependent on the
1616 virtual database that the client wants to search.
1619 When Metaproxy receives a Z39.50 Init request from a client, it
1620 doesn't immediately forward that request to the back-end server.
1621 Why not? Because it doesn't know <emphasis>which</emphasis>
1622 back-end server to forward it to until the client sends a Search
1623 request that specifies the database that it wants to search in.
1624 Instead, it just treasures the Init request up in its heart; and,
1625 later, the first time the client does a search on one of the
1626 specified virtual databases, a connection is forged to the
1627 appropriate server and the Init request is forwarded to it. If,
1628 later in the session, the same client searches in a different
1629 virtual database, then a connection is forged to the server that
1630 hosts it, and the same cached Init request is forwarded there,
1634 All of this clever Init-delaying is done by the
1635 <literal>frontend_net</literal> filter. The
1636 <literal>virt_db</literal> filter knows nothing about it; in
1637 fact, because the Init request that is received from the client
1638 doesn't get forwarded until a Search request is received, the
1639 <literal>virt_db</literal> filter (and the
1640 <literal>z3950_client</literal> filter behind it) doesn't even get
1641 invoked at Init time. The <emphasis>only</emphasis> thing that a
1642 <literal>virt_db</literal> filter ever does is rewrite the
1643 <literal>VAL_PROXY</literal> otherInfo in the requests that pass
1647 It is possible for a <literal>virt_db</literal> filter to contain
1649 <literal><target></literal>
1650 elements. What does this mean? Only that the filter will add
1651 multiple <literal>VAL_PROXY</literal> otherInfo packets to the
1652 Search requests that pass through it. That's because the virtual
1653 DB filter is dumb, and does exactly what it's told - no more, no
1655 If a Search request with multiple <literal>VAL_PROXY</literal>
1656 otherInfo packets reaches a <literal>z3950_client</literal>
1657 filter, this is an error. That filter doesn't know how to deal
1658 with multiple targets, so it will either just pick one and search
1659 in it, or (better) fail with an error message.
1662 The <literal>multi</literal> filter comes to the rescue! This is
1663 the only filter that knows how to deal with multiple
1664 <literal>VAL_PROXY</literal> otherInfo packets, and it does so by
1665 making multiple copies of the entire Search request: one for each
1666 <literal>VAL_PROXY</literal>. Each of these new copies is then
1667 passed down through the remaining filters in the route. (The
1668 copies are handled in parallel though the
1669 spawning of new threads.) Since the copies each have only one
1670 <literal>VAL_PROXY</literal> otherInfo, they can be handled by the
1671 <literal>z3950_client</literal> filter, which happily deals with
1672 each one individually. When the results of the individual
1673 searches come back up to the <literal>multi</literal> filter, it
1674 merges them into a single Search response, which is what
1675 eventually makes it back to the client.
1680 <imagedata fileref="multi.pdf" format="PDF" scale="50"/>
1683 <imagedata fileref="multi.png" format="PNG"/>
1686 <!-- Fall back if none of the images can be used -->
1688 [Here there should be a diagram showing the progress of
1689 packages through the filters during a simple virtual-database
1690 search and a multi-database search, but is seems that your
1691 tool chain has not been able to include the diagram in this
1696 <para>A picture is worth a thousand words (but only five hundred on 64-bit architectures)</para>
1703 <chapter id="sru-server">
1704 <title>Combined SRU webservice and Z39.50 server configuration</title>
1706 Metaproxy can act as
1707 <ulink url="&url.sru;">SRU</ulink> and
1708 web service server, which translates web service requests to
1709 <ulink url="&url.z39.50;">ANSI/NISO Z39.50</ulink> packages and
1710 sends them off to common available targets.
1713 A typical setup for this operation needs a filter route including the
1717 <table id="sru-server-table-config" frame="top">
1718 <title>SRU/Z39.50 Server Filter Route Configuration</title>
1722 <entry>Filter</entry>
1723 <entry>Importance</entry>
1724 <entry>Purpose</entry>
1730 <entry><literal>frontend_net</literal></entry>
1731 <entry>required</entry>
1732 <entry>Accepting HTTP connections and passing them to following
1733 filters. Since this filter also accepts Z39.50 connections, the
1734 server works as SRU and Z39.50 server on the same port.</entry>
1737 <entry><literal>sru_z3950</literal></entry>
1738 <entry>required</entry>
1739 <entry>Accepting SRU GET/POST/SOAP explain and
1740 searchRetrieve requests for the the configured databases.
1741 Explain requests are directly served from the static XML configuration.
1742 SearchRetrieve requests are
1743 transformed to Z39.50 search and present packages.
1744 All other HTTP and Z39.50 packages are passed unaltered.</entry>
1747 <entry><literal>http_file</literal></entry>
1748 <entry>optional</entry>
1749 <entry>Serving HTTP requests from the filesystem. This is only
1750 needed if the server should serve XSLT stylesheets, static HTML
1751 files or Java Script for thin browser based clients.
1752 Z39.50 packages are passed unaltered.</entry>
1755 <entry><literal>cql_rpn</literal></entry>
1756 <entry>required</entry>
1757 <entry>Usually, Z39.50 servers do not talk CQL, hence the
1758 translation of the CQL query language to RPN is mandatory in
1759 most cases. Affects only Z39.50 search packages.</entry>
1762 <entry><literal>record_transform</literal></entry>
1763 <entry>optional</entry>
1764 <entry>Some Z39.50 backend targets can not present XML record
1765 syntaxes in common wanted element sets. using this filter, one
1766 can transform binary MARC records to MARCXML records, and
1767 further transform those to any needed XML schema/format by XSLT
1768 transformations. Changes only Z39.50 present packages.</entry>
1771 <entry><literal>session_shared</literal></entry>
1772 <entry>optional</entry>
1773 <entry>The stateless nature of web services requires frequent
1774 re-searching of the same targets for display of paged result set
1775 records. This might be an unacceptable burden for the accessed
1776 backend Z39.50 targets, and this mosule can be added for
1777 efficient backend target resource pooling.</entry>
1780 <entry><literal>z3950_client</literal></entry>
1781 <entry>required</entry>
1782 <entry>Finally, a Z39.50 package sink is needed in the filter
1783 chain to provide the response packages. The Z39.50 client module
1784 is used to access external targets over the network, but any
1785 coming local Z39.50 package sink could be used instead of.</entry>
1788 <entry><literal>bounce</literal></entry>
1789 <entry>required</entry>
1790 <entry>Any Metaproxy package arriving here did not do so by
1791 purpose, and is bounced back with connection closure. this
1792 prevents inifinite package hanging inside the SRU server.</entry>
1798 A typical minimal example <ulink url="&url.sru;">SRU</ulink>
1799 server configuration file is found in the tarball distribution at
1800 <literal>etc/config-sru-to-z3950.xml</literal>.
1803 Off course, any other metaproxy modules can be integrated into a
1804 SRU server solution, including, but not limited to, load balancing,
1805 multiple target querying
1806 (see <xref linkend="multidb"/>), and complex RPN query rewrites.
1813 <chapter id="extensions">
1814 <title>Writing extensions for Metaproxy</title>
1815 <para>### To be written</para>
1821 <chapter id="classes">
1822 <title>Classes in the Metaproxy source code</title>
1825 <section id="classes-introductory-notes">
1826 <title>Introductory notes</title>
1828 <emphasis>Stop! Do not read this!</emphasis>
1829 You won't enjoy it at all. You should just skip ahead to
1830 <xref linkend="reference"/>,
1832 <!-- The remainder of this paragraph is lifted verbatim from
1833 Douglas Adams' _Hitch Hiker's Guide to the Galaxy_, chapter 8 -->
1834 you things you really need to know, like the fact that the
1835 fabulously beautiful planet Bethselamin is now so worried about
1836 the cumulative erosion by ten billion visiting tourists a year
1837 that any net imbalance between the amount you eat and the amount
1838 you excrete whilst on the planet is surgically removed from your
1839 bodyweight when you leave: so every time you go to the lavatory it
1840 is vitally important to get a receipt.
1843 This chapter contains documentation of the Metaproxy source code, and is
1844 of interest only to maintainers and developers. If you need to
1845 change Metaproxy's behavior or write a new filter, then you will most
1846 likely find this chapter helpful. Otherwise it's a waste of your
1847 good time. Seriously: go and watch a film or something.
1848 <citetitle>This is Spinal Tap</citetitle> is particularly good.
1851 Still here? OK, let's continue.
1854 In general, classes seem to be named big-endianly, so that
1855 <literal>FactoryFilter</literal> is not a filter that filters
1856 factories, but a factory that produces filters; and
1857 <literal>FactoryStatic</literal> is a factory for the statically
1858 registered filters (as opposed to those that are dynamically
1863 <section id="individual.classes">
1864 <title>Individual classes</title>
1866 The classes making up the Metaproxy application are here listed by
1867 class-name, with the names of the source files that define them in
1871 <section id="class-FactoryFilter">
1872 <title><literal>mp::FactoryFilter</literal>
1873 (<filename>factory_filter.cpp</filename>)</title>
1875 A factory class that exists primarily to provide the
1876 <literal>create()</literal> method, which takes the name of a
1877 filter class as its argument and returns a new filter of that
1878 type. To enable this, the factory must first be populated by
1879 calling <literal>add_creator()</literal> for static filters (this
1880 is done by the <literal>FactoryStatic</literal> class, see below)
1881 and <literal>add_creator_dyn()</literal> for filters loaded
1886 <section id="class-FactoryStatic">
1887 <title><literal>mp::FactoryStatic</literal>
1888 (<filename>factory_static.cpp</filename>)</title>
1890 A subclass of <literal>FactoryFilter</literal> which is
1891 responsible for registering all the statically defined filter
1892 types. It does this by knowing about all those filters'
1893 structures, which are listed in its constructor. Merely
1894 instantiating this class registers all the static classes. It is
1895 for the benefit of this class that <literal>struct
1896 metaproxy_1_filter_struct</literal> exists, and that all the filter
1897 classes provide a static object of that type.
1901 <section id="class-filter-Base">
1902 <title><literal>mp::filter::Base</literal>
1903 (<filename>filter.cpp</filename>)</title>
1905 The virtual base class of all filters. The filter API is, on the
1906 surface at least, extremely simple: two methods.
1907 <literal>configure()</literal> is passed an XML DOM tree representing
1908 that part of the configuration file that pertains to this filter
1909 instance, and is expected to walk that tree extracting relevant
1910 information. And <literal>process()</literal> processes a
1911 package (see below). That surface simplicity is a bit
1912 misleading, as <literal>process()</literal> needs to know a lot
1913 about the <literal>Package</literal> class in order to do
1918 <section id="class-AuthSimple">
1919 <title><literal>mp::filter::AuthSimple</literal>,
1920 <literal>Backend_test</literal>, etc.
1921 (<filename>filter_auth_simple.cpp</filename>,
1922 <filename>filter_backend_test.cpp</filename>, etc.)</title>
1924 Individual filters. Each of these is implemented by a header and
1925 a source file, named <filename>filter_*.hpp</filename> and
1926 <filename>filter_*.cpp</filename> respectively. All the header
1927 files should be pretty much identical, in that they declare the
1928 class, including a private <literal>Rep</literal> class and a
1929 member pointer to it, and the two public methods.
1932 The source file for each filter needs to supply:
1937 A definition of the private <literal>Rep</literal> class.
1942 Some boilerplate constructors and destructors.
1947 A <literal>configure()</literal> method that uses the
1948 appropriate XML fragment.
1953 Most important, the <literal>process()</literal> method that
1954 does all the actual work.
1960 <section id="class-Package">
1961 <title><literal>mp::Package</literal>
1962 (<filename>package.cpp</filename>)</title>
1964 Represents a package on its way through the series of filters
1965 that make up a route. This is essentially a Z39.50 or SRU APDU
1966 together with information about where it came from, which is
1967 modified as it passes through the various filters.
1971 <section id="class-Pipe">
1972 <title><literal>mp::Pipe</literal>
1973 (<filename>pipe.cpp</filename>)</title>
1975 This class provides a compatibility layer so that we have an IPC
1976 mechanism that works the same under Unix and Windows. It's not
1977 particularly exciting.
1981 <section id="class-RouterChain">
1982 <title><literal>mp::RouterChain</literal>
1983 (<filename>router_chain.cpp</filename>)</title>
1989 <section id="class-RouterFleXML">
1990 <title><literal>mp::RouterFleXML</literal>
1991 (<filename>router_flexml.cpp</filename>)</title>
1997 <section id="class-Session">
1998 <title><literal>mp::Session</literal>
1999 (<filename>session.cpp</filename>)</title>
2005 <section id="class-ThreadPoolSocketObserver">
2006 <title><literal>mp::ThreadPoolSocketObserver</literal>
2007 (<filename>thread_pool_observer.cpp</filename>)</title>
2013 <section id="class-util">
2014 <title><literal>mp::util</literal>
2015 (<filename>util.cpp</filename>)</title>
2017 A namespace of various small utility functions and classes,
2018 collected together for convenience. Most importantly, includes
2019 the <literal>mp::util::odr</literal> class, a wrapper for YAZ's
2024 <section id="class-xml">
2025 <title><literal>mp::xml</literal>
2026 (<filename>xmlutil.cpp</filename>)</title>
2028 A namespace of various XML utility functions and classes,
2029 collected together for convenience.
2035 <section id="other.source.files">
2036 <title>Other Source Files</title>
2038 In addition to the Metaproxy source files that define the classes
2039 described above, there are a few additional files which are
2040 briefly described here:
2044 <term><literal>metaproxy_prog.cpp</literal></term>
2047 The main function of the <command>metaproxy</command> program.
2052 <term><literal>ex_router_flexml.cpp</literal></term>
2055 Identical to <literal>metaproxy_prog.cpp</literal>: it's not clear why.
2060 <term><literal>test_*.cpp</literal></term>
2063 Unit-tests for various modules.
2069 ### Still to be described:
2070 <literal>ex_filter_frontend_net.cpp</literal>,
2071 <literal>filter_dl.cpp</literal>,
2072 <literal>plainfile.cpp</literal>,
2073 <literal>tstdl.cpp</literal>.
2079 <reference id="reference">
2080 <title>Reference</title>
2081 <partintro id="reference-introduction">
2083 The material in this chapter is drawn directly from the individual
2084 manual entries. In particular, the Metaproxy invocation section is
2085 available using <command>man metaproxy</command>, and the section
2086 on each individual filter is available using the name of the filter
2087 as the argument to the <command>man</command> command.
2093 <appendix id="license">
2094 <title>License</title>
2099 Metaproxy is free software; you can redistribute it and/or modify it under
2100 the terms of the GNU General Public License as published by the Free
2101 Software Foundation; either version 2, or (at your option) any later
2106 Metaproxy is distributed in the hope that it will be useful, but WITHOUT ANY
2107 WARRANTY; without even the implied warranty of MERCHANTABILITY or
2108 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
2113 You should have received a copy of the GNU General Public License
2114 along with Metaproxy; see the file LICENSE. If not, write to the
2115 Free Software Foundation,
2116 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
2124 <!-- Keep this comment at the end of the file
2127 nxml-child-indent: 1