1 <!-- $Id: book.xml,v 1.18 2006-04-26 15:32:22 mike Exp $ -->
3 <title>Metaproxy - User's Guide and Reference</title>
5 <firstname>Mike</firstname><surname>Taylor</surname>
8 <firstname>Adam</firstname><surname>Dickmeiss</surname>
12 <holder>Index Data ApS</holder>
16 Metaproxy is a universal router, proxy and encapsulated
17 metasearcher for information retrieval protocols. It accepts,
18 processes, interprets and redirects requests from IR clients using
19 standard protocols such as
20 <ulink url="&url.z39.50;">ANSI/NISO Z39.50</ulink>
21 (and in the future <ulink url="&url.sru;">SRU</ulink>
22 and <ulink url="&url.srw;">SRW</ulink>), as
23 well as functioning as a limited
24 <ulink url="&url.http;">HTTP</ulink> server.
25 Metaproxy is configured by an XML file which
26 specifies how the software should function in terms of routes that
27 the request packets can take through the proxy, each step on a
28 route being an instantiation of a filter. Filters come in many
29 types, one for each operation: accepting Z39.50 packets, logging,
30 query transformation, multiplexing, etc. Further filter-types can
31 be added as loadable modules to extend Metaproxy functionality,
35 The terms under which Metaproxy will be distributed have yet to be
36 established, but it will not necessarily be open source; so users
37 should not at this stage redistribute the code without explicit
38 written permission from the copyright holders, Index Data ApS.
43 <imagedata fileref="common/id.png" format="PNG"/>
46 <imagedata fileref="common/id.eps" format="EPS"/>
53 <chapter id="introduction">
54 <title>Introduction</title>
58 <ulink url="&url.metaproxy;">Metaproxy</ulink>
59 is a standalone program that acts as a universal router, proxy and
60 encapsulated metasearcher for information retrieval protocols such
61 as <ulink url="&url.z39.50;">Z39.50</ulink>, and in the future
62 <ulink url="&url.sru;">SRU</ulink> and <ulink url="&url.srw;">SRW</ulink>.
63 To clients, it acts as a server of these protocols: it can be searched,
64 records can be retrieved from it, etc.
65 To servers, it acts as a client: it searches in them,
66 retrieves records from them, etc. it satisfies its clients'
67 requests by transforming them, multiplexing them, forwarding them
68 on to zero or more servers, merging the results, transforming
69 them, and delivering them back to the client. In addition, it
70 acts as a simple <ulink url="&url.http;">HTTP</ulink> server; support
71 for further protocols can be added in a modular fashion, through the
72 creation of new filters.
77 Fish, bananas, cold pyjamas,
78 Mutton, beef and trout!
79 - attributed to Cole Porter.
82 Metaproxy is a more capable alternative to
83 <ulink url="&url.yazproxy;">YAZ Proxy</ulink>,
84 being more powerful, flexible, configurable and extensible. Among
85 its many advantages over the older, more pedestrian work are
86 support for multiplexing (encapsulated metasearching), routing by
87 database name, authentication and authorisation and serving local
88 files via HTTP. Equally significant, its modular architecture
89 facilitites the creation of pluggable modules implementing further
93 This manual will briefly describe Metaproxy's licensing situation
94 before giving an overview of its architecture, then discussing the
95 key concept of a filter in some depth and giving an overview of
96 the various filter types, then discussing the configuration file
97 format. After this come several optional chapters which may be
98 freely skipped: a detailed discussion of virtual databases and
99 multi-database searching, some notes on writing extensions
100 (additional filter types) and a high-level description of the
101 source code. Finally comes the reference guide, which contains
102 instructions for invoking the <command>metaproxy</command>
103 program, and detailed information on each type of filter,
110 <chapter id="licence">
111 <title>The Metaproxy Licence</title>
113 <emphasis role="strong">
114 No decision has yet been made on the terms under which
115 Metaproxy will be distributed.
117 It is possible that, unlike
118 other Index Data products, metaproxy may not be released under a
119 free-software licence such as the GNU GPL. Until a decision is
120 made and a public statement made, then, and unless it has been
121 delivered to you other specific terms, please treat Metaproxy as
122 though it were proprietary software.
123 The code should not be redistributed without explicit
124 written permission from the copyright holders, Index Data ApS.
128 <chapter id="installation">
129 <title>Installation</title>
131 Metaproxy depends on the following tools/libraries:
133 <varlistentry><term><ulink url="&url.yazplusplus;">YAZ++</ulink></term>
136 This is a C++ library based on <ulink url="&url.yaz;">YAZ</ulink>.
140 <varlistentry><term><ulink url="&url.libxslt;">Libxslt</ulink></term>
142 <para>This is an XSLT processor - based on
143 <ulink url="&url.libxml2;">Libxml2</ulink>. Both Libxml2 and
144 Libxslt must be installed with the development components
145 (header files, etc.) as well as the run-time libraries.
149 <varlistentry><term><ulink url="&url.boost;">Boost</ulink></term>
152 The popular C++ library.
159 In order to compile Metaproxy a modern C++ compiler is
160 required. Boost, in particular, requires the C++ compiler
161 to facilitate the newest features. Refer to Boost
162 <ulink url="&url.boost.compilers.status;">Compiler Status</ulink>
163 for more information.
166 We have succesfully used Metaproxy with Boost using the compilers
167 <ulink url="&url.gcc;">GCC</ulink> version 4.0 and
168 <ulink url="&url.vstudio;">Microsoft Visual Studio</ulink> 2003/2005.
171 <section id="installation.unix">
172 <title>Installation on Unix (from Source)</title>
174 Here is a quick step-by-step guide on how to compile all the
175 tools that Metaproxy uses. Only few systems have none of the required
176 tools binary packages. If, for example, Libxml2/libxslt are already
177 installed as development packages use those (and omit compilation).
184 gunzip -c libxml2-version.tar.gz|tar xf -
192 gunzip -c libxslt-version.tar.gz|tar xf -
203 gunzip -c yaz-version.tar.gz|tar xf -
211 gunzip -c yazpp-version.tar.gz|tar xf -
222 gunzip -c boost-version.tar.gz|tar xf -
233 gunzip -c metaproxy-version.tar.gz|tar xf -
242 <section id="installation.debian">
243 <title>Installation on Debian</title>
249 <section id="installation.windows">
250 <title>Installation on Windows</title>
257 <chapter id="architecture">
258 <title>The Metaproxy Architecture</title>
260 The Metaproxy architecture is based on three concepts:
261 the <emphasis>package</emphasis>,
262 the <emphasis>route</emphasis>
263 and the <emphasis>filter</emphasis>.
267 <term>Packages</term>
270 A package is request or response, encoded in some protocol,
271 issued by a client, making its way through Metaproxy, send to or
272 received from a server, or sent back to the client.
275 The core of a package is the protocol unit - for example, a
276 Z39.50 Init Request or Search Response, or an SRU searchRetrieve
277 URL or Explain Response. In addition to this core, a package
278 also carries some extra information added and used by Metaproxy
282 In general, packages are doctored as they pass through
283 Metaproxy. For example, when the proxy performs authentication
284 and authorisation on a Z39.50 Init request, it removes the
285 authentication credentials from the package so that they are not
286 passed onto the back-end server; and when search-response
287 packages are obtained from multiple servers, they are merged
288 into a single unified package that makes its way back to the
297 Packages make their way through routes, which can be thought of
298 as programs that operate on the package data-type. Each
299 incoming package initially makes its way through a default
300 route, but may be switched to a different route based on various
301 considerations. Routes are made up of sequences of filters (see
310 Filters provide the individual instructions within a route, and
311 effect the necessary transformations on packages. A particular
312 configuration of Metaproxy is essentially a set of filters,
313 described by configuration details and arranged in order in one
314 or more routes. There are many kinds of filter - about a dozen
315 at the time of writing with more appearing all the time - each
316 performing a specific function and configured by different
320 The word ``filter'' is sometimes used rather loosely, in two
321 different ways: it may be used to mean a particular
322 <emphasis>type</emphasis> of filter, as when we speak of ``the
323 auth_simplefilter'' or ``the multi filter''; or it may be used
324 to be a specific <emphasis>instance</emphasis> of a filter
325 within a Metaproxy configuration. For example, a single
326 configuration will often contain multiple instances of the
327 <literal>z3950_client</literal> filter. In
328 operational terms, of these is a separate filter. In practice,
329 context always make it clear which sense of the word ``filter''
333 Extensibility of Metaproxy is primarily through the creation of
334 plugins that provide new filters. The filter API is small and
335 conceptually simple, but there are many details to master. See
337 <link linkend="extensions">extensions</link>.
343 Since packages are created and handled by the system itself, and
344 routes are conceptually simple, most of the remainder of this
345 document concentrates on filters. After a brief overview of the
346 filter types follows, along with some thoughts on possible future
353 <chapter id="filters">
354 <title>Filters</title>
358 <title>Introductory notes</title>
360 It's useful to think of Metaproxy as an interpreter providing a small
361 number of primitives and operations, but operating on a very
362 complex data type, namely the ``package''.
365 A package represents a Z39.50 or SRU/W request (whether for Init,
366 Search, Scan, etc.) together with information about where it came
367 from. Packages are created by front-end filters such as
368 <literal>frontend_net</literal> (see below), which reads them from
369 the network; other front-end filters are possible. They then pass
370 along a route consisting of a sequence of filters, each of which
371 transforms the package and may also have side-effects such as
372 generating logging. Eventually, the route will yield a response,
373 which is sent back to the origin.
376 There are many kinds of filter: some that are defined statically
377 as part of Metaproxy, and others may be provided by third parties
378 and dynamically loaded. They all conform to the same simple API
379 of essentially two methods: <function>configure()</function> is
380 called at startup time, and is passed a DOM tree representing that
381 part of the configuration file that pertains to this filter
382 instance: it is expected to walk that tree extracting relevant
383 information; and <function>process()</function> is called every
384 time the filter has to processes a package.
387 While all filters provide the same API, there are different modes
388 of functionality. Some filters are sources: they create
390 (<literal>frontend_net</literal>);
391 others are sinks: they consume packages and return a result
392 (<literal>z3950_client</literal>,
393 <literal>backend_test</literal>,
394 <literal>http_file</literal>);
395 the others are true filters, that read, process and pass on the
396 packages they are fed
397 (<literal>auth_simple</literal>,
398 <literal>log</literal>,
399 <literal>multi</literal>,
400 <literal>query_rewrite</literal>,
401 <literal>session_shared</literal>,
402 <literal>template</literal>,
403 <literal>virt_db</literal>).
408 <section id="overview.filter.types">
409 <title>Overview of filter types</title>
411 We now briefly consider each of the types of filter supported by
412 the core Metaproxy binary. This overview is intended to give a
413 flavour of the available functionality; more detailed information
414 about each type of filter is included below in
415 <link linkend="filterref"
416 >the reference guide to Metaproxy filters</link>.
419 The filters are here named by the string that is used as the
420 <literal>type</literal> attribute of a
421 <literal><filter></literal> element in the configuration
422 file to request them, with the name of the class that implements
423 them in parentheses. (The classname is not needed for normal
424 configuration and use of Metaproxy; it is useful only to
428 The filters are here listed in alphabetical order:
432 <title><literal>auth_simple</literal>
433 (mp::filter::AuthSimple)</title>
435 Simple authentication and authorisation. The configuration
436 specifies the name of a file that is the user register, which
437 lists <varname>username</varname>:<varname>password</varname>
438 pairs, one per line, colon separated. When a session begins, it
439 is rejected unless username and passsword are supplied, and match
440 a pair in the register. The configuration file may also specific
441 the name of another file that is the target register: this lists
442 lists <varname>username</varname>:<varname>dbname</varname>,<varname>dbname</varname>...
443 sets, one per line, with multiple database names separated by
444 commas. When a search is processed, it is rejected unless the
445 database to be searched is one of those listed as available to
451 <title><literal>backend_test</literal>
452 (mp::filter::Backend_test)</title>
454 A sink that provides dummy responses in the manner of the
455 <literal>yaz-ztest</literal> Z39.50 server. This is useful only
456 for testing. Seriously, you don't need this. Pretend you didn't
457 even read this section.
462 <title><literal>frontend_net</literal>
463 (mp::filter::FrontendNet)</title>
465 A source that accepts Z39.50 connections from a port
466 specified in the configuration, reads protocol units, and
467 feeds them into the next filter in the route. When the result is
468 revceived, it is returned to the original origin.
473 <title><literal>http_file</literal>
474 (mp::filter::HttpFile)</title>
476 A sink that returns the contents of files from the local
477 filesystem in response to HTTP requests. (Yes, Virginia, this
478 does mean that Metaproxy is also a Web-server in its spare time. So
479 far it does not contain either an email-reader or a Lisp
480 interpreter, but that day is surely coming.)
485 <title><literal>log</literal>
486 (mp::filter::Log)</title>
488 Writes logging information to standard output, and passes on
489 the package unchanged.
494 <title><literal>multi</literal>
495 (mp::filter::Multi)</title>
497 Performs multicast searching.
499 <link linkend="multidb">the extended discussion</link>
500 of virtual databases and multi-database searching below.
505 <title><literal>query_rewrite</literal>
506 (mp::filter::QueryRewrite)</title>
508 Rewrites Z39.50 Type-1 and Type-101 (``RPN'') queries by a
509 three-step process: the query is transliterated from Z39.50
510 packet structures into an XML representation; that XML
511 representation is transformed by an XSLT stylesheet; and the
512 resulting XML is transliterated back into the Z39.50 packet
518 <title><literal>session_shared</literal>
519 (mp::filter::SessionShared)</title>
521 When this is finished, it will implement global sharing of
522 result sets (i.e. between threads and therefore between
523 clients), yielding performance improvements especially when
524 incoming requests are from a stateless environment such as a
525 web-server, in which the client process representing a session
526 might be any one of many. However:
530 This filter is not yet completed.
536 <title><literal>template</literal>
537 (mp::filter::Template)</title>
539 Does nothing at all, merely passing the packet on. (Maybe it
540 should be called <literal>nop</literal> or
541 <literal>passthrough</literal>?) This exists not to be used, but
542 to be copied - to become the skeleton of new filters as they are
543 written. As with <literal>backend_test</literal>, this is not
544 intended for civilians.
549 <title><literal>virt_db</literal>
550 (mp::filter::Virt_db)</title>
552 Performs virtual database selection: based on the name of the
553 database in the search request, a server is selected, and its
554 address added to the request in a <literal>VAL_PROXY</literal>
555 otherInfo packet. It will subsequently be used by a
556 <literal>z3950_client</literal> filter.
558 <link linkend="multidb">the extended discussion</link>
559 of virtual databases and multi-database searching below.
564 <title><literal>z3950_client</literal>
565 (mp::filter::Z3950Client)</title>
567 Performs Z39.50 searching and retrieval by proxying the
568 packages that are passed to it. Init requests are sent to the
569 address specified in the <literal>VAL_PROXY</literal> otherInfo
570 attached to the request: this may have been specified by client,
571 or generated by a <literal>virt_db</literal> filter earlier in
572 the route. Subsequent requests are sent to the same address,
573 which is remembered at Init time in a Session object.
579 <section id="future.directions">
580 <title>Future directions</title>
582 Some other filters that do not yet exist, but which would be
583 useful, are briefly described. These may be added in future
584 releases (or may be created by third parties, as loadable
590 <term><literal>frontend_cli</literal> (source)</term>
593 Command-line interface for generating requests.
598 <term><literal>frontend_sru</literal> (source)</term>
601 Receive SRU (and perhaps SRW) requests.
606 <term><literal>sru2z3950</literal> (filter)</term>
609 Translate SRU requests into Z39.50 requests.
614 <term><literal>sru_client</literal> (sink)</term>
617 SRU searching and retrieval.
622 <term><literal>srw_client</literal> (sink)</term>
625 SRW searching and retrieval.
630 <term><literal>opensearch_client</literal> (sink)</term>
633 A9 OpenSearch searching and retrieval.
643 <chapter id="configuration">
644 <title>Configuration: the Metaproxy configuration file format</title>
648 <title>Introductory notes</title>
650 If Metaproxy is an interpreter providing operations on packages, then
651 its configuration file can be thought of as a program for that
652 interpreter. Configuration is by means of a single file, the name
653 of which is supplied as the sole command-line argument to the
654 <command>metaproxy</command> program. (See
655 <link linkend="progref">the reference guide</link>
656 below for more information on invoking Metaproxy.)
659 The configuration files are written in XML. (But that's just an
660 implementation detail - they could just as well have been written
661 in YAML or Lisp-like S-expressions, or in a custom syntax.)
664 Since XML has been chosen, an XML schema,
665 <filename>config.xsd</filename>, is provided for validating
666 configuration files. This file is supplied in the
667 <filename>etc</filename> directory of the Metaproxy distribution. It
668 can be used by (among other tools) the <command>xmllint</command>
669 program supplied as part of the <literal>libxml2</literal>
673 xmllint --noout --schema etc/config.xsd my-config-file.xml
676 (A recent version of <literal>libxml2</literal> is required, as
677 support for XML Schemas is a relatively recent addition.)
681 <section id="overview.xml.structure">
682 <title>Overview of XML structure</title>
684 All elements and attributes are in the namespace
685 <ulink url="http://indexdata.dk/yp2/config/1"/>.
686 This is most easily achieved by setting the default namespace on
687 the top-level element, as here:
690 <yp2 xmlns="http://indexdata.dk/yp2/config/1">
693 The top-level element is <yp2>. This contains a
694 <start> element, a <filters> element and a
695 <routes> element, in that order. <filters> is
696 optional; the other two are mandatory. All three are
700 The <start> element is empty, but carries a
701 <literal>route</literal> attribute, whose value is the name of
702 route at which to start running - analogous to the name of the
703 start production in a formal grammar.
706 If present, <filters> contains zero or more <filter>
707 elements. Each filter carries a <literal>type</literal> attribute
708 which specifies what kind of filter is being defined
709 (<literal>frontend_net</literal>, <literal>log</literal>, etc.)
710 and contain various elements that provide suitable configuration
711 for a filter of its type. The filter-specific elements are
713 <link linkend="filterref">the reference guide below</link>.
714 Filters defined in this part of the file must carry an
715 <literal>id</literal> attribute so that they can be referenced
719 <routes> contains one or more <route> elements, each
720 of which must carry an <literal>id</literal> element. One of the
721 routes must have the ID value that was specified as the start
722 route in the <start> element's <literal>route</literal>
723 attribute. Each route contains zero or more <filter>
724 elements. These are of two types. They may be empty, but carry a
725 <literal>refid</literal> attribute whose value is the same as the
726 <literal>id</literal> of a filter previously defined in the
727 <filters> section. Alternatively, a route within a filter
728 may omit the <literal>refid</literal> attribute, but contain
729 configuration elements similar to those used for filters defined
730 in the <filters> section. (In other words, each filter in a
731 route may be included either by reference or by physical
737 <section id="example.configuration">
738 <title>An example configuration</title>
740 The following is a small, but complete, Metaproxy configuration
741 file (included in the distribution as
742 <literal>metaproxy/etc/config0.xml</literal>).
743 This file defines a very simple configuration that simply proxies
744 to whatever backend server the client requests, but logs each
745 request and response. This can be useful for debugging complex
746 client-server dialogues.
749 <?xml version="1.0"?>
750 <yp2 xmlns="http://indexdata.dk/yp2/config/1">
751 <start route="start"/>
753 <filter id="frontend" type="frontend_net">
756 <filter id="backend" type="z3950_client">
761 <filter refid="frontend"/>
763 <filter refid="backend"/>
769 It works by defining a single route, called
770 <literal>start</literal>, which consists of a sequence of three
771 filters. The first and last of these are included by reference:
772 their <literal><filter></literal> elements have
773 <literal>refid</literal> attributes that refer to filters defined
774 within the prior <literal><filters></literal> section. The
775 middle filter is included inline in the route.
778 The three filters in the route are as follows: first, a
779 <literal>frontend_net</literal> filter accepts Z39.50 requests
780 from any host on port 9000; then these requests are passed through
781 a <literal>log</literal> filter that emits a message for each
782 request; they are then fed into a <literal>z3950_client</literal>
783 filter, which forwards the requests to the client-specified
784 backend Z39.509 server. When the response arrives, it is handed
785 back to the <literal>log</literal> filter, which emits another
786 message; and then to the front-end filter, which returns the
787 response to the client.
794 <chapter id="multidb">
795 <title>Virtual databases and multi-database searching</title>
799 <title>Introductory notes</title>
801 <title>Lark's vomit</title>
803 This chapter goes into a level of technical detail that is
804 probably not necessary in order to configure and use Metaproxy.
805 It is provided only for those who like to know how things work.
806 You should feel free to skip on to the next section if this one
807 doesn't seem like fun.
811 Two of Metaproxy's filters are concerned with multiple-database
812 operations. Of these, <literal>virt_db</literal> can work alone
813 to control the routing of searches to one of a number of servers,
814 while <literal>multi</literal> can work with the output of
815 <literal>virt_db</literal> to perform multicast searching, merging
816 the results into a unified result-set. The interaction between
817 these two filters is necessarily complex: it reflecting the real,
818 irreducible complexity of multicast searching in a protocol such
819 as Z39.50 that separates initialisation from searching, and in
820 which the database to be searched is not known at initialisation
824 Hold on tight - this may get a little hairy.
830 <title>Virtual databases with the <literal>virt_db</literal> filter</title>
832 In the general course of things, a Z39.50 Init request may carry
833 with it an otherInfo packet of type <literal>VAL_PROXY</literal>,
834 whose value indicates the address of a Z39.50 server to which the
835 ultimate connection is to be made. (This otherInfo packet is
836 supported by YAZ-based Z39.50 clients and servers, but has not yet
837 been ratified by the Maintenance Agency and so is not widely used
838 in non-Index Data software. We're working on it.)
839 The <literal>VAL_PROXY</literal> packet functions
840 analogously to the absoluteURI-style Request-URI used with the GET
841 method when a web browser asks a proxy to forward its request: see
843 <ulink url="http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html#sec5.1.2"
846 <ulink url="http://www.w3.org/Protocols/rfc2616/rfc2616.html"
847 >the HTTP 1.1 specification</ulink>.
850 The role of the <literal>virt_db</literal> filter is to rewrite
851 this otherInfo packet dependent on the virtual database that the
852 client wants to search. For example, a <literal>virt_db</literal>
853 filter could be set up so that searches in the virtual database
854 ``lc'' are forwarded to the Library of Congress server, and
855 searches in the virtual database ``id'' are forwarded to the toy
856 GILS database that Index Data hosts for testing purposes. A
857 <literal>virt_db</literal> configuration to make this switch would
861 <filter type="virt_db">
863 <database>lc</database>
864 <target>z3950.loc.gov:7090/Voyager</target>
867 <database>id</database>
868 <target>indexdata.dk/gils</target>
870 </filter>]]></screen>
872 When Metaproxy receives a Z39.50 Init request from a client, it
873 doesn't immediately forward that request to the back-end server.
874 Why not? Because it doesn't know <emphasis>which</emphasis>
875 back-end server to forward it to until the client sends a search
876 request that specifies the database that it wants to search in.
877 Instead, it just treasures the Init request up in its heart; and,
878 later, the first time the client does a search on one of the
879 specified virtual databases, a connection is forged to the
880 appropriate server and the Init request is forwarded to it. If,
881 later in the session, the same client searches in a different
882 virtual database, then a connection is forged to the server that
883 hosts it, and the same cached Init request is forwarded there,
887 All of this clever Init-delaying is done by the
888 <literal>frontend_net</literal> filter. The
889 <literal>virt_db</literal> filter knows nothing about it; in
890 fact, because the Init request that is received from the client
891 doesn't get forwarded until a Search reqeust is received, the
892 <literal>virt_db</literal> filter (and the
893 <literal>z3950_client</literal> filter behind it) doesn't even get
894 invoked at Init time. The <emphasis>only</emphasis> thing that a
895 <literal>virt_db</literal> filter ever does is rewrite the
896 <literal>VAL_PROXY</literal> otherInfo in the requests that pass
902 <title>A picture is worth a thousand words (but only five hundred on 64-bit architectures)</title>
906 <imagedata fileref="multi.png" format="PNG"/>
909 <imagedata fileref="multi.eps" format="EPS"/>
918 <chapter id="extensions">
919 <title>Writing extensions for Metaproxy</title>
920 <para>### To be written</para>
926 <chapter id="classes">
927 <title>Classes in the Metaproxy source code</title>
931 <title>Introductory notes</title>
933 <emphasis>Stop! Do not read this!</emphasis>
934 You won't enjoy it at all. You should just skip ahead to
935 <link linkend="refguide">the reference guide</link>,
937 <!-- The remainder of this paragraph is lifted verbatim from
938 Douglas Adams' _Hitch Hiker's Guide to the Galaxy_, chapter 8 -->
939 you things you really need to know, like the fact that the
940 fabulously beautiful planet Bethselamin is now so worried about
941 the cumulative erosion by ten billion visiting tourists a year
942 that any net imbalance between the amount you eat and the amount
943 you excrete whilst on the planet is surgically removed from your
944 bodyweight when you leave: so every time you go to the lavatory it
945 is vitally important to get a receipt.
948 This chapter contains documentation of the Metaproxy source code, and is
949 of interest only to maintainers and developers. If you need to
950 change Metaproxy's behaviour or write a new filter, then you will most
951 likely find this chapter helpful. Otherwise it's a waste of your
952 good time. Seriously: go and watch a film or something.
953 <citetitle>This is Spinal Tap</citetitle> is particularly good.
956 Still here? OK, let's continue.
959 In general, classes seem to be named big-endianly, so that
960 <literal>FactoryFilter</literal> is not a filter that filters
961 factories, but a factory that produces filters; and
962 <literal>FactoryStatic</literal> is a factory for the statically
963 registered filters (as opposed to those that are dynamically
968 <section id="individual.classes">
969 <title>Individual classes</title>
971 The classes making up the Metaproxy application are here listed by
972 class-name, with the names of the source files that define them in
977 <title><literal>mp::FactoryFilter</literal>
978 (<filename>factory_filter.cpp</filename>)</title>
980 A factory class that exists primarily to provide the
981 <literal>create()</literal> method, which takes the name of a
982 filter class as its argument and returns a new filter of that
983 type. To enable this, the factory must first be populated by
984 calling <literal>add_creator()</literal> for static filters (this
985 is done by the <literal>FactoryStatic</literal> class, see below)
986 and <literal>add_creator_dyn()</literal> for filters loaded
992 <title><literal>mp::FactoryStatic</literal>
993 (<filename>factory_static.cpp</filename>)</title>
995 A subclass of <literal>FactoryFilter</literal> which is
996 responsible for registering all the statically defined filter
997 types. It does this by knowing about all those filters'
998 structures, which are listed in its constructor. Merely
999 instantiating this class registers all the static classes. It is
1000 for the benefit of this class that <literal>struct
1001 metaproxy_1_filter_struct</literal> exists, and that all the filter
1002 classes provide a static object of that type.
1007 <title><literal>mp::filter::Base</literal>
1008 (<filename>filter.cpp</filename>)</title>
1010 The virtual base class of all filters. The filter API is, on the
1011 surface at least, extremely simple: two methods.
1012 <literal>configure()</literal> is passed a DOM tree representing
1013 that part of the configuration file that pertains to this filter
1014 instance, and is expected to walk that tree extracting relevant
1015 information. And <literal>process()</literal> processes a
1016 package (see below). That surface simplicitly is a bit
1017 misleading, as <literal>process()</literal> needs to know a lot
1018 about the <literal>Package</literal> class in order to do
1024 <title><literal>mp::filter::AuthSimple</literal>,
1025 <literal>Backend_test</literal>, etc.
1026 (<filename>filter_auth_simple.cpp</filename>,
1027 <filename>filter_backend_test.cpp</filename>, etc.)</title>
1029 Individual filters. Each of these is implemented by a header and
1030 a source file, named <filename>filter_*.hpp</filename> and
1031 <filename>filter_*.cpp</filename> respectively. All the header
1032 files should be pretty much identical, in that they declare the
1033 class, including a private <literal>Rep</literal> class and a
1034 member pointer to it, and the two public methods. The only extra
1035 information in any filter header is additional private types and
1036 members (which should really all be in the <literal>Rep</literal>
1037 anyway) and private methods (which should also remain known only
1038 to the source file, but C++'s brain-damaged design requires this
1039 dirty laundry to be exhibited in public. Thanks, Bjarne!)
1042 The source file for each filter needs to supply:
1047 A definition of the private <literal>Rep</literal> class.
1052 Some boilerplate constructors and destructors.
1057 A <literal>configure()</literal> method that uses the
1058 appropriate XML fragment.
1063 Most important, the <literal>process()</literal> method that
1064 does all the actual work.
1071 <title><literal>mp::Package</literal>
1072 (<filename>package.cpp</filename>)</title>
1074 Represents a package on its way through the series of filters
1075 that make up a route. This is essentially a Z39.50 or SRU APDU
1076 together with information about where it came from, which is
1077 modified as it passes through the various filters.
1082 <title><literal>mp::Pipe</literal>
1083 (<filename>pipe.cpp</filename>)</title>
1085 This class provides a compatibility layer so that we have an IPC
1086 mechanism that works the same under Unix and Windows. It's not
1087 particularly exciting.
1092 <title><literal>mp::RouterChain</literal>
1093 (<filename>router_chain.cpp</filename>)</title>
1100 <title><literal>mp::RouterFleXML</literal>
1101 (<filename>router_flexml.cpp</filename>)</title>
1108 <title><literal>mp::Session</literal>
1109 (<filename>session.cpp</filename>)</title>
1116 <title><literal>mp::ThreadPoolSocketObserver</literal>
1117 (<filename>thread_pool_observer.cpp</filename>)</title>
1124 <title><literal>mp::util</literal>
1125 (<filename>util.cpp</filename>)</title>
1127 A namespace of various small utility functions and classes,
1128 collected together for convenience. Most importantly, includes
1129 the <literal>mp::util::odr</literal> class, a wrapper for YAZ's
1135 <title><literal>mp::xml</literal>
1136 (<filename>xmlutil.cpp</filename>)</title>
1138 A namespace of various XML utility functions and classes,
1139 collected together for convenience.
1145 <section id="other.source.files">
1146 <title>Other Source Files</title>
1148 In addition to the Metaproxy source files that define the classes
1149 described above, there are a few additional files which are
1150 briefly described here:
1154 <term><literal>metaproxy_prog.cpp</literal></term>
1157 The main function of the <command>metaproxy</command> program.
1162 <term><literal>ex_router_flexml.cpp</literal></term>
1165 Identical to <literal>metaproxy_prog.cpp</literal>: it's not clear why.
1170 <term><literal>test_*.cpp</literal></term>
1173 Unit-tests for various modules.
1179 ### Still to be described:
1180 <literal>ex_filter_frontend_net.cpp</literal>,
1181 <literal>filter_dl.cpp</literal>,
1182 <literal>plainfile.cpp</literal>,
1183 <literal>tstdl.cpp</literal>.
1190 <chapter id="refguide">
1191 <title>Reference guide</title>
1193 The material in this chapter is drawn directly from the individual
1194 manual entries. In particular, the Metaproxy invocation section is
1195 available using <command>man metaproxy</command>, and the section
1196 on each individual filter is available using the name of the filter
1197 as the argument to the <command>man</command> command.
1201 <section id="progref">
1202 <title>Metaproxy invocation</title>
1207 <section id="filterref">
1208 <title>Reference guide to Metaproxy filters</title>
1215 <!-- Keep this comment at the end of the file
1220 sgml-minimize-attributes:nil
1221 sgml-always-quote-attributes:t
1224 sgml-parent-document: "main.xml"
1225 sgml-local-catalogs: nil
1226 sgml-namecase-general:t