-<!-- $Id: book.xml,v 1.7 2006-04-19 16:01:41 mike Exp $ -->
+<!-- $Id: book.xml,v 1.13 2006-04-22 13:28:05 adam Exp $ -->
<bookinfo>
<title>Metaproxy - User's Guide and Reference</title>
<author>
</author>
<copyright>
<year>2006</year>
- <holder>Index Data</holder>
+ <holder>Index Data ApS</holder>
</copyright>
<abstract>
<simpara>
<para>
- <ulink url="http://indexdata.dk/metaproxy/">Metaproxy</ulink>
+ <ulink url="http://www.indexdata.com/metaproxy/">Metaproxy</ulink>
is a standalone program that acts as a universal router, proxy and
encapsulated metasearcher for information retrieval protocols such
as Z39.50, and in the future SRU and SRW. To clients, it acts as a
on to zero or more servers, merging the results, transforming
them, and delivering them back to the client. In addition, it
acts as a simple HTTP server; support for further protocols can be
- added in a module fashion, through the creation of new filters.
+ added in a modular fashion, through the creation of new filters.
</para>
<screen>
Anything goes in!
</screen>
<para>
Metaproxy is a more capable alternative to
- <ulink url="http://indexdata.dk/yazproxy/">YAZ Proxy</ulink>,
+ <ulink url="http://www.indexdata.com/yazproxy/">YAZ Proxy</ulink>,
being more powerful, flexible, configurable and extensible. Among
its many advantages over the older, more pedestrian work are
support for multiplexing (encapsulated metasearching), routing by
facilitites the creation of pluggable modules implementing further
functionality.
</para>
+ <para>
+ This manual will briefly describe Metaproxy's licensing situation
+ before giving an overview of its architecture, then discussing the
+ key concept of a filter in some depth and giving an overview of
+ the various filter types, then discussing the configuration file
+ format. After this come several optional chapters which may be
+ freely skipped: a detailed discussion of virtual databases and
+ multi-database searching, some notes on writing extensions
+ (additional filter types) and a high-level description of the
+ source code. Finally comes the reference guide, which contains
+ instructions for invoking the <command>metaproxy</command>
+ program, and detailed information on each type of filter,
+ including examples.
+ </para>
</chapter>
</section>
- <section>
+ <section id="overview.filter.types">
<title>Overview of filter types</title>
<para>
We now briefly consider each of the types of filter supported by
the core Metaproxy binary. This overview is intended to give a
flavour of the available functionality; more detailed information
- about each type of filter is included below in the Module
- Reference.
+ about each type of filter is included below in
+ <link linkend="filterref"
+ >the reference guide to Metaproxy filters</link>.
</para>
<para>
The filters are here named by the string that is used as the
</section>
- <section>
+ <section id="future.directions">
<title>Future directions</title>
<para>
Some other filters that do not yet exist, but which would be
- <chapter id="multidb">
- <title>Virtual databases and multi-database searching</title>
-
-
- <section>
- <title>Introductory notes</title>
- <para>
- Two of Metaproxy's filters are concerned with multiple-database
- operations. Of these, <literal>virt_db</literal> can work alone
- to control the routing of searches to one of a number of servers,
- while <literal>multi</literal> can work with the output of
- <literal>virt_db</literal> to perform multicast searching, merging
- the results into a unified result-set. The interaction between
- these two filters is necessarily complex, reflecting the real
- complexity of multicast searching in a protocol such as Z39.50
- that separates initialisation from searching, with the database to
- search known only during the latter operation.
- </para>
- <para>
- ### Much, much more to say!
- </para>
- </section>
- </chapter>
-
-
-
<chapter id="configuration">
<title>Configuration: the Metaproxy configuration file format</title>
its configuration file can be thought of as a program for that
interpreter. Configuration is by means of a single file, the name
of which is supplied as the sole command-line argument to the
- <command>yp2</command> program.
+ <command>metaproxy</command> program. (See
+ <link linkend="progref">the reference guide</link>
+ below for more information on invoking Metaproxy.)
</para>
<para>
The configuration files are written in XML. (But that's just an
</para>
</section>
- <section>
+ <section id="overview.xml.structure">
<title>Overview of XML structure</title>
<para>
All elements and attributes are in the namespace
<para>
The <start> element is empty, but carries a
<literal>route</literal> attribute, whose value is the name of
- route at which to start running - analogouse to the name of the
+ route at which to start running - analogous to the name of the
start production in a formal grammar.
</para>
<para>
If present, <filters> contains zero or more <filter>
- elements; filters carry a <literal>type</literal> attribute and
- contain various elements that provide suitable configuration for
- filters of that type. The filter-specific elements are described
- below. Filters defined in this part of the file must carry an
+ elements. Each filter carries a <literal>type</literal> attribute
+ which specifies what kind of filter is being defined
+ (<literal>frontend_net</literal>, <literal>log</literal>, etc.)
+ and contain various elements that provide suitable configuration
+ for a filter of its type. The filter-specific elements are
+ described in
+ <link linkend="filterref">the reference guide below</link>.
+ Filters defined in this part of the file must carry an
<literal>id</literal> attribute so that they can be referenced
from elsewhere.
</para>
<filters> section. Alternatively, a route within a filter
may omit the <literal>refid</literal> attribute, but contain
configuration elements similar to those used for filters defined
- in the <filters> section.
+ in the <filters> section. (In other words, each filter in a
+ route may be included either by reference or by physical
+ inclusion.)
</para>
</section>
- <section>
- <title>Filter configuration</title>
+ <section id="example.configuration">
+ <title>An example configuration</title>
<para>
- All <filter> elements have in common that they must carry a
- <literal>type</literal> attribute whose value is one of the
- supported ones, listed in the schema file and discussed below. In
- additional, <filters>s occurring the <filters> section
- must have an <literal>id</literal> attribute, and those occurring
- within a route must have either a <literal>refid</literal>
- attribute referencing a previously defined filter or contain its
- own configuration information.
+ The following is a small, but complete, Metaproxy configuration
+ file (included in the distribution as
+ <literal>metaproxy/etc/config0.xml</literal>).
+ This file defines a very simple configuration that simply proxies
+ to whatever backend server the client requests, but logs each
+ request and response. This can be useful for debugging complex
+ client-server dialogues.
</para>
+ <screen><![CDATA[
+<?xml version="1.0"?>
+<yp2 xmlns="http://indexdata.dk/yp2/config/1">
+ <start route="start"/>
+ <filters>
+ <filter id="frontend" type="frontend_net">
+ <port>@:9000</port>
+ </filter>
+ <filter id="backend" type="z3950_client">
+ </filter>
+ </filters>
+ <routes>
+ <route id="start">
+ <filter refid="frontend"/>
+ <filter type="log"/>
+ <filter refid="backend"/>
+ </route>
+ </routes>
+</yp2>
+]]></screen>
<para>
- In general, each filter recognises different configuration
- elements within its element, as each filter has different
- functionality. These are as follows:
+ It works by defining a single route, called
+ <literal>start</literal>, which consists of a sequence of three
+ filters. The first and last of these are included by reference:
+ their <literal><filter></literal> elements have
+ <literal>refid</literal> attributes that refer to filters defined
+ within the prior <literal><filters></literal> section. The
+ middle filter is included inline in the route.
</para>
+ <para>
+ The three filters in the route are as follows: first, a
+ <literal>frontend_net</literal> filter accepts Z39.50 requests
+ from any host on port 9000; then these requests are passed through
+ a <literal>log</literal> filter that emits a message for each
+ request; they are then fed into a <literal>z3950_client</literal>
+ filter, which forwards the requests to the client-specified
+ backend Z39.509 server. When the response arrives, it is handed
+ back to the <literal>log</literal> filter, which emits another
+ message; and then to the front-end filter, which returns the
+ response to the client.
+ </para>
+ </section>
+ </chapter>
- <section>
- <title><literal>auth_simple</literal></title>
- <screen>
- <filter type="auth_simple">
- <userRegister>../etc/example.simple-auth</userRegister>
- </filter>
- </screen>
- </section>
-
- <section>
- <title><literal>backend_test</literal></title>
- <screen>
- <filter type="backend_test"/>
- </screen>
- </section>
-
- <section>
- <title><literal>frontend_net</literal></title>
- <screen>
- <filter type="frontend_net">
- <threads>10</threads>
- <port>@:9000</port>
- </filter>
- </screen>
- </section>
-
- <section>
- <title><literal>http_file</literal></title>
- <screen>
- <filter type="http_file">
- <mimetypes>/etc/mime.types</mimetypes>
- <area>
- <documentroot>.</documentroot>
- <prefix>/etc</prefix>
- </area>
- </filter>
- </screen>
- </section>
-
- <section>
- <title><literal>log</literal></title>
- <screen>
- <filter type="log">
- <message>B</message>
- </filter>
- </screen>
- </section>
-
- <section>
- <title><literal>multi</literal></title>
- <screen>
- <filter type="multi"/>
- </screen>
- </section>
-
- <section>
- <title><literal>query_rewrite</literal></title>
- <screen>
- <filter type="query_rewrite">
- <xslt>pqf2pqf.xsl</xslt>
- </filter>
- </screen>
- </section>
- <section>
- <title><literal>session_shared</literal></title>
- <screen>
- <filter type="session_shared">
- ### Not yet defined
- </filter>
- </screen>
- </section>
- <section>
- <title><literal>template</literal></title>
- <screen>
- <filter type="template"/>
- </screen>
- </section>
+ <chapter id="multidb">
+ <title>Virtual databases and multi-database searching</title>
- <section>
- <title><literal>virt_db</literal></title>
- <screen>
- <filter type="virt_db">
- <virtual>
- <database>loc</database>
- <target>z3950.loc.gov:7090/voyager</target>
- </virtual>
- <virtual>
- <database>idgils</database>
- <target>indexdata.dk/gils</target>
- </virtual>
- </filter>
- </screen>
- </section>
- <section>
- <title><literal>z3950_client</literal></title>
- <screen>
- <filter type="z3950_client">
- <timeout>30</timeout>
- </filter>
- </screen>
- </section>
+ <section>
+ <title>Introductory notes</title>
+ <warning>
+ <title>Lark's vomit</title>
+ <para>
+ This chapter goes into a level of technical detail that is
+ probably not necessary in order to configure and use Metaproxy.
+ It is provided only for those who like to know how things work.
+ You should feel free to skip on to the next section if this one
+ doesn't seem like fun.
+ </para>
+ </warning>
+ <para>
+ Two of Metaproxy's filters are concerned with multiple-database
+ operations. Of these, <literal>virt_db</literal> can work alone
+ to control the routing of searches to one of a number of servers,
+ while <literal>multi</literal> can work with the output of
+ <literal>virt_db</literal> to perform multicast searching, merging
+ the results into a unified result-set. The interaction between
+ these two filters is necessarily complex: it reflecting the real,
+ irreducible complexity of multicast searching in a protocol such
+ as Z39.50 that separates initialisation from searching, and in
+ which the database to be searched is not known at initialisation
+ time.
+ </para>
+ <para>
+ Hold on tight - this may get a little hairy.
+ </para>
+ <para>
+ In the general course of things, a Z39.50 Init request may carry
+ with it an otherInfo packet of type <literal>VAL_PROXY</literal>,
+ whose value indicates the address of a Z39.50 server to which the
+ ultimate connection is to be made. (This otherInfo packet is
+ supported by YAZ-based Z39.50 clients and servers, but has not yet
+ been ratified by the Maintenance Agency and so is not widely used
+ in non-Index Data software. We're working on it.)
+ The <literal>VAL_PROXY</literal> packet functions
+ analogously to the absoluteURI-style Request-URI used with the GET
+ method when a web browser asks a proxy to forward its request: see
+ the
+ <ulink url="http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html#sec5.1.2"
+ >Request-URI</ulink>
+ section of
+ <ulink url="http://www.w3.org/Protocols/rfc2616/rfc2616.html"
+ >the HTTP 1.1 specification</ulink>.
+ </para>
+ <para>
+ The role of the <literal>virt_db</literal> filter is to rewrite
+ this otherInfo packet dependent on the virtual database that the
+ client wants to search. For example, a <literal>virt_db</literal>
+ filter could be set up so that searches in the virtual database
+ ``lc'' are forwarded to the Library of Congress server, and
+ searches in the virtual database ``id'' are forwarded to the toy
+ GILS database that Index Data hosts for testing purposes. A
+ <literal>virt_db</literal> configuration to make this switch would
+ look like this:
+ </para>
+ <screen><![CDATA[
+ <filter type="virt_db">
+ <virtual>
+ <database>lc</database>
+ <target>z3950.loc.gov:7090/Voyager</target>
+ </virtual>
+ <virtual>
+ <database>id</database>
+ <target>indexdata.dk/gils</target>
+ </virtual>
+ </filter>]]></screen>
+ <para>
+ When Metaproxy receives a Z39.50 Init request from a client, it
+ doesn't immediately forward that request to the back-end server.
+ Why not? Because it doesn't know <emphasis>which</emphasis>
+ back-end server to forward it to until the client sends a search
+ request that specifies the database that it wants to search in.
+ Instead, it just treasures the Init request up in its heart; and,
+ later, the first time the client does a search on one of the
+ specified virtual databases, a connection is forged to the
+ appropriate server and the Init request is forwarded to it. If,
+ later in the session, the same client searches in a different
+ virtual database, then a connection is forged to the server that
+ hosts it, and the same cached Init request is forwarded there,
+ too.
+ </para>
+ <para>
+ All of this clever Init-delaying is done by the
+ <literal>frontend_net</literal> filter. The
+ <literal>virt_db</literal> filter knows nothing about it; in
+ fact, because the Init request that is received from the client
+ doesn't get forwarded until a Search reqeust is received, the
+ <literal>virt_db</literal> filter (and the
+ <literal>z3950_client</literal> filter behind it) doesn't even get
+ invoked at Init time. The <emphasis>only</emphasis> thing that a
+ <literal>virt_db</literal> filter ever does is rewrite the
+ <literal>VAL_PROXY</literal> otherInfo in the requests that pass
+ through it.
+ </para>
</section>
</chapter>
- <chapter id="moduleref">
- <title>Module Reference</title>
- <para>
- The material in this chapter includes the man pages material
- </para>
- &manref;
- </chapter>
-
<chapter id="extensions">
<title>Writing extensions for Metaproxy</title>
- <para>###</para>
+ <para>### To be written</para>
</chapter>
+
+
+
<chapter id="classes">
<title>Classes in the Metaproxy source code</title>
<title>Introductory notes</title>
<para>
<emphasis>Stop! Do not read this!</emphasis>
- You won't enjoy it at all.
+ You won't enjoy it at all. You should just skip ahead to
+ <link linkend="refguide">the reference guide</link>,
+ which tells
+ <!-- The remainder of this paragraph is lifted verbatim from
+ Douglas Adams' _Hitch Hiker's Guide to the Galaxy_, chapter 8 -->
+ you things you really need to know, like the fact that the
+ fabulously beautiful planet Bethselamin is now so worried about
+ the cumulative erosion by ten billion visiting tourists a year
+ that any net imbalance between the amount you eat and the amount
+ you excrete whilst on the planet is surgically removed from your
+ bodyweight when you leave: so every time you go to the lavatory it
+ is vitally important to get a receipt.
</para>
<para>
This chapter contains documentation of the Metaproxy source code, and is
</para>
</section>
- <section>
+ <section id="individual.classes">
<title>Individual classes</title>
<para>
The classes making up the Metaproxy application are here listed by
structures, which are listed in its constructor. Merely
instantiating this class registers all the static classes. It is
for the benefit of this class that <literal>struct
- yp2_filter_struct</literal> exists, and that all the filter
+ metaproxy_1_filter_struct</literal> exists, and that all the filter
classes provide a static object of that type.
</para>
</section>
<title><literal>mp::RouterChain</literal>
(<filename>router_chain.cpp</filename>)</title>
<para>
- ###
+ ### to be written
</para>
</section>
<title><literal>mp::RouterFleXML</literal>
(<filename>router_flexml.cpp</filename>)</title>
<para>
- ###
+ ### to be written
</para>
</section>
<title><literal>mp::Session</literal>
(<filename>session.cpp</filename>)</title>
<para>
- ###
+ ### to be written
</para>
</section>
<title><literal>mp::ThreadPoolSocketObserver</literal>
(<filename>thread_pool_observer.cpp</filename>)</title>
<para>
- ###
+ ### to be written
</para>
</section>
</section>
- <section>
+ <section id="other.source.files">
<title>Other Source Files</title>
<para>
In addition to the Metaproxy source files that define the classes
<term><literal>metaproxy_prog.cpp</literal></term>
<listitem>
<para>
- The main function of the <command>yp2</command> program.
+ The main function of the <command>metaproxy</command> program.
</para>
</listitem>
</varlistentry>
<literal>plainfile.cpp</literal>,
<literal>tstdl.cpp</literal>.
</para>
-
-
- <!-- Epilogue -->
- <para>
- --
- </para>
- <screen>
- <!-- This is just a lame way to get some vertical whitespace at
- the end of the document -->
-
-
-
-
- </screen>
</section>
</chapter>
+
+
+ <chapter id="refguide">
+ <title>Reference guide</title>
+ <para>
+ The material in this chapter is drawn directly from the individual
+ manual entries. In particular, the Metaproxy invocation section is
+ available using <command>man metaproxy</command>, and the section
+ on each individual filter is available using the name of the filter
+ as the argument to the <command>man</command> command.
+ </para>
+
+
+ <section id="progref">
+ <title>Metaproxy invocation</title>
+ &progref;
+ </section>
+
+
+ <section id="filterref">
+ <title>Reference guide to Metaproxy filters</title>
+ &manref;
+ </section>
+ </chapter>
+
+
+
<!-- Keep this comment at the end of the file
Local variables:
mode: sgml
projects / metaproxy-moved-to-github.git / blobdiff