Use Odr_oid for OIDs. Requires YAZ 3.0.2

[metaproxy-moved-to-github.git] / doc / book.xml

diff --git a/doc/book.xml b/doc/book.xml
index 0654898..427962f 100644 (file)
--- a/doc/book.xml
+++ b/doc/book.xml
@@ -18,19 +18,22 @@
      -->
      <!NOTATION PDF SYSTEM "PDF">
 ]>
      -->
      <!NOTATION PDF SYSTEM "PDF">
 ]>

-<!-- $Id: book.xml,v 1.48 2007-01-11 13:23:09 adam Exp $ -->
+<!-- $Id: book.xml,v 1.58 2007-04-19 06:55:50 adam Exp $ -->

 <book id="metaproxy">
  <bookinfo>
   <title>Metaproxy - User's Guide and Reference</title>
 <book id="metaproxy">
  <bookinfo>
   <title>Metaproxy - User's Guide and Reference</title>

-  <author>
-   <firstname>Adam</firstname><surname>Dickmeiss</surname>
-  </author>
-  <author>
-   <firstname>Marc</firstname><surname>Cromme</surname>
-  </author>
-  <author>
-   <firstname>Mike</firstname><surname>Taylor</surname>
-  </author>
+  <authorgroup>
+   <author>
+    <firstname>Adam</firstname><surname>Dickmeiss</surname>
+   </author>
+   <author>
+    <firstname>Marc</firstname><surname>Cromme</surname>
+   </author>
+   <author>
+    <firstname>Mike</firstname><surname>Taylor</surname>
+   </author>
+  </authorgroup>
+  <releaseinfo>&version;</releaseinfo>

   <copyright>
    <year>2005-2007</year>
    <holder>Index Data ApS</holder>
   <copyright>
    <year>2005-2007</year>
    <holder>Index Data ApS</holder>

@@ -43,12 +46,15 @@
     Metaproxy is a universal router, proxy and encapsulated
     metasearcher for information retrieval protocols.  It accepts,
     processes, interprets and redirects requests from IR clients using
     Metaproxy is a universal router, proxy and encapsulated
     metasearcher for information retrieval protocols.  It accepts,
     processes, interprets and redirects requests from IR clients using

-    standard protocols such as
+    standard protocols such as the binary

     <ulink url="&url.z39.50;">ANSI/NISO Z39.50</ulink>
     <ulink url="&url.z39.50;">ANSI/NISO Z39.50</ulink>

-    (and in the future <ulink url="&url.sru;">SRU</ulink>
-    and <ulink url="&url.srw;">SRW</ulink>), as
+    and  the information search and retireval 
+    web services <ulink url="&url.sru;">SRU</ulink>
+    and <ulink url="&url.srw;">SRW</ulink>, as

     well as functioning as a limited
     <ulink url="&url.http;">HTTP</ulink> server. 
     well as functioning as a limited
     <ulink url="&url.http;">HTTP</ulink> server. 

+   </simpara>
+   <simpara>

     Metaproxy is configured by an XML file which
     specifies how the software should function in terms of routes that
     the request packets can take through the proxy, each step on a
     Metaproxy is configured by an XML file which
     specifies how the software should function in terms of routes that
     the request packets can take through the proxy, each step on a

@@ -337,6 +343,39 @@
    </para>
   </section>
 
    </para>
   </section>
 

+  <section id="installation.rpm">
+   <title>Installation on RPM based Linux Systems</title>
+   <para>
+    All external dependencies for Metaproxy are available as 
+    RPM packages, either from your distribution site, or from the 
+    <ulink url="http://fr.rpmfind.net/">RPMfind</ulink> site.
+   </para>
+   <para>
+    For example, an installation of the requires Boost C++ development 
+    libraries on RedHat Fedora C4 and C5 can be done like this:
+    <screen> 
+    wget ftp://fr.rpmfind.net/wlinux/fedora/core/updates/testing/4/SRPMS/boost-1.33.0-3.fc4.src.rpm
+    sudo rpmbuild --buildroot src/ --rebuild -p fc4/boost-1.33.0-3.fc4.src.rpm
+    sudo rpm -U /usr/src/redhat/RPMS/i386/boost-*rpm
+    </screen>
+   </para>
+   <para>
+    The  <ulink url="&url.yaz;">YAZ</ulink> library is needed to
+    compile &metaproxy;, see there
+    for more information on available RPM packages.
+   </para>
+   <para>
+    There is currently no official RPM package for YAZ++.
+    See the <ulink url="&url.yazplusplus;">YAZ++</ulink> pages 
+    for more information on a Unix tarball install.
+   </para>
+   <para>
+    With these packages installed, the usual configure + make
+    procedure can be used for Metaproxy as outlined in
+    <xref linkend="installation.unix"/>.
+   </para>
+  </section>
+

   <section id="installation.windows">
    <title>Installation on Windows</title>
    <para>
   <section id="installation.windows">
    <title>Installation on Windows</title>
    <para>

@@ -496,6 +535,152 @@
   </section>
  </chapter>
  
   </section>
  </chapter>
  

+<chapter id="yazproxy-comparison">
+ <title>YAZ Proxy Comparison</title>
+ <para>
+  The table below lists facilities either supported by either
+   <ulink url="&url.yazproxy;">YAZ Proxy</ulink> or Metaproxy.
+ </para>
+<table id="yazproxy-comparison-table">
+ <title>Metaproxy / YAZ Proxy comparison</title>
+ <tgroup cols="3">
+  <thead>
+   <row>
+    <entry>Facility</entry>
+    <entry>Metaproxy</entry>
+    <entry>YAZ Proxy</entry>
+   </row>
+  </thead>
+  <tbody>
+   <row>
+    <entry>Z39.50 server</entry>
+    <entry>Using filter <literal>frontend_net</literal></entry>
+    <entry>Supported</entry>
+   </row>
+   <row>
+    <entry>SRU server</entry>
+    <entry>Supported with filter <literal>sru_z3950</literal></entry>
+    <entry>Supported</entry>
+   </row>
+   <row>
+    <entry>Z39.50 client</entry>
+    <entry>Supported with filter <literal>z3950_client</literal></entry>
+    <entry>Supported</entry>
+   </row>
+   <row>
+    <entry>SRU client</entry>
+    <entry>Unsupported</entry>
+    <entry>Unsupported</entry>
+   </row>
+   <row>
+    <entry>Connection reuse</entry>
+    <entry>Supported with filter <literal>session_shared</literal></entry>
+    <entry>Supported</entry>
+   </row>
+   <row>
+    <entry>Connection share</entry>
+    <entry>Supported with filter <literal>session_shared</literal></entry>
+    <entry>Unsupported</entry>
+   </row>
+   <row>
+    <entry>Result set reuse</entry>
+    <entry>Supported with filter <literal>session_shared</literal></entry>
+    <entry>Within one Z39.50 session / HTTP keep-alive</entry>
+   </row>
+   <row>
+    <entry>Record cache</entry>
+    <entry>Unsupported</entry>
+    <entry>Supported for last result set within one Z39.50/HTTP-keep alive session</entry>
+   </row>
+   <row>
+    <entry>Z39.50 Virtual database, i.e. select any Z39.50 target for database</entry>
+    <entry>Supported with filter <literal>virt_db</literal></entry>
+    <entry>Unsupported</entry>
+   </row>
+   <row>
+    <entry>SRU Virtual database, i.e. select any Z39.50 target for path</entry>
+    <entry>Supported with filter <literal>virt_db</literal>, 
+     <literal>sru_z3950</literal></entry>
+    <entry>Supported</entry>
+   </row>
+   <row>
+    <entry>Multi target search</entry>
+    <entry>Supported with filter <literal>multi</literal> (round-robin)</entry>
+    <entry>Unsupported</entry>
+   </row>
+   <row>
+    <entry>Retrieval and search limits</entry>
+    <entry>Unsupported</entry>
+    <entry>Supported</entry>
+   </row>
+   <row>
+    <entry>Bandwidth limits</entry>
+    <entry>Unsupported</entry>
+    <entry>Supported</entry>
+   </row>
+   <row>
+    <entry>Connect limits</entry>
+    <entry>Unsupported</entry>
+    <entry>Supported</entry>
+   </row>
+   <row>
+    <entry>Retrieval sanity check and conversions</entry>
+    <entry>Supported using filter <literal>record_transform</literal></entry>
+    <entry>Supported</entry>
+   </row>
+   <row>
+    <entry>Query check</entry>
+    <entry>
+     Supported in a limited way using <literal>query_rewrite</literal>
+    </entry>
+    <entry>Supported</entry>
+   </row>
+   <row>
+    <entry>Query rewrite</entry>
+    <entry>Supported with <literal>query_rewrite</literal></entry>
+    <entry>Unsupported</entry>
+   </row>
+   <row>
+    <entry>Session invalidate for -1 hits</entry>
+    <entry>Unsupported</entry>
+    <entry>Supported</entry>
+   </row>
+   <row>
+    <entry>Architecture</entry>
+    <entry>Multi-threaded + select for networked modules such as
+      <literal>frontend_net</literal>)</entry>
+    <entry>Single-threaded using select</entry>
+   </row>
+
+   <row>
+    <entry>Extensability</entry>
+    <entry>Most functionality implemented as loadable modules</entry>
+    <entry>Unsupported and experimental</entry>
+   </row>
+
+   <row>
+    <entry><ulink url="&url.usemarcon;">USEMARCON</ulink></entry>
+    <entry>Unsupported</entry>
+    <entry>Supported</entry>
+   </row>
+
+   <row>
+    <entry>Portability</entry>
+    <entry>
+     Requires YAZ, YAZ++ and modern C++ compiler supporting
+     <ulink url="&url.boost;">Boost</ulink>.
+    </entry>
+    <entry>
+     Requires YAZ and YAZ++.
+     STL is not required so pretty much any C++ compiler out there should work.
+    </entry>
+   </row>
+
+  </tbody>
+ </tgroup>
+</table>
+</chapter>
+

  <chapter id="architecture">
   <title>The Metaproxy Architecture</title>
   <para>
  <chapter id="architecture">
   <title>The Metaproxy Architecture</title>
   <para>

@@ -576,7 +761,7 @@
       plugins that provide new filters.  The filter API is small and
       conceptually simple, but there are many details to master.  See
       the section below on
       plugins that provide new filters.  The filter API is small and
       conceptually simple, but there are many details to master.  See
       the section below on

-      <link linkend="extensions">extensions</link>.
+      <link linkend="filters">Filters</link>.

      </para>
     </listitem>
    </varlistentry>
      </para>
     </listitem>
    </varlistentry>

@@ -657,8 +842,7 @@
     the core Metaproxy binary.  This overview is intended to give a
     flavor of the available functionality; more detailed information
     about each type of filter is included below in
     the core Metaproxy binary.  This overview is intended to give a
     flavor of the available functionality; more detailed information
     about each type of filter is included below in

-    <link linkend="filterref"
-         >the reference guide to Metaproxy filters</link>.
+    <xref linkend="reference"/>.

    </para>
    <para>
     The filters are here named by the string that is used as the
    </para>
    <para>
     The filters are here named by the string that is used as the

@@ -736,8 +920,8 @@ Figure out what additional information we need in:
      sets Z39.50 packages to Z_Close, and HTTP_Request packages to
      HTTP_Response err code 400 packages, and adds a suitable bounce
      message. 
      sets Z39.50 packages to Z_Close, and HTTP_Request packages to
      HTTP_Response err code 400 packages, and adds a suitable bounce
      message. 

-     The bounce filter is usually added at end of each filter chain
-     config.xml to prevent infinite hanging of for example HTTP
+     The bounce filter is usually added at end of each filter chain route
+     to prevent infinite hanging of for example HTTP

      requests packages when only the Z39.50 client partial sink 
      filter is found in the
      route.  
      requests packages when only the Z39.50 client partial sink 
      filter is found in the
      route.  

@@ -745,6 +929,19 @@ Figure out what additional information we need in:
    </section>
    
    <section>
    </section>
    
    <section>

+    <title><literal>cql_rpn</literal>
+    (mp::filter::CQLtoRPN)</title>
+    <para>
+     A query language transforming filter which catches Z39.50 
+     <literal>searchRequest</literal>
+     packages containing <literal>CQL</literal> queries, transforms
+     those to <literal>RPN</literal> queries,
+     and sends the <literal>searchRequests</literal> on to the next
+     filters. It is among other things useful in a SRU context. 
+    </para>
+   </section>
+   
+   <section>

     <title><literal>frontend_net</literal>
      (mp::filter::FrontendNet)</title>
     <para>
     <title><literal>frontend_net</literal>
      (mp::filter::FrontendNet)</title>
     <para>

@@ -759,7 +956,8 @@ Figure out what additional information we need in:
     <title><literal>http_file</literal>
      (mp::filter::HttpFile)</title>
     <para>
     <title><literal>http_file</literal>
      (mp::filter::HttpFile)</title>
     <para>

-     A partial sink which swallows only HTTP_Request packages, and 
+     A partial sink which swallows only 
+     <literal>HTTP_Request</literal> packages, and 

      returns the contents of files from the local
      filesystem in response to HTTP requests.  
      It lets Z39.50 packages and all other forthcoming package types
      returns the contents of files from the local
      filesystem in response to HTTP requests.  
      It lets Z39.50 packages and all other forthcoming package types

@@ -816,7 +1014,9 @@ Figure out what additional information we need in:
    <title><literal>query_rewrite</literal>
      (mp::filter::QueryRewrite)</title>
     <para>
    <title><literal>query_rewrite</literal>
      (mp::filter::QueryRewrite)</title>
     <para>

-     Rewrites Z39.50 Type-1 and Type-101 (``RPN'') queries by a
+     Rewrites Z39.50 <literal>Type-1</literal> 
+     and <literal>Type-101</literal> (``<literal>RPN</literal>'') 
+     queries by a

      three-step process: the query is transliterated from Z39.50
      packet structures into an XML representation; that XML
      representation is transformed by an XSLT stylesheet; and the
      three-step process: the query is transliterated from Z39.50
      packet structures into an XML representation; that XML
      representation is transformed by an XSLT stylesheet; and the

@@ -844,18 +1044,11 @@ Figure out what additional information we need in:
     <title><literal>session_shared</literal>
      (mp::filter::SessionShared)</title>
     <para>
     <title><literal>session_shared</literal>
      (mp::filter::SessionShared)</title>
     <para>

-     When this is finished, it will implement global sharing of
+     This filter implements global sharing of

      result sets (i.e. between threads and therefore between
      result sets (i.e. between threads and therefore between

-     clients), yielding performance improvements especially when
-     incoming requests are from a stateless environment such as a
-     web-server, in which the client process representing a session
-     might be any one of many.  However:
+     clients), yielding performance improvements by clever resource
+     pooling. 

     </para>
     </para>

-    <warning>
-     <para>
-      This filter is not yet completed.
-     </para>
-    </warning>

    </section>
 
    <section>
    </section>
 
    <section>

@@ -1002,8 +1195,8 @@ Figure out what additional information we need in:
     interpreter.  Configuration is by means of a single XML file, the name
     of which is supplied as the sole command-line argument to the
     <command>metaproxy</command> program.  (See
     interpreter.  Configuration is by means of a single XML file, the name
     of which is supplied as the sole command-line argument to the
     <command>metaproxy</command> program.  (See

-    <link linkend="progref">the reference guide</link>
-    below for more information on invoking Metaproxy.)
+    <xref linkend="reference"/> below for more information on invoking
+    Metaproxy.)

    </para>
   </section>
   
    </para>
   </section>
   

@@ -1039,7 +1232,7 @@ Figure out what additional information we need in:
     and contain various elements that provide suitable configuration
     for a filter of its type.  The filter-specific elements are
     described in
     and contain various elements that provide suitable configuration
     for a filter of its type.  The filter-specific elements are
     described in

-    <link linkend="filterref">the reference guide below</link>.
+    <xref linkend="reference"/>.

     Filters defined in this part of the file must carry an
     <literal>id</literal> attribute so that they can be referenced
     from elsewhere.
     Filters defined in this part of the file must carry an
     <literal>id</literal> attribute so that they can be referenced
     from elsewhere.

@@ -1123,7 +1316,26 @@ Figure out what additional information we need in:
     which returns the response to the client.
    </para>
   </section>
     which returns the response to the client.
    </para>
   </section>

-  <section id="checking.xml.syntax">
+
+  <section id="config-file-modularity">
+   <title>Config file modularity</title>
+   <para>
+    Metaproxy XML configuration snippets can be reused by other
+    filters using the <literal>XInclude</literal> standard, as seen in
+    the <literal>/etc/config-sru-to-z3950.xml</literal> example SRU 
+    configuration.
+   <screen><![CDATA[
+    <filter id="sru" type="sru_z3950">
+      <database name="Default">
+       <xi:include xmlns:xi="http://www.w3.org/2001/XInclude"
+                    href="explain.xml"/>
+      </database>
+    </filter>
+]]></screen>
+    </para>
+  </section>
+
+  <section id="config-file-syntax-check">

    <title>Config file syntax checking</title>
    <para>
     The distribution contains RelaxNG Compact and XML syntax checking
    <title>Config file syntax checking</title>
    <para>
     The distribution contains RelaxNG Compact and XML syntax checking

@@ -1525,12 +1737,123 @@ Z>
  </chapter>
 
 
  </chapter>
 
 

+ <chapter id="sru-server">
+  <title>Combined SRU webservice and Z39.50 server configuration</title>
+  <para>
+   Metaproxy can act as 
+   <ulink url="&url.sru;">SRU</ulink> and 
+   <ulink url="&url.srw;">SRW</ulink> 
+   web service server, which translates web service requests to 
+   <ulink url="&url.z39.50;">ANSI/NISO Z39.50</ulink> packages and
+   sends them off to common available targets.
+  </para>
+  <para>
+  A typical setup for this operation needs a filter route including the
+  following modules: 
+  </para>
+   
+  <table id="sru-server-table-config" frame="top">
+   <title>SRU/Z39.50 Server Filter Route Configuration</title>
+   <tgroup cols="3">
+    <thead>
+     <row>
+      <entry>Filter</entry>
+      <entry>Importance</entry>
+      <entry>Purpose</entry>
+     </row>
+    </thead>
+    
+    <tbody>
+     <row>
+      <entry><literal>frontend_net</literal></entry>
+      <entry>required</entry>
+      <entry>Accepting HTTP connections and passing them to following
+      filters. Since this filter also accepts Z39.50 connections, the
+      server works as SRU and Z39.50 server on the same port.</entry>
+     </row>
+     <row>
+      <entry><literal>sru_z3950</literal></entry>
+      <entry>required</entry>
+      <entry>Accepting SRU GET/POST/SOAP explain and
+       searchRetrieve requests for the the configured databases.
+       Explain requests are directly served from the static XML configuration.
+       SearchRetrieve requests are
+       transformed  to Z39.50 search and present packages.
+       All other HTTP and Z39.50 packages are passed unaltered.</entry>
+     </row>
+     <row>
+      <entry><literal>http_file</literal></entry>
+      <entry>optional</entry>
+      <entry>Serving HTTP requests from the filesystem. This is only
+      needed if the server should serve XSLT stylesheets, static HTML
+      files or Java Script for thin browser based clients.
+       Z39.50 packages are passed unaltered.</entry>
+     </row>
+     <row>
+      <entry><literal>cql_rpn</literal></entry>
+      <entry>required</entry>
+      <entry>Usually, Z39.50 servers do not talk CQL, hence the
+      translation of the CQL query language to RPN is mandatory in
+      most cases. Affects only  Z39.50 search packages.</entry>
+     </row>
+     <row>
+      <entry><literal>record_transform</literal></entry>
+      <entry>optional</entry>
+      <entry>Some Z39.50 backend targets can not present XML record
+      syntaxes in common wanted element sets. using this filter, one
+      can transform binary MARC records to MARCXML records, and
+      further transform those to any needed XML schema/format by XSLT
+      transformations. Changes only  Z39.50 present packages.</entry>
+     </row>
+     <row>
+      <entry><literal>session_shared</literal></entry>
+      <entry>optional</entry>
+      <entry>The stateless nature of web services requires frequent
+      re-searching of the same targets for display of paged result set
+      records. This might be an unacceptable burden for the accessed
+      backend Z39.50 targets, and this mosule can be added for
+      efficient backend target resource pooling.</entry>
+     </row>
+     <row>
+      <entry><literal>z3950_client</literal></entry>
+      <entry>required</entry>
+      <entry>Finally, a Z39.50 package sink is needed in the filter
+      chain to provide the response packages. The Z39.50 client module
+      is used to access external targets over the network, but any
+      coming local Z39.50 package sink could be used instead of.</entry>
+     </row>
+     <row>
+      <entry><literal>bounce</literal></entry>
+      <entry>required</entry>
+      <entry>Any Metaproxy package arriving here did not do so by
+      purpose, and is bounced back with connection closure. this
+      prevents inifinite package hanging inside the SRU server.</entry>
+     </row>
+    </tbody>
+   </tgroup>
+  </table>
+  <para> 
+   A typical minimal example  <ulink url="&url.sru;">SRU</ulink> and 
+   <ulink url="&url.srw;">SRW</ulink> server configuration file is found
+   in the tarball distribution at 
+   <literal>etc/config-sru-to-z3950.xml</literal>.
+  </para> 
+  <para>
+   Off course, any other metaproxy modules can be integrated into a
+   SRU server solution, including, but not limited to, load balancing, 
+   multiple target querying 
+   (see  <xref linkend="multidb"/>), and complex RPN query rewrites. 
+  </para>
+
+
+ </chapter>

 
 

+ <!--

  <chapter id="extensions">
   <title>Writing extensions for Metaproxy</title>
   <para>### To be written</para>
  </chapter>
  <chapter id="extensions">
   <title>Writing extensions for Metaproxy</title>
   <para>### To be written</para>
  </chapter>

-
+ -->

 
 
 
 
 
 

@@ -1543,7 +1866,7 @@ Z>
    <para>
     <emphasis>Stop!  Do not read this!</emphasis>
     You won't enjoy it at all.  You should just skip ahead to
    <para>
     <emphasis>Stop!  Do not read this!</emphasis>
     You won't enjoy it at all.  You should just skip ahead to

-    <link linkend="refguide">the reference guide</link>,
+    <xref linkend="reference"/>,

     which tells
     <!-- The remainder of this paragraph is lifted verbatim from
     Douglas Adams' _Hitch Hiker's Guide to the Galaxy_, chapter 8 -->
     which tells
     <!-- The remainder of this paragraph is lifted verbatim from
     Douglas Adams' _Hitch Hiker's Guide to the Galaxy_, chapter 8 -->

@@ -1792,9 +2115,9 @@ Z>
  </chapter>
  
  
  </chapter>
  
  

- 
- <reference id="refguide">
-  <title>Reference guide</title>
+ <reference id="reference">
+  <title>Reference</title>
+   <partintro>

     <para>
      The material in this chapter is drawn directly from the individual
      manual entries.  In particular, the Metaproxy invocation section is
     <para>
      The material in this chapter is drawn directly from the individual
      manual entries.  In particular, the Metaproxy invocation section is

@@ -1802,7 +2125,8 @@ Z>
      on each individual filter is available using the name of the filter
      as the argument to the <command>man</command> command.
     </para>
      on each individual filter is available using the name of the filter
      as the argument to the <command>man</command> command.
     </para>

-    &manref;
+   </partintro>
+   &manref;

  </reference>
 </book>
 
  </reference>
 </book>