Comment briefly outlining P2P changes.

[metaproxy-moved-to-github.git] / doc / book.xml

diff --git a/doc/book.xml b/doc/book.xml
index 0b861da..be53892 100644 (file)
--- a/doc/book.xml
+++ b/doc/book.xml
@@ -1,4 +1,24 @@
-<!-- $Id: book.xml,v 1.24 2006-04-27 16:16:28 mike Exp $ -->
+<?xml version="1.0" standalone="no"?>
+<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1//EN"
+    "http://www.oasis-open.org/docbook/xml/4.1/docbookx.dtd" 
+[
+     <!ENTITY local SYSTEM "local.ent">
+     <!ENTITY manref SYSTEM "manref.xml">
+     <!ENTITY progref SYSTEM "progref.xml">
+     <!ENTITY % common SYSTEM "common/common.ent">
+     %common;
+     <!-- Next line allows imagedata/@format="PDF" and is taken from 
+         http://lists.oasis-open.org/archives/docbook/200303/msg00163.html
+     -->
+     <!ENTITY % local.notation.class "| PDF">
+     <!-- Next line is necessary for some XML parsers, for reasons I
+          don't understand.  I got this from
+         http://lists.oasis-open.org/archives/docbook/200303/msg00180.html
+     -->
+     <!NOTATION PDF SYSTEM "PDF">
+]>
+<!-- $Id: book.xml,v 1.36 2006-09-07 09:42:53 mike Exp $ -->
+<book id="metaproxy">

  <bookinfo>
   <title>Metaproxy - User's Guide and Reference</title>
   <author>
  <bookinfo>
   <title>Metaproxy - User's Guide and Reference</title>
   <author>

@@ -32,10 +52,10 @@
     using the filter API.
    </simpara>
    <simpara>
     using the filter API.
    </simpara>
    <simpara>

-    The terms under which Metaproxy will be distributed have yet to be
-    established, but it will not necessarily be open source; so users
-    should not at this stage redistribute the code without explicit
-    written permission from the copyright holders, Index Data ApS.
+    Metaproxy is <emphasis>not</emphasis> open-source software, but
+    may be freely downloaded, unpacked, inspected, built and run for
+    evaluation purposes.  Deployment requires a separate, commercial,
+    license.

    </simpara>
    <simpara>
     <inlinemediaobject>
    </simpara>
    <simpara>
     <inlinemediaobject>

@@ -90,7 +110,7 @@
    functionality.
   </para>
   <para>
    functionality.
   </para>
   <para>

-   This manual will briefly describe Metaproxy's licensing situation
+   This manual will describe how to install Metaproxy

    before giving an overview of its architecture, then discussing the
    key concept of a filter in some depth and giving an overview of
    the various filter types, then discussing the configuration file
    before giving an overview of its architecture, then discussing the
    key concept of a filter in some depth and giving an overview of
    the various filter types, then discussing the configuration file

@@ -105,26 +125,60 @@
   </para>
  </chapter>
 
   </para>
  </chapter>
 

-
-
- <chapter id="licence">
-  <title>The Metaproxy Licence</title>
-  <para>
-   <emphasis role="strong">
-    No decision has yet been made on the terms under which
-    Metaproxy will be distributed.
-   </emphasis>
-   It is possible that, unlike
-   other Index Data products, metaproxy may not be released under a
-   free-software licence such as the GNU GPL.  Until a decision is
-   made and a public statement made, then, and unless it has been
-   delivered to you other specific terms, please treat Metaproxy as
-   though it were proprietary software.
-   The code should not be redistributed without explicit
-   written permission from the copyright holders, Index Data ApS.
-  </para>
+ <chapter id="license">
+  <title>The Metaproxy License</title>
+  <orderedlist numeration="arabic">
+   <listitem>
+    <para>
+     You are allowed to download this software for evaluation purposes.
+     You can unpack it, build it, run it, see how it works and how it fits
+     your needs, all at zero cost.
+    </para>
+   </listitem>
+   <listitem>
+    <para>
+     You may NOT deploy the software.  For the purposes of this license,
+     deployment means running it for any purpose other than evaluation,
+     whether or not you or anyone else makes a profit from doing so.  If
+     you wish to deploy the software, you must first contact Index Data and
+     arrange to purchase a DEPLOYMENT LICENCE.  If you are unsure
+     whether or not your proposed use of the software constitutes
+     deployment, email us at <literal>info@indexdata.com</literal>
+     for clarification.
+    </para>
+   </listitem>
+   <listitem>
+    <para>
+     You may modify your copy of the software (fix bugs, add features)
+     if you need to.  We encourage you to send your changes back to us for
+     integration into the master copy, but you are not obliged to do so.  You
+      may NOT pass your changes on to any other party.
+    </para>
+   </listitem>
+   <listitem>
+    <para>
+     There is NO WARRANTY for this software, to the extent permitted by
+     applicable law.  We provide the software ``as is'' without warranty of
+     any kind, either expressed or implied, including, but not limited to, the
+     implied warranties of MERCHANTABILITY and FITNESS FOR A
+     PARTICULAR PURPOSE.  The entire risk as to the quality and
+     performance of the software is with you.  Should the software prove
+     defective, you assume the cost of all necessary servicing, repair or
+     correction.  In no event unless required by applicable law will we be
+     liable to you for damages, arising out of the use of the software,
+     including but not limited to loss of data or data being rendered
+     inaccurate.
+    </para>
+   </listitem>
+   <listitem>
+    <para>
+     All rights to the software are reserved by Index Data except where
+     this license explicitly says otherwise.
+    </para>
+   </listitem>
+  </orderedlist>

  </chapter>
  </chapter>

-
+ 

  <chapter id="installation">
   <title>Installation</title>
   <para>
  <chapter id="installation">
   <title>Installation</title>
   <para>

@@ -164,7 +218,7 @@
    for more information.
   </para>
   <para>
    for more information.
   </para>
   <para>

-   We have succesfully used Metaproxy with Boost using the compilers
+   We have succesfully built Metaproxy using the compilers

    <ulink url="&url.gcc;">GCC</ulink> version 4.0 and
    <ulink url="&url.vstudio;">Microsoft Visual Studio</ulink> 2003/2005.
   </para>
    <ulink url="&url.gcc;">GCC</ulink> version 4.0 and
    <ulink url="&url.vstudio;">Microsoft Visual Studio</ulink> 2003/2005.
   </para>

@@ -241,38 +295,82 @@
   </section>
 
   <section id="installation.debian">
   </section>
 
   <section id="installation.debian">

-   <title>Installation on Debian</title>
+   <title>Installation on Debian GNU/Linux</title>

    <para>
    <para>

-    ### To be written
+    All dependencies for Metaproxy are available as 
+    <ulink url="&url.debian;">Debian</ulink>
+    packages for the sarge (stable in 2005) and etch (testing in 2005)
+    distributions.

    </para>
    </para>

-   </section>
+   <para>
+    The procedures for Debian based systems, such as
+    <ulink url="&url.ubuntu;">Ubuntu</ulink> is probably similar
+   </para>
+   <para>
+    There is currently no official Debian package for YAZ++.
+    And the Debian package for YAZ is probably too old.
+    Update the <filename>/etc/apt/sources.list</filename>
+    to include the Index Data repository.
+    See YAZ' <ulink url="&url.yaz.download.debian;">Download Debian</ulink>
+    for more information.
+   </para>
+   <screen>
+    apt-get install libxslt1-dev
+    apt-get install libyazpp-dev
+    apt-get install libboost-dev
+    apt-get install libboost-thread-dev
+    apt-get install libboost-date-time-dev
+    apt-get install libboost-program-options-dev
+    apt-get install libboost-test-dev
+   </screen>
+   <para>
+    With these packages installed, the usual configure + make
+    procedure can be used for Metaproxy as outlined in
+    <xref linkend="installation.unix"/>.
+   </para>
+  </section>

 
   <section id="installation.windows">
    <title>Installation on Windows</title>
    <para>
 
   <section id="installation.windows">
    <title>Installation on Windows</title>
    <para>

-    Compilation of Metaproxy can be done using
-    Microsoft <ulink url="&url.vstudio;">Visual Studio</ulink>.
-    We know Version 2003 works. We expect Version 2005 to
-    work as well.
+    Metaproxy can be compiled with Microsoft
+    <ulink url="&url.vstudio;">Visual Studio</ulink>.
+    Version 2003 (C 7.1) and 2005 (C 8.0) is known to work.

    </para>
    <section id="installation.windows.boost">
     <title>Boost</title>
     <para>
      Get Boost from its <ulink url="&url.boost;">home page</ulink>.
      You also need Boost Jam (an alternative to make).
    </para>
    <section id="installation.windows.boost">
     <title>Boost</title>
     <para>
      Get Boost from its <ulink url="&url.boost;">home page</ulink>.
      You also need Boost Jam (an alternative to make).

-     That's also available from this
-     home page. The files download are called something like:
-     <literal>boost_1_33-1.exe</literal>
+     That's also available from the Boost home page.
+     The files to be downloaded are called something like:
+     <filename>boost_1_33-1.exe</filename>

      and
      and

-     <literal>boost-jam-3.1.12-1-ntx86.zip</literal>.
-     Unpack Boost Jam first. Put <literal>bjam.exe</literal>
+     <filename>boost-jam-3.1.12-1-ntx86.zip</filename>.
+     Unpack Boost Jam first. Put <filename>bjam.exe</filename>

      in your system path. Make a command prompt and ensure
      it can be found automatically. If not check the PATH.
      The Boost .exe is a self-extracting exe with
      complete source for Boost. Compile that source with
      Boost Jam (An alternative to Make).
      The compilation takes a while.
      in your system path. Make a command prompt and ensure
      it can be found automatically. If not check the PATH.
      The Boost .exe is a self-extracting exe with
      complete source for Boost. Compile that source with
      Boost Jam (An alternative to Make).
      The compilation takes a while.

-     By default, the Boost build process puts the resulting
+     For Visual Studio 2003, use
+     <screen>
+      bjam "-sTOOLS=vc-7_1"
+     </screen>
+     Here <literal>vc-7_1</literal> refers to a "Toolset" (compiler system).
+     For Visual Studio 2005, use
+     <screen>
+      bjam "-sTOOLS=vc-8_0"
+     </screen>
+     To install the libraries in a common place, use
+     <screen>
+      bjam "-sTOOLS=vc-7_1" install
+     </screen>
+     (or vc-8_0 for VS 2005).
+    </para>
+    <para>
+     By default, the Boost build process installs the resulting

      libraries + header files in
      <literal>\boost\lib</literal>, <literal>\boost\include</literal>.
     </para>
      libraries + header files in
      <literal>\boost\lib</literal>, <literal>\boost\include</literal>.
     </para>

@@ -328,6 +426,58 @@
      to point to the proper locations of Boost, Libxslt, Libxml2,
      zlib, iconv, yaz and yazpp.
     </para>
      to point to the proper locations of Boost, Libxslt, Libxml2,
      zlib, iconv, yaz and yazpp.
     </para>

+
+    <variablelist>
+     <varlistentry><term><literal>DEBUG</literal></term>
+      <listitem><para>
+       If set to 1, the software is
+       compiled with debugging libraries (code generation is
+       multi-threaded debug DLL).
+       If set to 0, the software is compiled with release libraries
+       (code generation is multi-threaded DLL).
+       </para></listitem>
+     </varlistentry>
+
+     <varlistentry>
+      <term><literal>BOOST</literal></term>
+      <listitem>
+       <para>
+       Boost install location
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>
+      <term><literal>BOOST_VERSION</literal></term>
+      <listitem>
+       <para>
+       Boost version (replace . with _).
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>
+      <term><literal>BOOST_TOOLSET</literal></term>
+      <listitem>
+       <para>
+       Boost toolset.
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>
+      <term><literal>LIBXSLT_DIR</literal>,
+       <literal>LIBXML2_DIR</literal> ..</term>
+      <listitem>
+       <para>
+       Specify the locations of Libxslt, libiconv, libxml2 and
+       libxslt.
+       </para>
+      </listitem>
+     </varlistentry>
+      
+    </variablelist>
+    

     <para>
      After succesful compilation you'll find
      <literal>metaproxy.exe</literal> in the
     <para>
      After succesful compilation you'll find
      <literal>metaproxy.exe</literal> in the

@@ -335,6 +485,7 @@
     </para>
    </section>
 
     </para>
    </section>
 

+

   </section>
  </chapter>
  
   </section>
  </chapter>
  

@@ -404,7 +555,7 @@
       The word ``filter'' is sometimes used rather loosely, in two
       different ways: it may be used to mean a particular
       <emphasis>type</emphasis> of filter, as when we speak of ``the
       The word ``filter'' is sometimes used rather loosely, in two
       different ways: it may be used to mean a particular
       <emphasis>type</emphasis> of filter, as when we speak of ``the

-      auth_simplefilter'' or ``the multi filter''; or it may be used
+      auth_simple filter'' or ``the multi filter''; or it may be used

       to be a specific <emphasis>instance</emphasis> of a filter
       within a Metaproxy configuration.  For example, a single
       configuration will often contain multiple instances of the
       to be a specific <emphasis>instance</emphasis> of a filter
       within a Metaproxy configuration.  For example, a single
       configuration will often contain multiple instances of the

@@ -512,6 +663,29 @@
     The filters are here listed in alphabetical order:
    </para>
    
     The filters are here listed in alphabetical order:
    </para>
    

+<!--
+
+### New filters:
+
+New virt_db-alike that does inteligent peer choice, explain merging,
+adds FD&N to explain.  Keeps init responses (like "virt_db Classic"),
+makes routing choices based on local explain knowledge.  Ref IDDI
+paper.
+
+Filter to convert Explain Classic to ZeeRex.
+
+CQL2PQF (which needs augmented ZeeRex) - MARC for Talis.
+
+SRU2Z39.50 (ditto).
+
+Figure out what additional information we need in:
+       ZeeRex (check against D3.1)
+       Init request (e.g. loop detection)
+       Query package (e.g. number of hops)
+       Query response (e.g. record source)
+
+-->
+

    <section>
     <title><literal>auth_simple</literal>
      (mp::filter::AuthSimple)</title>
    <section>
     <title><literal>auth_simple</literal>
      (mp::filter::AuthSimple)</title>

@@ -1087,6 +1261,27 @@ Z>
     be metasearched in this way: issues of resource usage and
     administrative complexity dictate the practical limits.
    </para>
     be metasearched in this way: issues of resource usage and
     administrative complexity dictate the practical limits.
    </para>

+   <para>
+    What happens when one of the databases doesn't respond?  By default,
+    the entire multi-database search fails, and the appropriate
+    diagnostic is returned to the client.  This is usually appropriate
+    during development, when technicians need maximum information, but
+    can be inconvenient in deployment, when users typically don't want
+    to be bothered with problems of this kind and prefer just to get
+    the records from the databases that are available.  To obtain this
+    latter behaviour add an empty
+    <literal>&lt;hideunavailable&gt;</literal>
+    element inside the
+    <literal>multi</literal> filter:
+   </para>
+   <screen><![CDATA[      <filter type="multi">
+        <hideunavailable/>
+      </filter>]]></screen>
+   <para>
+    Under this regime, an error is reported to the client only if
+    <emphasis>all</emphasis> the databases in a multi-database search
+    are unavailable.
+   </para>

   </section>
 
 
   </section>
 
 

@@ -1124,15 +1319,18 @@ Z>
           >the HTTP 1.1 specification</ulink>.
    </para>
    <para>
           >the HTTP 1.1 specification</ulink>.
    </para>
    <para>

-    The role of the <literal>virt_db</literal> filter is to rewrite
-    this otherInfo packet dependent on the virtual database that the
-    client wants to search.  
+    Within Metaproxy, Search requests that are part of the same
+    session as an Init request that carries a
+    <literal>VAL_PROXY</literal> otherInfo are also annotated with the
+    same information.  The role of the <literal>virt_db</literal>
+    filter is to rewrite this otherInfo packet dependent on the
+    virtual database that the client wants to search.

    </para>
    <para>
     When Metaproxy receives a Z39.50 Init request from a client, it
     doesn't immediately forward that request to the back-end server.
     Why not?  Because it doesn't know <emphasis>which</emphasis>
    </para>
    <para>
     When Metaproxy receives a Z39.50 Init request from a client, it
     doesn't immediately forward that request to the back-end server.
     Why not?  Because it doesn't know <emphasis>which</emphasis>

-    back-end server to forward it to until the client sends a search
+    back-end server to forward it to until the client sends a Search

     request that specifies the database that it wants to search in.
     Instead, it just treasures the Init request up in its heart; and,
     later, the first time the client does a search on one of the
     request that specifies the database that it wants to search in.
     Instead, it just treasures the Init request up in its heart; and,
     later, the first time the client does a search on one of the

@@ -1157,9 +1355,35 @@ Z>
     through it.
    </para>
    <para>
     through it.
    </para>
    <para>

-    ### Describe the use of multiple <literal>VAL_PROXY</literal>
-    otherInfos, added by <literal>virt_db</literal> and used by
-    <literal>multi</literal>.
+    It is possible for a <literal>virt_db</literal> filter to contain
+    multiple
+    <literal>&lt;target&gt;</literal>
+    elements.  What does this mean?  Only that the filter will add
+    multiple <literal>VAL_PROXY</literal> otherInfo packets to the
+    Search requests that pass through it.  That's because the virtual
+    DB filter is dumb, and does exactly what it's told - no more, no
+    less.
+    If a Search request with multiple <literal>VAL_PROXY</literal>
+    otherInfo packets reaches a <literal>z3950_client</literal>
+    filter, this is an error.  That filter doesn't know how to deal
+    with multiple targets, so it will either just pick one and search
+    in it, or (better) fail with an error message.
+   </para>
+   <para>
+    The <literal>multi</literal> filter comes to the rescue!  This is
+    the only filter that knows how to deal with multiple
+    <literal>VAL_PROXY</literal> otherInfo packets, and it does so by
+    making multiple copies of the entire Search request: one for each
+    <literal>VAL_PROXY</literal>.  Each of these new copies is then
+    passed down through the remaining filters in the route.  (The
+    copies are handled in parallel though the
+    spawning of new threads.)  Since the copies each have only one
+    <literal>VAL_PROXY</literal> otherInfo, they can be handled by the
+    <literal>z3950_client</literal> filter, which happily deals with
+    each one individually.  When the results of the individual
+    searches come back up to the <literal>multi</literal> filter, it
+    merges them into a single Search response, which is what
+    eventually makes it back to the client.

    </para>
   </section>
 
    </para>
   </section>
 

@@ -1169,7 +1393,7 @@ Z>
    <simpara>
     <inlinemediaobject>
      <imageobject>
    <simpara>
     <inlinemediaobject>
      <imageobject>

-      <imagedata fileref="multi.pdf" format="PDF"/>
+      <imagedata fileref="multi.pdf" format="PDF" scale="50"/>

      </imageobject>
      <imageobject>
       <imagedata fileref="multi.png" format="PNG"/>
      </imageobject>
      <imageobject>
       <imagedata fileref="multi.png" format="PNG"/>

@@ -1491,8 +1715,7 @@ Z>
    &manref;
   </section>
  </chapter>
    &manref;
   </section>
  </chapter>

-
-
+</book>

 
  <!-- Keep this comment at the end of the file
  Local variables:
 
  <!-- Keep this comment at the end of the file
  Local variables:

@@ -1503,7 +1726,7 @@ Z>
  sgml-always-quote-attributes:t
  sgml-indent-step:1
  sgml-indent-data:t
  sgml-always-quote-attributes:t
  sgml-indent-step:1
  sgml-indent-data:t

- sgml-parent-document: "main.xml"
+ sgml-parent-document: nil

  sgml-local-catalogs: nil
  sgml-namecase-general:t
  End:
  sgml-local-catalogs: nil
  sgml-namecase-general:t
  End: