1 <chapter id="introduction">
2 <!-- $Id: introduction.xml,v 1.13 2002-09-20 09:58:04 mike Exp $ -->
3 <title>Introduction</title>
6 <title>Overview</title>
9 <ulink url="http://www.indexdata.dk/zebra/">
11 is a high-performance, general-purpose structured text
12 indexing and retrieval engine. It reads structured records in a
13 variety of input formats (eg. email, XML, MARC) and provides access
14 to them through a powerful combination of boolean search
15 expressions and relevance-ranked free-text queries.
19 Zebra supports large databases (tens of millions of records,
20 tens of gigabytes of data). It allows safe, incremental
21 database updates on live systems. Because Zebra supports
22 the industry-standard information retrieval protocol, Z39.50,
23 you can search Zebra databases using an enormous variety of
24 programs and toolkits, both commercial and free, which understand
25 this protocol. Application libraries are available to allow
26 bespoke clients to be written in Perl, C, C++, Java, Tcl, Visual
27 Basic, Python, PHP and more - see
28 <ulink url="http://zoom.z3950.org/">the ZOOM web site</ulink>
29 for more information on some of these client toolkits.
33 This document is an introduction to the Zebra system. It explains
34 how to compile the software, how to prepare your first database,
35 and how to configure the server to give you the
36 functionality that you need.
40 If you use Zebra, you should visit its
41 <ulink url="http://www.indexdata.dk/zebra/">web site</ulink>,
42 where you can join the
43 <ulink url="http://www.indexdata.dk/mailman/listinfo/zebralist">
46 <email>### zebra-subscribe@mailman.indexdata.dk</email>
52 <title>Features</title>
55 This is an overview of some of Zebra's most important features:
63 Very large databases: files for indexes, etc. can be
64 automatically partitioned over multiple disks.
70 Arbitrarily complex records. The internal data format
71 is an structured format conceptually similar to XML or GRS-1,
72 which allows nested structured data elements and
73 variant forms of data.
79 Robust updating - records can be added and deleted ``on the fly''
80 without rebuilding the index from scratch.
81 Records can be safely updated even while users are accessing
83 The update procedure is tolerant to crashes or hard interrupts
84 during database updating - data can be reconstructed following
91 Configurable to understand many input formats.
92 A system of input filters driven by
93 regular expressions allows you to easily process most ASCII-based
94 data formats. SGML, XML, ISO2709 (MARC), and raw text are also
101 Searching supports a powerful combination of boolean queries as
102 well as relevance-ranking (free-text) queries. Truncation,
103 masking, full regular expression matching and "approximate
104 matching" (eg. spelling mistakes) are all supported.
110 Index-only databases: data can be, and usually is, imported
111 into Zebra's own storage, but Zebra can also refer to
112 external files, building and maintaining indexes of "live"
119 Zebra is written in portable C, so it runs on most Unix-like systems
120 as well as Windows NT. A binary distribution for Windows NT is
130 Z39.50 protocol support:
137 Protocol facilities: Init, Search, Present (retrieval), Delete,
138 Scan (index browsing) and Sort.
144 Piggy-backed presents are honored in the search-request.
150 Named result sets are supported.
156 Easily configured to support different application profiles, with
157 tables for attribute sets, tag sets, and abstract syntaxes.
158 Additional tables control facilities such as element mappings to
159 different schema (eg., GILS-to-USMARC).
165 Complex composition specifications using Espec-1 (partial support).
166 Element sets are defined using the Espec-1 capability,
167 and are specified in configuration files as simple element
168 requests (and, optionally, variant requests).
174 Multiple record syntaxes
175 for data retrieval: GRS-1, SUTRS,
176 XML, ISO2709 (MARC), etc. Records can be mapped between record syntaxes
177 and schemas on the fly.
188 <title>Applications</title>
190 Zebra has been deployed in numerous applications, in both the
191 academic and commercial worlds, in application domains as diverse
192 as bibliographic catalogues, geospatial information, structured
193 vocabulary browsing, government information locators, civic
194 information systems, environmental observations, museum information
198 Notable applications include the following:
202 <title>DADS - the DTV Article Database Service</title>
204 DADS is a huge database of more than ten million records, totalling
205 over ten gigabytes of data. The records are metadata about academic
206 journal articles, primarily scientific; about 10% of these
207 metadata records link to the full text of the articles they
208 describe, a body of about a terabyte of information (although the
209 full text is not indexed.)
212 It allows students and researchers at DTU (Danmarks Tekniske
213 Universitet, the Technical College of Denmark) to find and order
214 articles from multiple databases in a single query. The database
215 contains literature on all engineering subjects. It's available
216 on-line through a web gateway, though currently only to registered
220 More information can be found at
221 <ulink url="http://www.dtv.dk/help/dads/index_e.htm"/>
226 <title>NLI-Z39.50 - a Natural Language Interface for Libraries</title>
228 Fernuniversität Hagen in Germany have developed a natural
229 language interface for access to library databases.
230 <ulink url="http://ki212.fernuni-hagen.de/nli/NLIintro.html"/>
231 In order to evaluate this interface for recall and precision, they
232 chose Zebra as the basis for retrieval effectiveness. The Zebra
233 server contains a copy of the GIRT database, consisting of more
234 than 76000 records in SGML format (bibliographic records from
235 social science), which are mapped to MARC for presentation.
238 (GIRT is the German Indexing and Retrieval Testdatabase. It is a
239 standard German-language test database for intelligent indexing
240 and retrieval systems. See
241 <ulink url="http://www.gesis.org/forschung/informationstechnologie/clef-delos.htm"/>
244 Evaluation will take place as part of the TREC/CLEF campaign 2003
245 <ulink url="http://clef.iei.pi.cnr.it or http://www4.eurospider.ch/CLEF/"/>
248 For more information, contact Johannes Leveling
249 <email>Johannes.Leveling@FernUni-Hagen.De</email>
254 <title>ULS (Union List of Serials)</title>
256 The London School of Economics (### I think)
257 are involved in a projects called ULS to provide a union catalogue
258 for periodicals in 21 member libraries. They do this with an
259 unusual architecture which they call a
260 ``non-distributed virtual union catalogue''.
263 The member libraries send in data files representing their
264 periodicals, including both brief bibliographic data and summary
265 holdings. Then 21 individual Z39.50 targets are created, each
266 using Zebra, and all mounted on the single hardware server.
267 The live service provides a web gateway allowing Z39.50 searching
268 of all 21 targets or a selection of them.
271 More information can be found at
272 <ulink url="http://www.m25lib.ac.uk/ULS/"/>
277 <title>Various web indexes</title>
279 Zebra has been used by a variety of institutions to construct
280 indexes of large web sites, typically in the region of tens of
281 millions of pages. In this role, it functions somewhat similarly
282 to the engine of google or altavista, but for a selected intranet
283 or subset of the whole Web.
286 For example, Liverpool University's web-search facility (see on
288 <ulink url="http://www.liv.ac.uk/"/>
289 and many sub-pages) works by relevance-searching a Zebra database
290 which is populated by the Harvest-NG web-crawling software.
293 For more information, contact John Gilbertson
294 <email>jgilbert@liverpool.ac.uk</email>
300 <title>Future Directions</title>
303 These are some of the plans that we have for the software in the near
304 and far future, ordered approximately as we expect to work on them.
312 Improved support for XML in search and retrieval. Eventually,
313 the goal is for Zebra to pull double duty as a flexible
314 information retrieval engine and high-performance XML
321 Access to search engine through SOAP/RPC API to allow the
322 construction of applications without requiring Z39.50 tools.
328 Finalisation and documentation of Zebra's C programming
329 API, allowing updates, database management and other functions
330 not readily expressed in Z39.50. We will also consider
331 exposing the API through SOAP.
337 Improved free-text searching. We're first and foremost octet jockeys and
338 we're actively looking for organisations or people who'd like
339 to contribute experience in relevance ranking and text
348 Programmers thrive on user feedback. If you are interested in a
349 facility that you don't see mentioned here, or if there's something
350 you think we could do better, please drop us a mail. Better still,
351 implement it and send us the patches.
354 If you think it's all really neat, you're welcome to drop us a line
355 saying that, too. You'll find contact info at the end of this file.
360 <!-- Keep this comment at the end of the file
365 sgml-minimize-attributes:nil
366 sgml-always-quote-attributes:t
369 sgml-parent-document: "zebra.xml"
370 sgml-local-catalogs: nil
371 sgml-namecase-general:t