1 <!-- $Id: frontend.xml,v 1.31 2006-07-31 11:47:04 adam Exp $ -->
2 <chapter id="server"><title>Generic server</title>
3 <sect1><title>Introduction</title>
6 If you aren't into documentation, a good way to learn how the
7 back end interface works is to look at the <filename>backend.h</filename>
8 file. Then, look at the small dummy-server in
9 <filename>ztest/ztest.c</filename>. The <filename>backend.h</filename>
10 file also makes a good reference, once you've chewed your way through
11 the prose of this file.
15 If you have a database system that you would like to make available by
16 means of Z39.50 or SRU, &yaz; basically offers your two options. You
17 can use the APIs provided by the &asn;, &odr;, and &comstack;
19 create and decode PDUs, and exchange them with a client.
20 Using this low-level interface gives you access to all fields and
21 options of the protocol, and you can construct your server as close
22 to your existing database as you like.
23 It is also a fairly involved process, requiring
24 you to set up an event-handling mechanism, protocol state machine,
25 etc. To simplify server implementation, we have implemented a compact
26 and simple, but reasonably full-functioned server-frontend that will
27 handle most of the protocol mechanics, while leaving you to
28 concentrate on your database interface.
33 The backend interface was designed in anticipation of a specific
34 integration task, while still attempting to achieve some degree of
35 generality. We realize fully that there are points where the
36 interface can be improved significantly. If you have specific
37 functions or parameters that you think could be useful, send us a
38 mail (or better, sign on to the mailing list referred to in the
39 top-level README file). We will try to fit good suggestions into future
40 releases, to the extent that it can be done without requiring
41 too many structural changes in existing applications.
47 The &yaz; server does not support XCQL.
52 <sect1 id="server.frontend"><title>The Database Frontend</title>
55 We refer to this software as a generic database frontend. Your
56 database system is the <emphasis>backend database</emphasis>, and the
57 interface between the two is called the <emphasis>backend API</emphasis>.
58 The backend API consists of a small number of function handlers and
59 structure definitions. You are required to provide the
60 <function>main()</function> routine for the server (which can be
61 quite simple), as well as a set of handlers to match each of the
63 The interface functions that you write can use any mechanism you like
64 to communicate with your database system: You might link the whole
65 thing together with your database application and access it by
66 function calls; you might use IPC to talk to a database server
67 somewhere; or you might link with third-party software that handles
68 the communication for you (like a commercial database client library).
69 At any rate, the handlers will perform the tasks of:
87 Scanning the database index (optional - if you wish to implement SCAN).
91 Extended Services (optional).
95 Result-Set Delete (optional).
99 Result-Set Sort (optional).
103 Return Explain for SRU (optional).
109 (more functions will be added in time to support as much of
110 Z39.50-1995 as possible).
114 <sect1 id="server.backend"><title>The Backend API</title>
117 The header file that you need to use the interface are in the
118 <filename>include/yaz</filename> directory. It's called
119 <filename>backend.h</filename>. It will include other files from
120 the <filename>include/yaz</filename> directory, so you'll
121 probably want to use the -I option of your compiler to tell it
122 where to find the files. When you run
123 <literal>make</literal> in the top-level &yaz; directory,
124 everything you need to create your server is to link with the
125 <filename>lib/libyaz.la</filename> library.
129 <sect1 id="server.main"><title>Your main() Routine</title>
132 As mentioned, your <function>main()</function> routine can be quite brief.
133 If you want to initialize global parameters, or read global configuration
134 tables, this is the place to do it. At the end of the routine, you should
139 int statserv_main(int argc, char **argv,
140 bend_initresult *(*bend_init)(bend_initrequest *r),
141 void (*bend_close)(void *handle));
145 The third and fourth arguments are pointers to handlers. Handler
146 <function>bend_init</function> is called whenever the server receives
147 an Initialize Request, so it serves as a Z39.50 session initializer. The
148 <function>bend_close</function> handler is called when the session is
153 <function>statserv_main</function> will establish listening sockets
154 according to the parameters given. When connection requests are received,
155 the event handler will typically <function>fork()</function> and
156 create a sub-process to handle a new connection.
157 Alternatively the server may be setup to create threads for each
159 If you do use global variables and forking, you should be aware, then,
160 that these cannot be shared between associations, unless you explicitly
161 disable forking by command line parameters.
165 The server provides a mechanism for controlling some of its behavior
166 without using command-line options. The function
170 statserv_options_block *statserv_getcontrol(void);
174 will return a pointer to a <literal>struct statserv_options_block</literal>
175 describing the current default settings of the server. The structure
176 contains these elements:
180 <literal>int dynamic</literal></term><listitem><para>
181 A boolean value, which determines whether the server
182 will fork on each incoming request (TRUE), or not (FALSE). Default is
183 TRUE. This flag is only read by UNIX-based servers (WIN32 based servers
185 </para></listitem></varlistentry>
188 <literal>int threads</literal></term><listitem><para>
189 A boolean value, which determines whether the server
190 will create a thread on each incoming request (TRUE), or not (FALSE).
191 Default is FALSE. This flag is only read by UNIX-based servers
192 that offer POSIX Threads support.
193 WIN32-based servers always operate in threaded mode.
194 </para></listitem></varlistentry>
197 <literal>int inetd</literal></term><listitem><para>
198 A boolean value, which determines whether the server
199 will operates under a UNIX INET daemon (inetd). Default is FALSE.
200 </para></listitem></varlistentry>
203 <literal>char logfile[ODR_MAXNAME+1]</literal></term>
204 <listitem><para>File for diagnostic output ("": stderr).
205 </para></listitem></varlistentry>
208 <literal>char apdufile[ODR_MAXNAME+1]</literal></term>
210 Name of file for logging incoming and outgoing APDUs
211 ("": don't log APDUs, "-":
212 <literal>stderr</literal>).
213 </para></listitem></varlistentry>
216 <literal>char default_listen[1024]</literal></term>
217 <listitem><para>Same form as the command-line specification of
218 listener address. "": no default listener address.
219 Default is to listen at "tcp:@:9999". You can only
220 specify one default listener address in this fashion.
221 </para></listitem></varlistentry>
224 <literal>enum oid_proto default_proto;</literal></term>
225 <listitem><para>Either <literal>PROTO_Z3950</literal> or
226 <literal>PROTO_SR</literal>.
227 Default is <literal>PROTO_Z39_50</literal>.
228 </para></listitem></varlistentry>
231 <literal>int idle_timeout;</literal></term>
232 <listitem><para>Maximum session idle-time, in minutes. Zero indicates
233 no (infinite) timeout. Default is 15 minutes.
234 </para></listitem></varlistentry>
237 <literal>int maxrecordsize;</literal></term>
238 <listitem><para>Maximum permissible record (message) size. Default
239 is 1Mb. This amount of memory will only be allocated if a
240 client requests a very large amount of records in one operation
242 Set it to a lower number if you are worried about resource
243 consumption on your host system.
244 </para></listitem></varlistentry>
247 <literal>char configname[ODR_MAXNAME+1]</literal></term>
248 <listitem><para>Passed to the backend when a new connection is received.
249 </para></listitem></varlistentry>
252 <literal>char setuid[ODR_MAXNAME+1]</literal></term>
253 <listitem><para>Set user id to the user specified, after binding
254 the listener addresses.
255 </para></listitem></varlistentry>
258 <literal>void (*bend_start)(struct statserv_options_block *p)</literal>
260 <listitem><para>Pointer to function which is called after the
261 command line options have been parsed - but before the server
263 For forked UNIX servers this handler is called in the mother
264 process; for threaded servers this handler is called in the
266 The default value of this pointer is NULL in which case it
267 isn't invoked by the frontend server.
268 When the server operates as an NT service this handler is called
269 whenever the service is started.
270 </para></listitem></varlistentry>
273 <literal>void (*bend_stop)(struct statserv_options_block *p)</literal>
275 <listitem><para>Pointer to function which is called whenever the server
276 has stopped listening for incoming connections. This function pointer
277 has a default value of NULL in which case it isn't called.
278 When the server operates as an NT service this handler is called
279 whenever the service is stopped.
280 </para></listitem></varlistentry>
283 <literal>void *handle</literal></term>
284 <listitem><para>User defined pointer (default value NULL).
285 This is a per-server handle that can be used to specify "user-data".
286 Do not confuse this with the session-handle as returned by bend_init.
287 </para></listitem></varlistentry>
293 The pointer returned by <literal>statserv_getcontrol</literal> points to
294 a static area. You are allowed to change the contents of the structure,
295 but the changes will not take effect before you call
299 void statserv_setcontrol(statserv_options_block *block);
304 that you should generally update this structure before calling
305 <function>statserv_main()</function>.
310 <sect1 id="server.backendfunctions"><title>The Backend Functions</title>
313 For each service of the protocol, the backend interface declares one or
314 two functions. You are required to provide implementations of the
315 functions representing the services that you wish to implement.
318 <sect2><title>Init</title>
321 bend_initresult (*bend_init)(bend_initrequest *r);
325 This handler is called once for each new connection request, after
326 a new process/thread has been created, and an Initialize Request has
327 been received from the client. The pointer to the
328 <function>bend_init</function> handler is passed in the call to
329 <function>statserv_start</function>.
333 This handler is also called when operating in SRU mode - when
334 a connection has been made (even though SRU does not offer
339 Unlike previous versions of YAZ, the <function>bend_init</function> also
340 serves as a handler that defines the Z39.50 services that the backend
341 wish to support. Pointers to <emphasis>all</emphasis> service handlers,
342 including search - and fetch must be specified here in this handler.
345 The request - and result structures are defined as
349 typedef struct bend_initrequest
351 Z_IdAuthentication *auth;
352 ODR stream; /* encoding stream */
353 ODR print; /* printing stream */
354 Z_ReferenceId *referenceId;/* reference ID */
355 char *peer_name; /* dns host of peer (client) */
357 char *implementation_id;
358 char *implementation_name;
359 char *implementation_version;
360 int (*bend_sort) (void *handle, bend_sort_rr *rr);
361 int (*bend_search) (void *handle, bend_search_rr *rr);
362 int (*bend_fetch) (void *handle, bend_fetch_rr *rr);
363 int (*bend_present) (void *handle, bend_present_rr *rr);
364 int (*bend_esrequest) (void *handle, bend_esrequest_rr *rr);
365 int (*bend_delete)(void *handle, bend_delete_rr *rr);
366 int (*bend_scan)(void *handle, bend_scan_rr *rr);
367 int (*bend_segment)(void *handle, bend_segment_rr *rr);
369 ODR decode; /* decoding stream */
370 /* character set and language negotiation - see include/yaz/z-charneg.h */
371 Z_CharSetandLanguageNegotiation *charneg_request;
372 Z_External *charneg_response;
373 int (*bend_srw_explain)(void *handle, bend_explain_rr *rr);
374 int (*bend_srw_scan)(void *handle, bend_scan_rr *rr);
377 typedef struct bend_initresult
379 int errcode; /* 0==OK */
380 char *errstring; /* system error string or NULL */
381 void *handle; /* private handle to the backend module */
386 In general, the server frontend expects that the
387 <literal>bend_*result</literal> pointer that you return is valid at
388 least until the next call to a <literal>bend_* function</literal>.
389 This applies to all of the functions described herein. The parameter
390 structure passed to you in the call belongs to the server frontend, and
391 you should not make assumptions about its contents after the current
392 function call has completed. In other words, if you want to retain any
393 of the contents of a request structure, you should copy them.
397 The <literal>errcode</literal> should be zero if the initialization of
398 the backend went well. Any other value will be interpreted as an error.
399 The <literal>errstring</literal> isn't used in the current version, but
400 one option would be to stick it in the initResponse as a VisibleString.
401 The <literal>handle</literal> is the most important parameter. It should
402 be set to some value that uniquely identifies the current session to
403 the backend implementation. It is used by the frontend server in any
404 future calls to a backend function.
405 The typical use is to set it to point to a dynamically allocated state
406 structure that is private to your backend module.
410 The <literal>auth</literal> member holds the authentication information
411 part of the Z39.50 Initialize Request. Interpret this if your serves
412 requires authentication.
416 The members <literal>peer_name</literal>,
417 <literal>implementation_id</literal>,
418 <literal>implementation_name</literal> and
419 <literal>implementation_version</literal> holds
420 DNS of client, ID of implementor, name
421 of client (Z39.50) implementation - and version.
425 The <literal>bend_</literal> - members are set to NULL when
426 <function>bend_init</function> is called. Modify the pointers by
427 setting them to point to backend functions.
432 <sect2><title>Search and retrieve</title>
434 <para>We now describe the handlers that are required to support search -
435 and retrieve. You must support two functions - one for search - and one
436 for fetch (retrieval of one record). If desirable you can provide a
437 third handler which is called when a present request is received which
438 allows you to optimize retrieval of multiple-records.
442 int (*bend_search) (void *handle, bend_search_rr *rr);
445 char *setname; /* name to give to this set */
446 int replace_set; /* replace set, if it already exists */
447 int num_bases; /* number of databases in list */
448 char **basenames; /* databases to search */
449 Z_ReferenceId *referenceId;/* reference ID */
450 Z_Query *query; /* query structure */
451 ODR stream; /* encode stream */
452 ODR decode; /* decode stream */
453 ODR print; /* print stream */
455 bend_request request;
456 bend_association association;
458 int hits; /* number of hits */
459 int errcode; /* 0==OK */
460 char *errstring; /* system error string or NULL */
461 Z_OtherInformation *search_info;
466 The <function>bend_search</function> handler is a fairly close
467 approximation of a protocol Z39.50 Search Request - and Response PDUs
468 The <literal>setname</literal> is the resultSetName from the protocol.
469 You are required to establish a mapping between the set name and whatever
470 your backend database likes to use.
471 Similarly, the <literal>replace_set</literal> is a boolean value
472 corresponding to the resultSetIndicator field in the protocol.
473 <literal>num_bases/basenames</literal> is a length of/array of character
474 pointers to the database names provided by the client.
475 The <literal>query</literal> is the full query structure as defined in
476 the protocol ASN.1 specification.
477 It can be either of the possible query types, and it's up to you to
478 determine if you can handle the provided query type.
479 Rather than reproduce the C interface here, we'll refer you to the
480 structure definitions in the file
481 <filename>include/yaz/z-core.h</filename>. If you want to look at the
482 attributeSetId OID of the RPN query, you can either match it against
483 your own internal tables, or you can use the
484 <literal>oid_getentbyoid</literal> function provided by &yaz;.
488 The structure contains a number of hits, and an
489 <literal>errcode/errstring</literal> pair. If an error occurs
490 during the search, or if you're unhappy with the request, you should
491 set the errcode to a value from the BIB-1 diagnostic set. The value
492 will then be returned to the user in a nonsurrogate diagnostic record
493 in the response. The <literal>errstring</literal>, if provided, will
494 go in the addinfo field. Look at the protocol definition for the
495 defined error codes, and the suggested uses of the addinfo field.
499 The <function>bend_search</function> handler is also called when
500 the frontend server receives a SRU SearchRetrieveRequest.
501 For SRU, a CQL query is usually provided by the client.
502 The CQL query is available as part of <literal>Z_Query</literal>
503 structure (note that CQL is now part of Z39.50 via an external).
504 To support CQL in existing implementations that only do Type-1,
505 we refer to the CQL-to-PQF tool described
506 <link linkend="tools.cql.pqf">here</link>.
510 To maintain backwards compatibility, the frontend server
511 of yaz always assume that error codes are BIB-1 diagnostics.
512 For SRU operation, a Bib-1 diagnostic code is mapped to
517 int (*bend_fetch) (void *handle, bend_fetch_rr *rr);
519 typedef struct bend_fetch_rr {
520 char *setname; /* set name */
521 int number; /* record number */
522 Z_ReferenceId *referenceId;/* reference ID */
523 oid_value request_format; /* One of the CLASS_RECSYN members */
524 int *request_format_raw; /* same as above (raw OID) */
525 Z_RecordComposition *comp; /* Formatting instructions */
526 ODR stream; /* encoding stream - memory source if req */
527 ODR print; /* printing stream */
529 char *basename; /* name of database that provided record */
530 int len; /* length of record or -1 if structured */
531 char *record; /* record */
532 int last_in_set; /* is it? */
533 oid_value output_format; /* format */
534 int *output_format_raw; /* used instead of above if not-null */
535 int errcode; /* 0==success */
536 char *errstring; /* system error string or NULL */
537 int surrogate_flag; /* surrogate diagnostic */
538 char *schema; /* string record schema input/output */
543 The frontend server calls the <function>bend_fetch</function> handler
544 when it needs database records to fulfill a Z39.50 Search Request, a
545 Z39.50 Present Request or a SRU SearchRetrieveRequest.
546 The <literal>setname</literal> is simply the name of the result set
547 that holds the reference to the desired record.
548 The <literal>number</literal> is the offset into the set (with 1
549 being the first record in the set). The <literal>format</literal> field
550 is the record format requested by the client (See
551 <xref linkend="asn.oid"/>).
552 The value <literal>VAL_NONE</literal> indicates that the client did
553 not request a specific format. The <literal>stream</literal> argument
554 is an &odr; stream which should be used for
555 allocating space for structured data records.
556 The stream will be reset when all records have been assembled, and
557 the response package has been transmitted.
558 For unstructured data, the backend is responsible for maintaining a
559 static or dynamic buffer for the record between calls.
563 If a SRU SearchRetrieveRequest is received by the frontend server,
564 the <literal>referenceId</literal> is NULL and the
565 <literal>request_format</literal> (transfer syntax) is XML (OID name
566 <literal>VAL_TEXT_XML</literal>).
567 The schema for SRU is stored in both the
568 <literal>Z_RecordComposition</literal>
569 structure and <literal>schema</literal> (simple string).
573 In the structure, the <literal>basename</literal> is the name of the
574 database that holds the
575 record. <literal>len</literal> is the length of the record returned, in
576 bytes, and <literal>record</literal> is a pointer to the record.
577 <literal>last_in_set</literal> should be nonzero only if the record
578 returned is the last one in the given result set.
579 <literal>errcode</literal> and <literal>errstring</literal>, if
580 given, will be interpreted as a global error pertaining to the
581 set, and will be returned in a non-surrogate-diagnostic.
582 If you wish to return the error as a surrogate-diagnostic
583 (local error) you can do this by setting
584 <literal>surrogate_flag</literal> to 1 also.
588 If the <literal>len</literal> field has the value -1, then
589 <literal>record</literal> is assumed to point to a constructed data
590 type. The <literal>format</literal> field will be used to determine
591 which encoder should be used to serialize the data.
596 If your backend generates structured records, it should use
597 <function>odr_malloc()</function> on the provided stream for allocating
598 data: This allows the frontend server to keep track of the record sizes.
603 The <literal>format</literal> field is mapped to an object identifier
604 in the direct reference of the resulting EXTERNAL representation
610 The current version of &yaz; only supports the direct reference mode.
615 int (*bend_present) (void *handle, bend_present_rr *rr);
618 char *setname; /* set name */
620 int number; /* record number */
621 oid_value format; /* One of the CLASS_RECSYN members */
622 Z_ReferenceId *referenceId;/* reference ID */
623 Z_RecordComposition *comp; /* Formatting instructions */
624 ODR stream; /* encoding stream */
625 ODR print; /* printing stream */
626 bend_request request;
627 bend_association association;
629 int hits; /* number of hits */
630 int errcode; /* 0==OK */
631 char *errstring; /* system error string or NULL */
636 The <function>bend_present</function> handler is called when
637 the server receives a Z39.50 Present Request.
638 The <literal>setname</literal>,
639 <literal>start</literal> and <literal>number</literal> is the
640 name of the result set - start position - and number of records to
641 be retrieved respectively. <literal>format</literal> and
642 <literal>comp</literal> is the preferred transfer syntax and element
643 specifications of the present request.
646 Note that this is handler serves as a supplement for
647 <function>bend_fetch</function> and need not to be defined in order to
648 support search - and retrieve.
653 <sect2><title>Delete</title>
656 For back-ends that supports delete of a result set only one handler
661 int (*bend_delete)(void *handle, bend_delete_rr *rr);
663 typedef struct bend_delete_rr {
667 Z_ReferenceId *referenceId;
668 int delete_status; /* status for the whole operation */
669 int *statuses; /* status each set - indexed as setnames */
677 The delete set function definition is rather primitive, mostly because
678 we have had no practical need for it as of yet. If someone wants
679 to provide a full delete service, we'd be happy to add the
680 extra parameters that are required. Are there clients out there
681 that will actually delete sets they no longer need?
687 <sect2><title>scan</title>
690 For servers that wish to offer the scan service one handler
695 int (*bend_delete)(void *handle, bend_delete_rr *rr);
698 BEND_SCAN_SUCCESS, /* ok */
699 BEND_SCAN_PARTIAL /* not all entries could be found */
702 typedef struct bend_scan_rr {
703 int num_bases; /* number of elements in database list */
704 char **basenames; /* databases to search */
705 oid_value attributeset;
706 Z_ReferenceId *referenceId; /* reference ID */
707 Z_AttributesPlusTerm *term;
708 ODR stream; /* encoding stream - memory source if required */
709 ODR print; /* printing stream */
711 int *step_size; /* step size */
712 int term_position; /* desired index of term in result list/returned */
713 int num_entries; /* number of entries requested/returned */
715 struct scan_entry *entries;
716 bend_scan_status status;
719 char *scanClause; /* CQL scan clause */
723 This backend server handles both Z39.50 scan
724 and SRU scan. In order for a handler to distinguish between SRU (CQL) scan
725 Z39.50 Scan , it must check for a non-NULL value of
726 <literal>scanClause</literal>.
730 if designed today, it would be a choice using a union or similar,
731 but that would break binary compatibility with existing servers.
737 <sect1 id="server.invocation"><title>Application Invocation</title>
740 The finished application has the following
741 invocation syntax (by way of <function>statserv_main()</function>):
754 A listener specification consists of a transport mode followed by a
755 colon (:) followed by a listener address. The transport mode is
756 either <literal>tcp</literal>, <literal>unix:</literal> or
757 <literal>ssl</literal>.
761 For TCP and SSL, an address has the form
765 hostname | IP-number [: portnumber]
769 The port number defaults to 210 (standard Z39.50 port).
773 For UNIX, the address is the filename of socket.
777 For TCP/IP and SSL, the special hostname <literal>@</literal>
778 (at sign) is mapped to the address <literal>INADDR_ANY</literal>,
779 which causes the server to listen on any local interface.
782 <example><title>Running the GFS on Unix</title>
784 Assuming the server application <replaceable>appname</replaceable> is
785 started as root, the following will make it listen on port 210.
786 The server will change identity to <literal>nobody</literal>
787 and write its log to <filename>/var/log/app.log</filename>.
789 <replaceable>appname</replaceable> -l /var/log/app.log -u nobody tcp:@:210
793 The server will accept Z39.50 requests and offer SRU service on port 210.
796 <example><title>Setting up Apache as SRU Frontend</title>
798 If you use <ulink url="&url.apache;">Apache</ulink>
799 as your public web server and want to offer HTTP port 80
800 access to the YAZ server on 210, you can use the
801 <ulink url="&url.apache.directive.proxypass;">
802 <literal>ProxyPass</literal></ulink>
804 If you have virtual host
805 <literal>srw.mydomain</literal> you can use the following directives
806 in Apache's httpd.conf:
809 ErrorLog /home/srw/logs/error_log
810 TransferLog /home/srw/logs/access_log
811 ProxyPass / http://srw.mydomain:210/
816 The above for the Apache 1.3 series.
819 <example><title>Running a server with local access only</title>
821 Servers that is only being accessed from the local host should listen
822 on UNIX file socket rather than a Internet socket. To listen on
823 <filename>/tmp/mysocket</filename> start the server as follows:
825 <replaceable>appname</replaceable> tcp:/tmp/mysocket
830 <sect1 id="server.vhosts"><title>Virtual Hosts</title>
835 <!-- Keep this comment at the end of the file
840 sgml-minimize-attributes:nil
841 sgml-always-quote-attributes:t
844 sgml-parent-document: "yaz.xml"
845 sgml-local-catalogs: nil
846 sgml-namecase-general:t