1 <!-- $Id: frontend.xml,v 1.30 2006-06-13 16:01:51 adam Exp $ -->
2 <chapter id="server"><title>Generic server</title>
3 <sect1><title>Introduction</title>
6 If you aren't into documentation, a good way to learn how the
7 back end interface works is to look at the <filename>backend.h</filename>
8 file. Then, look at the small dummy-server in
9 <filename>ztest/ztest.c</filename>. The <filename>backend.h</filename>
10 file also makes a good reference, once you've chewed your way through
11 the prose of this file.
15 If you have a database system that you would like to make available by
16 means of Z39.50 or SRU, &yaz; basically offers your two options. You
17 can use the APIs provided by the &asn;, &odr;, and &comstack;
19 create and decode PDUs, and exchange them with a client.
20 Using this low-level interface gives you access to all fields and
21 options of the protocol, and you can construct your server as close
22 to your existing database as you like.
23 It is also a fairly involved process, requiring
24 you to set up an event-handling mechanism, protocol state machine,
25 etc. To simplify server implementation, we have implemented a compact
26 and simple, but reasonably full-functioned server-frontend that will
27 handle most of the protocol mechanics, while leaving you to
28 concentrate on your database interface.
33 The backend interface was designed in anticipation of a specific
34 integration task, while still attempting to achieve some degree of
35 generality. We realize fully that there are points where the
36 interface can be improved significantly. If you have specific
37 functions or parameters that you think could be useful, send us a
38 mail (or better, sign on to the mailing list referred to in the
39 top-level README file). We will try to fit good suggestions into future
40 releases, to the extent that it can be done without requiring
41 too many structural changes in existing applications.
47 The &yaz; server does not support XCQL.
52 <sect1 id="server.frontend"><title>The Database Frontend</title>
55 We refer to this software as a generic database frontend. Your
56 database system is the <emphasis>backend database</emphasis>, and the
57 interface between the two is called the <emphasis>backend API</emphasis>.
58 The backend API consists of a small number of function handlers and
59 structure definitions. You are required to provide the
60 <function>main()</function> routine for the server (which can be
61 quite simple), as well as a set of handlers to match each of the
63 The interface functions that you write can use any mechanism you like
64 to communicate with your database system: You might link the whole
65 thing together with your database application and access it by
66 function calls; you might use IPC to talk to a database server
67 somewhere; or you might link with third-party software that handles
68 the communication for you (like a commercial database client library).
69 At any rate, the handlers will perform the tasks of:
87 Scanning the database index (optional - if you wish to implement SCAN).
91 Extended Services (optional).
95 Result-Set Delete (optional).
99 Result-Set Sort (optional).
103 Return Explain for SRU (optional).
109 (more functions will be added in time to support as much of
110 Z39.50-1995 as possible).
114 <sect1 id="server.backend"><title>The Backend API</title>
117 The header file that you need to use the interface are in the
118 <filename>include/yaz</filename> directory. It's called
119 <filename>backend.h</filename>. It will include other files from
120 the <filename>include/yaz</filename> directory, so you'll
121 probably want to use the -I option of your compiler to tell it
122 where to find the files. When you run
123 <literal>make</literal> in the top-level &yaz; directory,
124 everything you need to create your server is to link with the
125 <filename>lib/libyaz.la</filename> library.
129 <sect1 id="server.main"><title>Your main() Routine</title>
132 As mentioned, your <function>main()</function> routine can be quite brief.
133 If you want to initialize global parameters, or read global configuration
134 tables, this is the place to do it. At the end of the routine, you should
139 int statserv_main(int argc, char **argv,
140 bend_initresult *(*bend_init)(bend_initrequest *r),
141 void (*bend_close)(void *handle));
145 The third and fourth arguments are pointers to handlers. Handler
146 <function>bend_init</function> is called whenever the server receives
147 an Initialize Request, so it serves as a Z39.50 session initializer. The
148 <function>bend_close</function> handler is called when the session is
153 <function>statserv_main</function> will establish listening sockets
154 according to the parameters given. When connection requests are received,
155 the event handler will typically <function>fork()</function> and
156 create a sub-process to handle a new connection.
157 Alternatively the server may be setup to create threads for each
159 If you do use global variables and forking, you should be aware, then,
160 that these cannot be shared between associations, unless you explicitly
161 disable forking by command line parameters.
165 The server provides a mechanism for controlling some of its behavior
166 without using command-line options. The function
170 statserv_options_block *statserv_getcontrol(void);
174 will return a pointer to a <literal>struct statserv_options_block</literal>
175 describing the current default settings of the server. The structure
176 contains these elements:
180 <literal>int dynamic</literal></term><listitem><para>
181 A boolean value, which determines whether the server
182 will fork on each incoming request (TRUE), or not (FALSE). Default is
183 TRUE. This flag is only read by UNIX-based servers (WIN32 based servers
185 </para></listitem></varlistentry>
188 <literal>int threads</literal></term><listitem><para>
189 A boolean value, which determines whether the server
190 will create a thread on each incoming request (TRUE), or not (FALSE).
191 Default is FALSE. This flag is only read by UNIX-based servers
192 that offer POSIX Threads support.
193 WIN32-based servers always operate in threaded mode.
194 </para></listitem></varlistentry>
197 <literal>int inetd</literal></term><listitem><para>
198 A boolean value, which determines whether the server
199 will operates under a UNIX INET daemon (inetd). Default is FALSE.
200 </para></listitem></varlistentry>
203 <literal>int loglevel</literal></term><listitem><para>
204 Set this by ORing the constants defined in
205 <filename>include/yaz/yaz-log.h</filename>.
206 </para></listitem></varlistentry>
209 <literal>char logfile[ODR_MAXNAME+1]</literal></term>
210 <listitem><para>File for diagnostic output ("": stderr).
211 </para></listitem></varlistentry>
214 <literal>char apdufile[ODR_MAXNAME+1]</literal></term>
216 Name of file for logging incoming and outgoing APDUs
217 ("": don't log APDUs, "-":
218 <literal>stderr</literal>).
219 </para></listitem></varlistentry>
222 <literal>char default_listen[1024]</literal></term>
223 <listitem><para>Same form as the command-line specification of
224 listener address. "": no default listener address.
225 Default is to listen at "tcp:@:9999". You can only
226 specify one default listener address in this fashion.
227 </para></listitem></varlistentry>
230 <literal>enum oid_proto default_proto;</literal></term>
231 <listitem><para>Either <literal>PROTO_Z3950</literal> or
232 <literal>PROTO_SR</literal>.
233 Default is <literal>PROTO_Z39_50</literal>.
234 </para></listitem></varlistentry>
237 <literal>int idle_timeout;</literal></term>
238 <listitem><para>Maximum session idle-time, in minutes. Zero indicates
239 no (infinite) timeout. Default is 15 minutes.
240 </para></listitem></varlistentry>
243 <literal>int maxrecordsize;</literal></term>
244 <listitem><para>Maximum permissible record (message) size. Default
245 is 1Mb. This amount of memory will only be allocated if a
246 client requests a very large amount of records in one operation
248 Set it to a lower number if you are worried about resource
249 consumption on your host system.
250 </para></listitem></varlistentry>
253 <literal>char configname[ODR_MAXNAME+1]</literal></term>
254 <listitem><para>Passed to the backend when a new connection is received.
255 </para></listitem></varlistentry>
258 <literal>char setuid[ODR_MAXNAME+1]</literal></term>
259 <listitem><para>Set user id to the user specified, after binding
260 the listener addresses.
261 </para></listitem></varlistentry>
264 <literal>void (*bend_start)(struct statserv_options_block *p)</literal>
266 <listitem><para>Pointer to function which is called after the
267 command line options have been parsed - but before the server
269 For forked UNIX servers this handler is called in the mother
270 process; for threaded servers this handler is called in the
272 The default value of this pointer is NULL in which case it
273 isn't invoked by the frontend server.
274 When the server operates as an NT service this handler is called
275 whenever the service is started.
276 </para></listitem></varlistentry>
279 <literal>void (*bend_stop)(struct statserv_options_block *p)</literal>
281 <listitem><para>Pointer to function which is called whenever the server
282 has stopped listening for incoming connections. This function pointer
283 has a default value of NULL in which case it isn't called.
284 When the server operates as an NT service this handler is called
285 whenever the service is stopped.
286 </para></listitem></varlistentry>
289 <literal>void *handle</literal></term>
290 <listitem><para>User defined pointer (default value NULL).
291 This is a per-server handle that can be used to specify "user-data".
292 Do not confuse this with the session-handle as returned by bend_init.
293 </para></listitem></varlistentry>
299 The pointer returned by <literal>statserv_getcontrol</literal> points to
300 a static area. You are allowed to change the contents of the structure,
301 but the changes will not take effect before you call
305 void statserv_setcontrol(statserv_options_block *block);
310 that you should generally update this structure before calling
311 <function>statserv_main()</function>.
316 <sect1 id="server.backendfunctions"><title>The Backend Functions</title>
319 For each service of the protocol, the backend interface declares one or
320 two functions. You are required to provide implementations of the
321 functions representing the services that you wish to implement.
324 <sect2><title>Init</title>
327 bend_initresult (*bend_init)(bend_initrequest *r);
331 This handler is called once for each new connection request, after
332 a new process/thread has been created, and an Initialize Request has
333 been received from the client. The pointer to the
334 <function>bend_init</function> handler is passed in the call to
335 <function>statserv_start</function>.
339 This handler is also called when operating in SRU mode - when
340 a connection has been made (even though SRU does not offer
345 Unlike previous versions of YAZ, the <function>bend_init</function> also
346 serves as a handler that defines the Z39.50 services that the backend
347 wish to support. Pointers to <emphasis>all</emphasis> service handlers,
348 including search - and fetch must be specified here in this handler.
351 The request - and result structures are defined as
355 typedef struct bend_initrequest
357 Z_IdAuthentication *auth;
358 ODR stream; /* encoding stream */
359 ODR print; /* printing stream */
360 Z_ReferenceId *referenceId;/* reference ID */
361 char *peer_name; /* dns host of peer (client) */
363 char *implementation_id;
364 char *implementation_name;
365 char *implementation_version;
366 int (*bend_sort) (void *handle, bend_sort_rr *rr);
367 int (*bend_search) (void *handle, bend_search_rr *rr);
368 int (*bend_fetch) (void *handle, bend_fetch_rr *rr);
369 int (*bend_present) (void *handle, bend_present_rr *rr);
370 int (*bend_esrequest) (void *handle, bend_esrequest_rr *rr);
371 int (*bend_delete)(void *handle, bend_delete_rr *rr);
372 int (*bend_scan)(void *handle, bend_scan_rr *rr);
373 int (*bend_segment)(void *handle, bend_segment_rr *rr);
375 ODR decode; /* decoding stream */
376 /* character set and language negotiation - see include/yaz/z-charneg.h */
377 Z_CharSetandLanguageNegotiation *charneg_request;
378 Z_External *charneg_response;
379 int (*bend_srw_explain)(void *handle, bend_explain_rr *rr);
380 int (*bend_srw_scan)(void *handle, bend_scan_rr *rr);
383 typedef struct bend_initresult
385 int errcode; /* 0==OK */
386 char *errstring; /* system error string or NULL */
387 void *handle; /* private handle to the backend module */
392 In general, the server frontend expects that the
393 <literal>bend_*result</literal> pointer that you return is valid at
394 least until the next call to a <literal>bend_* function</literal>.
395 This applies to all of the functions described herein. The parameter
396 structure passed to you in the call belongs to the server frontend, and
397 you should not make assumptions about its contents after the current
398 function call has completed. In other words, if you want to retain any
399 of the contents of a request structure, you should copy them.
403 The <literal>errcode</literal> should be zero if the initialization of
404 the backend went well. Any other value will be interpreted as an error.
405 The <literal>errstring</literal> isn't used in the current version, but
406 one option would be to stick it in the initResponse as a VisibleString.
407 The <literal>handle</literal> is the most important parameter. It should
408 be set to some value that uniquely identifies the current session to
409 the backend implementation. It is used by the frontend server in any
410 future calls to a backend function.
411 The typical use is to set it to point to a dynamically allocated state
412 structure that is private to your backend module.
416 The <literal>auth</literal> member holds the authentication information
417 part of the Z39.50 Initialize Request. Interpret this if your serves
418 requires authentication.
422 The members <literal>peer_name</literal>,
423 <literal>implementation_id</literal>,
424 <literal>implementation_name</literal> and
425 <literal>implementation_version</literal> holds
426 DNS of client, ID of implementor, name
427 of client (Z39.50) implementation - and version.
431 The <literal>bend_</literal> - members are set to NULL when
432 <function>bend_init</function> is called. Modify the pointers by
433 setting them to point to backend functions.
438 <sect2><title>Search and retrieve</title>
440 <para>We now describe the handlers that are required to support search -
441 and retrieve. You must support two functions - one for search - and one
442 for fetch (retrieval of one record). If desirable you can provide a
443 third handler which is called when a present request is received which
444 allows you to optimize retrieval of multiple-records.
448 int (*bend_search) (void *handle, bend_search_rr *rr);
451 char *setname; /* name to give to this set */
452 int replace_set; /* replace set, if it already exists */
453 int num_bases; /* number of databases in list */
454 char **basenames; /* databases to search */
455 Z_ReferenceId *referenceId;/* reference ID */
456 Z_Query *query; /* query structure */
457 ODR stream; /* encode stream */
458 ODR decode; /* decode stream */
459 ODR print; /* print stream */
461 bend_request request;
462 bend_association association;
464 int hits; /* number of hits */
465 int errcode; /* 0==OK */
466 char *errstring; /* system error string or NULL */
467 Z_OtherInformation *search_info;
472 The <function>bend_search</function> handler is a fairly close
473 approximation of a protocol Z39.50 Search Request - and Response PDUs
474 The <literal>setname</literal> is the resultSetName from the protocol.
475 You are required to establish a mapping between the set name and whatever
476 your backend database likes to use.
477 Similarly, the <literal>replace_set</literal> is a boolean value
478 corresponding to the resultSetIndicator field in the protocol.
479 <literal>num_bases/basenames</literal> is a length of/array of character
480 pointers to the database names provided by the client.
481 The <literal>query</literal> is the full query structure as defined in
482 the protocol ASN.1 specification.
483 It can be either of the possible query types, and it's up to you to
484 determine if you can handle the provided query type.
485 Rather than reproduce the C interface here, we'll refer you to the
486 structure definitions in the file
487 <filename>include/yaz/z-core.h</filename>. If you want to look at the
488 attributeSetId OID of the RPN query, you can either match it against
489 your own internal tables, or you can use the
490 <literal>oid_getentbyoid</literal> function provided by &yaz;.
494 The structure contains a number of hits, and an
495 <literal>errcode/errstring</literal> pair. If an error occurs
496 during the search, or if you're unhappy with the request, you should
497 set the errcode to a value from the BIB-1 diagnostic set. The value
498 will then be returned to the user in a nonsurrogate diagnostic record
499 in the response. The <literal>errstring</literal>, if provided, will
500 go in the addinfo field. Look at the protocol definition for the
501 defined error codes, and the suggested uses of the addinfo field.
505 The <function>bend_search</function> handler is also called when
506 the frontend server receives a SRU SearchRetrieveRequest.
507 For SRU, a CQL query is usually provided by the client.
508 The CQL query is available as part of <literal>Z_Query</literal>
509 structure (note that CQL is now part of Z39.50 via an external).
510 To support CQL in existing implementations that only do Type-1,
511 we refer to the CQL-to-PQF tool described
512 <link linkend="tools.cql.pqf">here</link>.
516 To maintain backwards compatibility, the frontend server
517 of yaz always assume that error codes are BIB-1 diagnostics.
518 For SRU operation, a Bib-1 diagnostic code is mapped to
523 int (*bend_fetch) (void *handle, bend_fetch_rr *rr);
525 typedef struct bend_fetch_rr {
526 char *setname; /* set name */
527 int number; /* record number */
528 Z_ReferenceId *referenceId;/* reference ID */
529 oid_value request_format; /* One of the CLASS_RECSYN members */
530 int *request_format_raw; /* same as above (raw OID) */
531 Z_RecordComposition *comp; /* Formatting instructions */
532 ODR stream; /* encoding stream - memory source if req */
533 ODR print; /* printing stream */
535 char *basename; /* name of database that provided record */
536 int len; /* length of record or -1 if structured */
537 char *record; /* record */
538 int last_in_set; /* is it? */
539 oid_value output_format; /* format */
540 int *output_format_raw; /* used instead of above if not-null */
541 int errcode; /* 0==success */
542 char *errstring; /* system error string or NULL */
543 int surrogate_flag; /* surrogate diagnostic */
544 char *schema; /* string record schema input/output */
549 The frontend server calls the <function>bend_fetch</function> handler
550 when it needs database records to fulfill a Z39.50 Search Request, a
551 Z39.50 Present Request or a SRU SearchRetrieveRequest.
552 The <literal>setname</literal> is simply the name of the result set
553 that holds the reference to the desired record.
554 The <literal>number</literal> is the offset into the set (with 1
555 being the first record in the set). The <literal>format</literal> field
556 is the record format requested by the client (See
557 <xref linkend="asn.oid"/>).
558 The value <literal>VAL_NONE</literal> indicates that the client did
559 not request a specific format. The <literal>stream</literal> argument
560 is an &odr; stream which should be used for
561 allocating space for structured data records.
562 The stream will be reset when all records have been assembled, and
563 the response package has been transmitted.
564 For unstructured data, the backend is responsible for maintaining a
565 static or dynamic buffer for the record between calls.
569 If a SRU SearchRetrieveRequest is received by the frontend server,
570 the <literal>referenceId</literal> is NULL and the
571 <literal>request_format</literal> (transfer syntax) is XML (OID name
572 <literal>VAL_TEXT_XML</literal>).
573 The schema for SRU is stored in both the
574 <literal>Z_RecordComposition</literal>
575 structure and <literal>schema</literal> (simple string).
579 In the structure, the <literal>basename</literal> is the name of the
580 database that holds the
581 record. <literal>len</literal> is the length of the record returned, in
582 bytes, and <literal>record</literal> is a pointer to the record.
583 <literal>last_in_set</literal> should be nonzero only if the record
584 returned is the last one in the given result set.
585 <literal>errcode</literal> and <literal>errstring</literal>, if
586 given, will be interpreted as a global error pertaining to the
587 set, and will be returned in a non-surrogate-diagnostic.
588 If you wish to return the error as a surrogate-diagnostic
589 (local error) you can do this by setting
590 <literal>surrogate_flag</literal> to 1 also.
594 If the <literal>len</literal> field has the value -1, then
595 <literal>record</literal> is assumed to point to a constructed data
596 type. The <literal>format</literal> field will be used to determine
597 which encoder should be used to serialize the data.
602 If your backend generates structured records, it should use
603 <function>odr_malloc()</function> on the provided stream for allocating
604 data: This allows the frontend server to keep track of the record sizes.
609 The <literal>format</literal> field is mapped to an object identifier
610 in the direct reference of the resulting EXTERNAL representation
616 The current version of &yaz; only supports the direct reference mode.
621 int (*bend_present) (void *handle, bend_present_rr *rr);
624 char *setname; /* set name */
626 int number; /* record number */
627 oid_value format; /* One of the CLASS_RECSYN members */
628 Z_ReferenceId *referenceId;/* reference ID */
629 Z_RecordComposition *comp; /* Formatting instructions */
630 ODR stream; /* encoding stream */
631 ODR print; /* printing stream */
632 bend_request request;
633 bend_association association;
635 int hits; /* number of hits */
636 int errcode; /* 0==OK */
637 char *errstring; /* system error string or NULL */
642 The <function>bend_present</function> handler is called when
643 the server receives a Z39.50 Present Request.
644 The <literal>setname</literal>,
645 <literal>start</literal> and <literal>number</literal> is the
646 name of the result set - start position - and number of records to
647 be retrieved respectively. <literal>format</literal> and
648 <literal>comp</literal> is the preferred transfer syntax and element
649 specifications of the present request.
652 Note that this is handler serves as a supplement for
653 <function>bend_fetch</function> and need not to be defined in order to
654 support search - and retrieve.
659 <sect2><title>Delete</title>
662 For back-ends that supports delete of a result set only one handler
667 int (*bend_delete)(void *handle, bend_delete_rr *rr);
669 typedef struct bend_delete_rr {
673 Z_ReferenceId *referenceId;
674 int delete_status; /* status for the whole operation */
675 int *statuses; /* status each set - indexed as setnames */
683 The delete set function definition is rather primitive, mostly because
684 we have had no practical need for it as of yet. If someone wants
685 to provide a full delete service, we'd be happy to add the
686 extra parameters that are required. Are there clients out there
687 that will actually delete sets they no longer need?
693 <sect2><title>scan</title>
696 For servers that wish to offer the scan service one handler
701 int (*bend_delete)(void *handle, bend_delete_rr *rr);
704 BEND_SCAN_SUCCESS, /* ok */
705 BEND_SCAN_PARTIAL /* not all entries could be found */
708 typedef struct bend_scan_rr {
709 int num_bases; /* number of elements in database list */
710 char **basenames; /* databases to search */
711 oid_value attributeset;
712 Z_ReferenceId *referenceId; /* reference ID */
713 Z_AttributesPlusTerm *term;
714 ODR stream; /* encoding stream - memory source if required */
715 ODR print; /* printing stream */
717 int *step_size; /* step size */
718 int term_position; /* desired index of term in result list/returned */
719 int num_entries; /* number of entries requested/returned */
721 struct scan_entry *entries;
722 bend_scan_status status;
725 char *scanClause; /* CQL scan clause */
729 This backend server handles both Z39.50 scan
730 and SRU scan. In order for a handler to distinguish between SRU (CQL) scan
731 Z39.50 Scan , it must check for a non-NULL value of
732 <literal>scanClause</literal>.
736 if designed today, it would be a choice using a union or similar,
737 but that would break binary compatibility with existing servers.
743 <sect1 id="server.invocation"><title>Application Invocation</title>
746 The finished application has the following
747 invocation syntax (by way of <function>statserv_main()</function>):
760 A listener specification consists of a transport mode followed by a
761 colon (:) followed by a listener address. The transport mode is
762 either <literal>tcp</literal>, <literal>unix:</literal> or
763 <literal>ssl</literal>.
767 For TCP and SSL, an address has the form
771 hostname | IP-number [: portnumber]
775 The port number defaults to 210 (standard Z39.50 port).
779 For UNIX, the address is the filename of socket.
783 For TCP/IP and SSL, the special hostname <literal>@</literal>
784 (at sign) is mapped to the address <literal>INADDR_ANY</literal>,
785 which causes the server to listen on any local interface.
788 <example><title>Running the GFS on Unix</title>
790 Assuming the server application <replaceable>appname</replaceable> is
791 started as root, the following will make it listen on port 210.
792 The server will change identity to <literal>nobody</literal>
793 and write its log to <filename>/var/log/app.log</filename>.
795 <replaceable>appname</replaceable> -l /var/log/app.log -u nobody tcp:@:210
799 The server will accept Z39.50 requests and offer SRU service on port 210.
802 <example><title>Setting up Apache as SRU Frontend</title>
804 If you use <ulink url="&url.apache;">Apache</ulink>
805 as your public web server and want to offer HTTP port 80
806 access to the YAZ server on 210, you can use the
807 <ulink url="&url.apache.directive.proxypass;">
808 <literal>ProxyPass</literal></ulink>
810 If you have virtual host
811 <literal>srw.mydomain</literal> you can use the following directives
812 in Apache's httpd.conf:
815 ErrorLog /home/srw/logs/error_log
816 TransferLog /home/srw/logs/access_log
817 ProxyPass / http://srw.mydomain:210/
822 The above for the Apache 1.3 series.
825 <example><title>Running a server with local access only</title>
827 Servers that is only being accessed from the local host should listen
828 on UNIX file socket rather than a Internet socket. To listen on
829 <filename>/tmp/mysocket</filename> start the server as follows:
831 <replaceable>appname</replaceable> tcp:/tmp/mysocket
836 <sect1 id="server.vhosts"><title>Virtual Hosts</title>
841 <!-- Keep this comment at the end of the file
846 sgml-minimize-attributes:nil
847 sgml-always-quote-attributes:t
850 sgml-parent-document: "yaz.xml"
851 sgml-local-catalogs: nil
852 sgml-namecase-general:t