1 <!-- $Id: frontend.xml,v 1.15 2002-09-17 20:17:44 adam Exp $ -->
2 <chapter id="server"><title>Generic server</title>
3 <sect1><title>Introduction</title>
6 If you aren't into documentation, a good way to learn how the
7 back end interface works is to look at the <filename>backend.h</filename>
8 file. Then, look at the small dummy-server in
9 <filename>ztest/ztest.c</filename>. The <filename>backend.h</filename>
10 file also makes a good reference, once you've chewed your way through
11 the prose of this file.
15 If you have a database system that you would like to make available by
16 means of Z39.50, &yaz; basically offers your two options. You
17 can use the APIs provided by the &asn;, &odr;, and &comstack;
19 create and decode PDUs, and exchange them with a client.
20 Using this low-level interface gives you access to all fields and
21 options of the protocol, and you can construct your server as close
22 to your existing database as you like.
23 It is also a fairly involved process, requiring
24 you to set up an event-handling mechanism, protocol state machine,
25 etc. To simplify server implementation, we have implemented a compact
26 and simple, but reasonably full-functioned server-frontend that will
27 handle most of the protocol mechanics, while leaving you to
28 concentrate on your database interface.
33 The backend interface was designed in anticipation of a specific
34 integration task, while still attempting to achieve some degree of
35 generality. We realize fully that there are points where the
36 interface can be improved significantly. If you have specific
37 functions or parameters that you think could be useful, send us a
38 mail (or better, sign on to the mailing list referred to in the
39 top-level README file). We will try to fit good suggestions into future
40 releases, to the extent that it can be done without requiring
41 too many structural changes in existing applications.
46 <sect1 id="server.frontend"><title>The Database Frontend</title>
49 We refer to this software as a generic database frontend. Your
50 database system is the <emphasis>backend database</emphasis>, and the
51 interface between the two is called the <emphasis>backend API</emphasis>.
52 The backend API consists of a small number of function handlers and
53 structure definitions. You are required to provide the
54 <function>main()</function> routine for the server (which can be
55 quite simple), as well as a set of handlers to match each of the
57 The interface functions that you write can use any mechanism you like
58 to communicate with your database system: You might link the whole
59 thing together with your database application and access it by
60 function calls; you might use IPC to talk to a database server
61 somewhere; or you might link with third-party software that handles
62 the communication for you (like a commercial database client library).
63 At any rate, the handlers will perform the tasks of:
81 Scanning the database index (optional - if you wish to implement SCAN).
85 Extended Services (optional).
89 Result-Set Delete (optional).
93 Result-Set Sort (optional).
99 (more functions will be added in time to support as much of
100 Z39.50-1995 as possible).
104 <sect1 id="server.backend"><title>The Backend API</title>
107 The header file that you need to use the interface are in the
108 <filename>include/yaz</filename> directory. It's called
109 <filename>backend.h</filename>. It will include other files from
110 the <filename>include/yaz</filename> directory, so you'll
111 probably want to use the -I option of your compiler to tell it
112 where to find the files. When you run
113 <literal>make</literal> in the top-level &yaz; directory,
114 everything you need to create your server is to link with the
115 <filename>lib/libyaz.la</filename> library.
119 <sect1 id="server.main"><title>Your main() Routine</title>
122 As mentioned, your <function>main()</function> routine can be quite brief.
123 If you want to initialize global parameters, or read global configuration
124 tables, this is the place to do it. At the end of the routine, you should
129 int statserv_main(int argc, char **argv,
130 bend_initresult *(*bend_init)(bend_initrequest *r),
131 void (*bend_close)(void *handle));
135 The third and fourth arguments are pointers to handlers. Handler
136 <function>bend_init</function> is called whenever the server receives
137 an Initialize Request, so it serves as a Z39.50 session initializer. The
138 <function>bend_close</function> handler is called when the session is
143 <function>statserv_main</function> will establish listening sockets
144 according to the parameters given. When connection requests are received,
145 the event handler will typically <function>fork()</function> and
146 create a sub-process to handle a new connection.
147 Alternatively the server may be setup to create threads for each
149 If you do use global variables and forking, you should be aware, then,
150 that these cannot be shared between associations, unless you explicitly
151 disable forking by command line parameters.
155 The server provides a mechanism for controlling some of its behavior
156 without using command-line options. The function
160 statserv_options_block *statserv_getcontrol(void);
164 will return a pointer to a <literal>struct statserv_options_block</literal>
165 describing the current default settings of the server. The structure
166 contains these elements:
170 <literal>int dynamic</literal></term><listitem><para>
171 A boolean value, which determines whether the server
172 will fork on each incoming request (TRUE), or not (FALSE). Default is
173 TRUE. This flag is only read by UNIX-based servers (WIN32 based servers
175 </para></listitem></varlistentry>
178 <literal>int threads</literal></term><listitem><para>
179 A boolean value, which determines whether the server
180 will create a thread on each incoming request (TRUE), or not (FALSE).
181 Default is FALSE. This flag is only read by UNIX-based servers
182 that offer POSIX Threads support.
183 WIN32-based servers always operate in threaded mode.
184 </para></listitem></varlistentry>
187 <literal>int inetd</literal></term><listitem><para>
188 A boolean value, which determines whether the server
189 will operates under a UNIX INET daemon (inetd). Default is FALSE.
190 </para></listitem></varlistentry>
193 <literal>int loglevel</literal></term><listitem><para>
194 Set this by ORing the constants defined in
195 <filename>include/yaz/yaz-log.h</filename>.
196 </para></listitem></varlistentry>
199 <literal>char logfile[ODR_MAXNAME+1]</literal></term>
200 <listitem><para>File for diagnostic output ("": stderr).
201 </para></listitem></varlistentry>
204 <literal>char apdufile[ODR_MAXNAME+1]</literal></term>
206 Name of file for logging incoming and outgoing APDUs
207 ("": don't log APDUs, "-":
208 <literal>stderr</literal>).
209 </para></listitem></varlistentry>
212 <literal>char default_listen[1024]</literal></term>
213 <listitem><para>Same form as the command-line specification of
214 listener address. "": no default listener address.
215 Default is to listen at "tcp:@:9999". You can only
216 specify one default listener address in this fashion.
217 </para></listitem></varlistentry>
220 <literal>enum oid_proto default_proto;</literal></term>
221 <listitem><para>Either <literal>PROTO_Z3950</literal> or
222 <literal>PROTO_SR</literal>.
223 Default is <literal>PROTO_Z39_50</literal>.
224 </para></listitem></varlistentry>
227 <literal>int idle_timeout;</literal></term>
228 <listitem><para>Maximum session idle-time, in minutes. Zero indicates
229 no (infinite) timeout. Default is 120 minutes.
230 </para></listitem></varlistentry>
233 <literal>int maxrecordsize;</literal></term>
234 <listitem><para>Maximum permissible record (message) size. Default
235 is 1Mb. This amount of memory will only be allocated if a
236 client requests a very large amount of records in one operation
238 Set it to a lower number if you are worried about resource
239 consumption on your host system.
240 </para></listitem></varlistentry>
243 <literal>char configname[ODR_MAXNAME+1]</literal></term>
244 <listitem><para>Passed to the backend when a new connection is received.
245 </para></listitem></varlistentry>
248 <literal>char setuid[ODR_MAXNAME+1]</literal></term>
249 <listitem><para>Set user id to the user specified, after binding
250 the listener addresses.
251 </para></listitem></varlistentry>
254 <literal>void (*bend_start)(struct statserv_options_block *p)</literal>
256 <listitem><para>Pointer to function which is called after the
257 command line options have been parsed - but before the server
259 For forked UNIX servers this handler is called in the mother
260 process; for threaded servers this handler is called in the
262 The default value of this pointer is NULL in which case it
263 isn't invoked by the frontend server.
264 When the server operates as an NT service this handler is called
265 whenever the service is started.
266 </para></listitem></varlistentry>
269 <literal>void (*bend_stop)(struct statserv_options_block *p)</literal>
271 <listitem><para>Pointer to function which is called whenever the server
272 has stopped listening for incoming connections. This function pointer
273 has a default value of NULL in which case it isn't called.
274 When the server operates as an NT service this handler is called
275 whenever the service is stopped.
276 </para></listitem></varlistentry>
279 <literal>void *handle</literal></term>
280 <listitem><para>User defined pointer (default value NULL).
281 This is a per-server handle that can be used to specify "user-data".
282 Do not confuse this with the session-handle as returned by bend_init.
283 </para></listitem></varlistentry>
289 The pointer returned by <literal>statserv_getcontrol</literal> points to
290 a static area. You are allowed to change the contents of the structure,
291 but the changes will not take effect before you call
295 void statserv_setcontrol(statserv_options_block *block);
300 that you should generally update this structure before calling
301 <function>statserv_main()</function>.
306 <sect1 id="server.backendfunctions"><title>The Backend Functions</title>
309 For each service of the protocol, the backend interface declares one or
310 two functions. You are required to provide implementations of the
311 functions representing the services that you wish to implement.
314 <sect2><title>Init</title>
317 bend_initresult (*bend_init)(bend_initrequest *r);
321 This handler is called once for each new connection request, after
322 a new process/thread has been created, and an Initialize Request has
323 been received from the client. The pointer to the
324 <function>bend_init</function> handler is passed in the call to
325 <function>statserv_start</function>.
328 Unlike previous versions of YAZ, the <function>bend_init</function> also
329 serves as a handler that defines the Z39.50 services that the backend
330 wish to support. Pointers to <emphasis>all</emphasis> service handlers,
331 including search - and fetch must be specified here in this handler.
334 The request - and result structures are defined as
338 typedef struct bend_initrequest
340 Z_IdAuthentication *auth;
341 ODR stream; /* encoding stream */
342 ODR print; /* printing stream */
343 Z_ReferenceId *referenceId;/* reference ID */
344 char *peer_name; /* dns host of peer (client) */
346 char *implementation_id;
347 char *implementation_name;
348 char *implementation_version;
349 int (*bend_sort) (void *handle, bend_sort_rr *rr);
350 int (*bend_search) (void *handle, bend_search_rr *rr);
351 int (*bend_fetch) (void *handle, bend_fetch_rr *rr);
352 int (*bend_present) (void *handle, bend_present_rr *rr);
353 int (*bend_esrequest) (void *handle, bend_esrequest_rr *rr);
354 int (*bend_delete)(void *handle, bend_delete_rr *rr);
355 int (*bend_scan)(void *handle, bend_scan_rr *rr);
356 int (*bend_segment)(void *handle, bend_segment_rr *rr);
359 typedef struct bend_initresult
361 int errcode; /* 0==OK */
362 char *errstring; /* system error string or NULL */
363 void *handle; /* private handle to the backend module */
368 In general, the server frontend expects that the
369 <literal>bend_*result</literal> pointer that you return is valid at
370 least until the next call to a <literal>bend_* function</literal>.
371 This applies to all of the functions described herein. The parameter
372 structure passed to you in the call belongs to the server frontend, and
373 you should not make assumptions about its contents after the current
374 function call has completed. In other words, if you want to retain any
375 of the contents of a request structure, you should copy them.
379 The <literal>errcode</literal> should be zero if the initialization of
380 the backend went well. Any other value will be interpreted as an error.
381 The <literal>errstring</literal> isn't used in the current version, but
382 one option would be to stick it in the initResponse as a VisibleString.
383 The <literal>handle</literal> is the most important parameter. It should
384 be set to some value that uniquely identifies the current session to
385 the backend implementation. It is used by the frontend server in any
386 future calls to a backend function.
387 The typical use is to set it to point to a dynamically allocated state
388 structure that is private to your backend module.
392 The <literal>auth</literal> member holds the authentication information
393 part of the Z39.50 Initialize Request. Interpret this if your serves
394 requires authentication.
398 The members <literal>peer_name</literal>,
399 <literal>implementation_id</literal>,
400 <literal>implementation_name</literal> and
401 <literal>implementation_version</literal> holds
402 DNS of client, ID of implementor, name
403 of client (Z39.50) implementation - and version.
407 The <literal>bend_</literal> - members are set to NULL when
408 <function>bend_init</function> is called. Modify the pointers by
409 setting them to point to backend functions.
414 <sect2><title>Search and retrieve</title>
416 <para>We now describe the handlers that are required to support search -
417 and retrieve. You must support two functions - one for search - and one
418 for fetch (retrieval of one record). If desirable you can provide a
419 third handler which is called when a present request is received which
420 allows you to optimize retrieval of multiple-records.
424 int (*bend_search) (void *handle, bend_search_rr *rr);
427 char *setname; /* name to give to this set */
428 int replace_set; /* replace set, if it already exists */
429 int num_bases; /* number of databases in list */
430 char **basenames; /* databases to search */
431 Z_ReferenceId *referenceId;/* reference ID */
432 Z_Query *query; /* query structure */
433 ODR stream; /* encode stream */
434 ODR decode; /* decode stream */
435 ODR print; /* print stream */
437 bend_request request;
438 bend_association association;
440 int hits; /* number of hits */
441 int errcode; /* 0==OK */
442 char *errstring; /* system error string or NULL */
448 The <function>bend_search</function> handler is a fairly close
449 approximation of a protocol Search Request - and Response PDUs
450 The <literal>setname</literal> is the resultSetName from the protocol.
451 You are required to establish a mapping between the set name and whatever
452 your backend database likes to use.
453 Similarly, the <literal>replace_set</literal> is a boolean value
454 corresponding to the resultSetIndicator field in the protocol.
455 <literal>num_bases/basenames</literal> is a length of/array of character
456 pointers to the database names provided by the client.
457 The <literal>query</literal> is the full query structure as defined in
458 the protocol ASN.1 specification.
459 It can be either of the possible query types, and it's up to you to
460 determine if you can handle the provided query type.
461 Rather than reproduce the C interface here, we'll refer you to the
462 structure definitions in the file
463 <filename>include/yaz/z-core.h</filename>. If you want to look at the
464 attributeSetId OID of the RPN query, you can either match it against
465 your own internal tables, or you can use the
466 <literal>oid_getentbyoid</literal> function provided by &yaz;.
470 The structure contains a number of hits, and an
471 <literal>errcode/errstring</literal> pair. If an error occurs
472 during the search, or if you're unhappy with the request, you should
473 set the errcode to a value from the BIB-1 diagnostic set. The value
474 will then be returned to the user in a nonsurrogate diagnostic record
475 in the response. The <literal>errstring</literal>, if provided, will
476 go in the addinfo field. Look at the protocol definition for the
477 defined error codes, and the suggested uses of the addinfo field.
482 int (*bend_fetch) (void *handle, bend_fetch_rr *rr);
484 typedef struct bend_fetch_rr {
485 char *setname; /* set name */
486 int number; /* record number */
487 Z_ReferenceId *referenceId;/* reference ID */
488 oid_value request_format; /* One of the CLASS_RECSYN members */
489 int *request_format_raw; /* same as above (raw OID) */
490 Z_RecordComposition *comp; /* Formatting instructions */
491 ODR stream; /* encoding stream - memory source if req */
492 ODR print; /* printing stream */
494 char *basename; /* name of database that provided record */
495 int len; /* length of record or -1 if structured */
496 char *record; /* record */
497 int last_in_set; /* is it? */
498 oid_value output_format; /* format */
499 int *output_format_raw; /* used instead of above if not-null */
500 int errcode; /* 0==success */
501 char *errstring; /* system error string or NULL */
502 int surrogate_flag; /* surrogate diagnostic */
507 The frontend server calls the <function>bend_fetch</function> handler
508 when it needs database records to fulfill a Search Request or a Present
510 The <literal>setname</literal> is simply the name of the result set
511 that holds the reference to the desired record.
512 The <literal>number</literal> is the offset into the set (with 1
513 being the first record in the set). The <literal>format</literal> field
514 is the record format requested by the client (See section
515 <link linkend="oid">Object Identifiers</link>). The value
516 <literal>VAL_NONE</literal> indicates that the client did not
517 request a specific format. The <literal>stream</literal> argument
518 is an &odr; stream which should be used for
519 allocating space for structured data records.
520 The stream will be reset when all records have been assembled, and
521 the response package has been transmitted.
522 For unstructured data, the backend is responsible for maintaining a
523 static or dynamic buffer for the record between calls.
527 In the structure, the <literal>basename</literal> is the name of the
528 database that holds the
529 record. <literal>len</literal> is the length of the record returned, in
530 bytes, and <literal>record</literal> is a pointer to the record.
531 <literal>Last_in_set</literal> should be nonzero only if the record
532 returned is the last one in the given result set.
533 <literal>errcode</literal> and <literal>errstring</literal>, if
534 given, will be interpreted as a global error pertaining to the
535 set, and will be returned in a non-surrogate-diagnostic.
536 If you wish to return the error as a surrogate-diagnostic
537 (local error) you can do this by setting
538 <literal>surrogate_flag</literal> to 1 also.
542 If the <literal>len</literal> field has the value -1, then
543 <literal>record</literal> is assumed to point to a constructed data
544 type. The <literal>format</literal> field will be used to determine
545 which encoder should be used to serialize the data.
550 If your backend generates structured records, it should use
551 <function>odr_malloc()</function> on the provided stream for allocating
552 data: This allows the frontend server to keep track of the record sizes.
557 The <literal>format</literal> field is mapped to an object identifier
558 in the direct reference of the resulting EXTERNAL representation
564 The current version of &yaz; only supports the direct reference mode.
569 int (*bend_present) (void *handle, bend_present_rr *rr);
572 char *setname; /* set name */
574 int number; /* record number */
575 oid_value format; /* One of the CLASS_RECSYN members */
576 Z_ReferenceId *referenceId;/* reference ID */
577 Z_RecordComposition *comp; /* Formatting instructions */
578 ODR stream; /* encoding stream */
579 ODR print; /* printing stream */
580 bend_request request;
581 bend_association association;
583 int hits; /* number of hits */
584 int errcode; /* 0==OK */
585 char *errstring; /* system error string or NULL */
590 The <function>bend_present</function> handler is called when
591 the server receives a Present Request. The <literal>setname</literal>,
592 <literal>start</literal> and <literal>number</literal> is the
593 name of the result set - start position - and number of records to
594 be retrieved respectively. <literal>format</literal> and
595 <literal>comp</literal> is the preferred transfer syntax and element
596 specifications of the present request.
599 Note that this is handler serves as a supplement for
600 <function>bend_fetch</function> and need not to be defined in order to
601 support search - and retrieve.
606 <sect2><title>Delete</title>
609 For back-ends that supports delete of a result set only one handler
614 int (*bend_delete)(void *handle, bend_delete_rr *rr);
616 typedef struct bend_delete_rr {
620 Z_ReferenceId *referenceId;
621 int delete_status; /* status for the whole operation */
622 int *statuses; /* status each set - indexed as setnames */
630 The delete set function definition is rather primitive, mostly because
631 we have had no practical need for it as of yet. If someone wants
632 to provide a full delete service, we'd be happy to add the
633 extra parameters that are required. Are there clients out there
634 that will actually delete sets they no longer need?
640 <sect2><title>scan</title>
643 For servers that wish to offer the scan service one handler
648 int (*bend_delete)(void *handle, bend_delete_rr *rr);
651 BEND_SCAN_SUCCESS, /* ok */
652 BEND_SCAN_PARTIAL /* not all entries could be found */
655 typedef struct bend_scan_rr {
656 int num_bases; /* number of elements in database list */
657 char **basenames; /* databases to search */
658 oid_value attributeset;
659 Z_ReferenceId *referenceId; /* reference ID */
660 Z_AttributesPlusTerm *term;
661 ODR stream; /* encoding stream - memory source if required */
662 ODR print; /* printing stream */
664 int *step_size; /* step size */
665 int term_position; /* desired index of term in result list/returned */
666 int num_entries; /* number of entries requested/returned */
668 struct scan_entry *entries;
669 bend_scan_status status;
677 <sect1 id="server.invocation"><title>Application Invocation</title>
680 The finished application has the following
681 invocation syntax (by way of <function>statserv_main()</function>):
685 <command>appname</command>
686 <arg choice="opt"><option>-a <replaceable>file</replaceable></option></arg>
687 <arg choice="opt"><option>-v <replaceable>level</replaceable></option></arg>
688 <arg choice="opt"><option>-l <replaceable>file</replaceable></option></arg>
689 <arg choice="opt"><option>-u <replaceable>uid</replaceable></option></arg>
690 <arg choice="opt"><option>-c <replaceable>config</replaceable></option></arg>
691 <arg choice="opt"><option>-t <replaceable>minutes</replaceable></option></arg>
693 <arg choice="opt"><option>-k <replaceable>kilobytes</replaceable></option></arg>
694 <arg choice="opt"><option>-d <replaceable>daemon</replaceable></option></arg>
695 <arg choice="opt"><option>-w <replaceable>dir</replaceable></option></arg>
696 <arg choice="opt"><option>-ziST1</option></arg>
697 <arg choice="opt" rep="repeat">listener-spec</arg>
708 A listener specification consists of a transport mode followed by a
709 colon (:) followed by a listener address. The transport mode is
710 either <literal>tcp</literal>, <literal>unix:</literal> or
711 <literal>ssl</literal>.
715 For TCP and SSL, an address has the form
719 hostname | IP-number [: portnumber]
723 The port number defaults to 210 (standard Z39.50 port).
727 For UNIX, the address is the filename of socket.
731 For TCP/IP and SSL, the special hostname "@" is mapped to
732 the address <literal>INADDR_ANY</literal>, which causes the
733 server to listen on any local interface.
750 <!-- Keep this comment at the end of the file
755 sgml-minimize-attributes:nil
756 sgml-always-quote-attributes:t
759 sgml-parent-document: "yaz.xml"
760 sgml-local-catalogs: nil
761 sgml-namecase-general:t