% -*- mode: LaTeX; fill-column: 75; -*- % % $Id: libxds.tex,v 1.18 2001/08/30 15:02:53 simons Exp $ % \documentclass[a4paper,10pt,pointlessnumbers,bibtotoc]{scrartcl} \usepackage[dvips,xdvi]{graphicx} \usepackage{fancyvrb} \typearea[2cm]{12} \fussy \begin{document} \titlehead{Cable \& Wireless Deutschland GmbH\\Application Services\\Development Team} \title{OSSP XDS ---\\Extensible Data Serialization} \author{Peter Simons $<$simons@computer.org$>$} \date{2001-08-01} \maketitle \section{Introduction} In today's networked world, computer systems of all brands and flavours communicate with each other. Unfortunately, these systems are far from being compatible: Many systems use different internal representations for the same thing. Look at the (hexadecimal) number \$1234 for instance: On a big endian machine, this number is be stored in memory the way you'd intuitively expect: \$12~\$34 --- the more significant byte preceeds the less significant one. On a little endian machine, though, the number \$1234 is be stored like this: \$34~\$12 --- exactly the other way round. As a result, you cannot just write the number \$1234 to a socket and expect the other end to understand it correctly, because if the endians differ, the reader will read a different number than the writer sent. Things will get even more complicated when you start exchanging floating point numbers, for which about a dozen different encodings exist! \begin{figure}[tbh] \begin{center} \includegraphics[width=\textwidth]{data-exchange.eps} \caption{Data exchange using XDS} \label{data exchange} \end{center} \end{figure} Solving these problems is the domain of XDS; its purpose is to encode data in a way that allows this data to be exchanged between different computer systems. Assume you'd want to transfer the value \$1234 from host A to host B. Then you would encode it using XDS, transfer the encoded data over the network, and decode the value again at the other end. Every program that follows this process will read the correct value no matter what native representation is used internally. There is a rich variety of applications for such a functionality: XDS may be used to encode data before it is written to disk or read from the disk, it may be used to encode data to be exchanged between processes over the network, etc. Because of this variety, special attention has been paid to the library design. \paragraph{The library has been designed to be extensible.} The functionality is split into a generic encoding and decoding framework and a set of encoding and decoding engines. These engines can be plugged into the framework at run-time to actually do the encoding and decoding of data. Because of this architecture, XDS can be customized to deploy any data format the developer sees fit. Included in the distribution are engines for the XDR format specified in \cite{xdr} and for the XML format specified in \cite{xml}. \paragraph{The library is convenient to use.} An arbitrary number of variables can be encoded or decoded with one single function call. All memory management is done by XDS, the developer doesn't need bother to allocate or to manage buffers for the encoded or decoded data. Automatic buffer management can be switched off at run-time, though, for maximum performance. \paragraph{Performance.} Since all transferred data has to pass through XDS, the library has been written to encode and decode with maximum performance. The generic encoding framework adds almost no run-time overhead to the actual encoding process: If non-automatic buffer management has been selected, hardly anything but the actual encoding/decoding engine is executed. \paragraph{Robustness.} In order to verify that the library works correctly, a set of regression tests is included in the distribution. These test suites will --- among other things --- encode known values and compare the result with the expected (correct) values. This ensures that XDS works correctly on any platform. \paragraph{Use of standard formats.} The supported XDR and XML formats are widely known and accepted, meaning that XDS is interoperable with other marshaling implementations: It is possible to encode data with XDS and to decode it with an entirely different XDR implementation or vice versa. \paragraph{Portability.} XDS has been written with portability in mind. Development took place on FreeBSD, Linux and Solaris; other platforms were used to test the results. It is expected that XDS will compile and function on virtually any POSIX.1-compliant system with a moderately modern ISO-C compiler. GNU's C~Compiler~(gcc) is known to compile the library just fine. For maximum portability, GNU Autoconf has been used to determine the target system's properties. \section{Architecture of XDS} \begin{figure}[htb] \begin{center} \includegraphics[width=\textwidth]{architecture.eps} \caption{Components of XDS} \label{XDS components} \end{center} \end{figure} The architecture of XDS is illustrated in figure~\ref{XDS components}. XDS consists of three components: The generic encoding and decoding framework, a set of engines to encode and decode values in a certain format, and a run-time context, which is used to manage buffers, registered engines, etc. In order to use the library, the first thing the developer has to do is to create a valid XDS context by calling \textsf{xds\_init()}. The routine requires one parameter that determines whether to operate in encoding- or decoding mode. A context can be used for encoding or decoding only; it is not possible to use the same context for both operations. Once a valid XDS context has been obtained, the routine \textsf{xds\_register()} can be used to register an arbitrary number of encoding or decoding engines within the context. Two sets of engines are included in the library. These routines will handle any elementary datatype defined by the ISO-C language, such as 32-bit integers, 64-bit integers, unsigned integers (of both 32- and 64-bit), floating point numbers, strings and octet streams. Once all required encoding/decoding engines are registered, the routines \textsf{xds\_encode()} or \textsf{xds\_\-decode()} may be used to actually perform the encoding or decoding process. Any data type for which an engine has been registered can be handled by the library. This means, that it is possible for the developer to write custom engines for any data type he desires to use and to register them in the context --- as long as these engines adhere to the \textsf{xds\_engine\_t} interface defined in \textsf{xds.h}. In particular, it is possible to register meta engines. That is an engine designed to encode or decode data types, which consist of several elementary data types. Such an engine will simply re-use the existing engines to encode or decode the elements of the structure. \section{Using the XDS library} \subsection{Encoding} The following example program will encode three variables using the XDR engines. The encoded results will be written to the standard output stream, which can be redirected to a file or piped into the decoding program described in the next section. Just take a look at the source code for a moment, we will then go on to discuss all relevant sections line by line. \begin{Verbatim}[numbers=left,fontsize=\small,frame=lines] #include #include #include #include #include static void error_exit(int rc, const char* msg, ...) { va_list args; va_start(args, msg); vfprintf(stderr, msg, args); va_end(args); exit(rc); } int main() { xds_t* xds; char* buffer; size_t buffer_size; xds_int32_t int32 = -42; xds_uint32_t uint32 = 0x12345678; const char* string = "This is a test."; xds = xds_init(XDS_ENCODE); if (xds == NULL) error_exit(1, "Failed to initialize XDS context: %s\n", strerror(errno)); if (xds_register(xds, "int32", &xdr_encode_int32, NULL) != XDS_OK || xds_register(xds, "uint32", &xdr_encode_uint32, NULL) != XDS_OK || xds_register(xds, "string", &xdr_encode_string, NULL) != XDS_OK) error_exit(1, "Failed to register my encoding engines!\n"); if (xds_encode(xds, "int32 uint32 string", int32, uint32, string) != XDS_OK) error_exit(1, "xds_encode() failed!\n"); if (xds_getbuffer(xds, XDS_GIFT, (void**)&buffer, &buffer_size) != XDS_OK) error_exit(1, "getbuffer() failed.\n"); xds_destroy(xds); write(STDOUT_FILENO, buffer, buffer_size); free(buffer); fprintf(stderr, "Encoded data:\n"); fprintf(stderr, "\tint32 = %d\n", int32); fprintf(stderr, "\tuint32 = 0x%x\n", uint32); fprintf(stderr, "\tstring = \"%s\"\n", string); return 0; } \end{Verbatim} \paragraph{Lines 1--5.} The program starts by including several system headers, which define the prototypes for the routines we use. The most interesting header in our case is of course \textsf{xds.h} --- the header of XDS. Please note that all declarations required to use XDS are included in that file. \paragraph{Lines 7--13.} The \textsf{error\_exit()} routine is not relevant for the example; we just define it to make the rest of the source code shorter and easier to read. \paragraph{Lines 16--53.} This is the interesting part: The \textsf{main()} routine. This function will create the variables to be encoded on the stack, assign values to them, initialize the XDS library, use it to encode the values, and write the result of the encoding process to the standard output stream. Read on for further details. \paragraph{Lines 26--28.} First of all we have to obtain an XDS context. This is done by calling \textsf{xds\_init()}. Since we intend to \emph{encode} data, we initialize the context in encoding mode. The only other mode of operation would be decoding mode, but this is demonstrated in the next section. All other routines in XDS return a code from a small list of return codes defined in \textsf{xds.h}, but \textsf{xds\_init()} is different: It returns a pointer to an \textsf{xds\_t} in case of success and \textsf{NULL} in case of failure. One reason why \textsf{xds\_init()} would fail is because it can't allocate the memory required to initialize the context. In this case, the system variable \textsf{errno} is set to \textsf{ENOMEM}. The reason why \textsf{xds\_init()} would fail is because the mode parameter is invalid, in which case \textsf{errno} woulde be set to \textsf{EINVAL}. If XDS has been compiled with assertions enabled, such an error would result in an assertion error, terminating the program with a diagnostic message immediately. \paragraph{Lines 30--33.} Once we have obtained a valid XDS context, we register the engines we need. In this example, we'll encode a signed 32-bit integer, an unsigned 32-bit integer, and a string. We'll be using XDR encoding in this case, so the engines to register are \textsf{xdr\_encode\_int32()}, \textsf{xdr\_encode\_uint32()}, and \textsf{xdr\_encode\_string()}. (A complete list of available engines can be found in \textsf{xds.h}, in section~\ref{xdr}~and~\ref{xml}, or in the manual pages for the library.) Please note that we could switch the encoding format simply by using the corresponding \textsf{xml\_encode\_XXX()} engines here. We could even mix XDR and XML encoding as we see fit, but it's hard to think of a case where this would make sense~\dots{} The developer is free to choose a name he'd like to register an engine under, but the name may only contain alphanumerical characters, hyphens (``\verb#-#'') or underscores (``\verb#_#''). You can choose any name you want, but it is recommended to follow the naming scheme of the corresponding engine. Why this is recommended will become clear in section~\ref{meta engines}. \paragraph{Lines 35--36.} This is where the actual encoding takes place. As parameters, \textsf{xds\_encode()} requires a valid encoding context plus a format string that describes how the following parameters are to be interpreted. While the concept is obviously identical to \textsf{sprintf(3)}, the syntax is different: The format string may contain an arbitrary number of names, which are delimited by an arbitrary number of any character that is not a legal character for engine names. We recommend to delimit the names by colons or blanks. For each valid engine name in the format string, a corresponding parameter must follow in the varadic argument. How these parameters are interpreted depends on the engine you're using. The engines provided with the XDS library will expect the value to encode, but theoretically developers are free to write encoding and decoding engines that expect virtually any kind of information here. More about this will explained in section~\ref{meta engines}. \paragraph{Lines 38--39.} We have encoded all values we wanted to encode, now we can get the result from the library. This happens by calling \textsf{xds\_getbuffer()}. The routine will store the buffer's address and length at the locations we provided as parameters. Please note that we can choose whether we want the buffer as a ``gift'' (\textsf{XDS\_GIFT}) or as a ``loan'' (\textsf{XDS\_LOAN}). The buffer being a ``loan'' means that the buffer is still owned by the library --- we're only allowed to peek at it. Any call to an XDS routine may modify the buffer or even change the buffers location. Hence the result of a \textsf{xds\_getbuffer()} call with loaning semantics is only valid until the next XDS routine is called. If we ``gift'' semantics, the buffer we receive will be owned by us; the library will not touch the buffer again. This means of course, that we're responsible for \textsf{free(3)}ing the buffer when we don't need it anymore. \paragraph{Line 41.} Destroy the XDS context and all data associated with it. This is possible because we requested the buffer as ``gift''; the buffer is not associated with XDS anymore. \paragraph{Line 43.} Write the buffer with the encoded data to the standard output stream. \paragraph{Line 45.} Now that we don't need the buffer anymore, we can return the memory it uses to the system. \paragraph{Lines 47--50.} Write a short report of what we have done to the standard error channel. \bigskip Finally, let us compile and execute the example program shown above. For convenience, it is included in the distribution as \textsf{docs/encode.c}. You can compile and execute the program as follows: \begin{quote} \begin{verbatim} $ gcc -I.. encode.c -o encode -L.. -lxds $ ./encode >output Encoded data: int32 = -42 uint32 = 0x12345678 string = "This is a test." $ ls -l output -rw-r--r-- 1 simons simons 28 Aug 2 15:21 output \end{verbatim} \end{quote} The result of executing the programm --- the file \textsf{output} --- can be displayed with \textsf{hexdump(1)} or \textsf{od(1)} and should look like this: \begin{quote} \begin{Verbatim}[fontsize=\small] $ hexdump -C output 00000000 ff ff ff d6 12 34 56 78 00 00 00 0f 54 68 69 73 |.....4Vx....This| 00000010 20 69 73 20 61 20 74 65 73 74 2e 00 | is a test..| 0000001c \end{Verbatim} \end{quote} \noindent We will also re-use this file in the next section, where we'll read it and decode those values again. \subsection{Decoding} The following example program will read the result of the encoding example shown in the previous section and decode the values back into the native representation. Then it will print those values to the standard error stream so that the user can see the values are correct. Please take a look at the source now, we'll discuss all relevant details in the following paragraphs. \begin{Verbatim}[numbers=left,fontsize=\small,frame=lines] #include #include #include #include #include static void error_exit(int rc, const char* msg, ...) { va_list args; va_start(args, msg); vfprintf(stderr, msg, args); va_end(args); exit(rc); } int main() { xds_t* xds; char buffer[1024]; size_t buffer_len; int rc; xds_int32_t int32; xds_uint32_t uint32; char* string; buffer_len = 0; do { rc = read(STDIN_FILENO, buffer + buffer_len, sizeof(buffer) - buffer_len); if (rc < 0) error_exit(1, "read() failed: %s\n", strerror(errno)); else if (rc > 0) buffer_len += rc; } while (rc > 0 && buffer_len < sizeof(buffer)); if (buffer_len >= sizeof(buffer)) error_exit(1, "Too much input data for our buffer.\n"); xds = xds_init(XDS_DECODE); if (xds == NULL) error_exit(1, "Failed to initialize XDS context: %s\n", strerror(errno)); if (xds_register(xds, "int32", &xdr_decode_int32, NULL) != XDS_OK || xds_register(xds, "uint32", &xdr_decode_uint32, NULL) != XDS_OK || xds_register(xds, "string", &xdr_decode_string, NULL) != XDS_OK) error_exit(1, "Failed to register my decoding engines!\n"); if (xds_setbuffer(xds, XDS_LOAN, buffer, buffer_len) != XDS_OK) error_exit(1, "setbuffer() failed.\n"); if (xds_decode(xds, "int32 uint32 string", &int32, &uint32, &string) != XDS_OK) error_exit(1, "xds_decode() failed!\n"); xds_destroy(xds); fprintf(stderr, "Decoded data:\n"); fprintf(stderr, "\tint32 = %d\n", int32); fprintf(stderr, "\tuint32 = 0x%x\n", uint32); fprintf(stderr, "\tstring = \"%s\"\n", string); free(string); return 0; } \end{Verbatim} \paragraph{Lines 1--25.} Include the required header files, define the \textsf{error\_exit()} helper function, and create the required variables on the stack. \paragraph{Lines 27--39.} These instructions will read an any number of bytes from the standard input stream --- as long as the input does not exceed the size of the \textsf{buffer} variable. In order to provide the program with the appropriate input, redirect the standard input stream to the file \textsf{output} created in the previous section or connect the encoding and decoding programs directly by a pipe. \paragraph{Lines 41-43.} Create a context for decoding the values. The semantics are identical to those described in the encoding example. \paragraph{Lines 45--48.} Register the decoding engines in the context. Please note that obviously the decoding engines must correspond to the encoding engines used to create the data we're about to process. Using, say, an XML engine to decode XDR data will at best return an error --- in the worst case, it will return incorrect results! \paragraph{Lines 50-51.} Here we do not \emph{get} a buffer from the library, we \emph{set} the buffer we've read earlier in the context for decoding. Please note that we use loan semantics in this case, not gift semantics. This is necessary because \textsf{buffer} has not been allocated by \textsf{malloc(3)} --- the variable is located on the stack. This means, that we cannot give it to XDS because XDS expects to be able to \textsf{free(3)} the buffer when the context is destroyed. Loan semantics are fine, though, all we have to do is to take care that we don't erase or modify the contents of \textsf{buffer} while XDS operates on it. The library itself will never touch the buffer in decode mode, no matter whether loan or gift semantics have been chosen. \paragraph{Lines 53--54.} Here comes the actual decoding of the buffer's content using \textsf{xds\_decode()}. The syntax is identical to \textsf{xds\_encode()}'s, the only difference is that the decoding engines do not expect values like the encoding engines did, but the location where to store the value. Thus, we pass the addresses of the appropriate variables here. If the routine returns with \textsf{XDS\_OK}, the decoded values will have been stored in those locations. It should be noted that the decoded string cannot trivially be returned this way. Instead, \textsf{xds\_decode()} will use \textsf{malloc(3)} to allocate a buffer large enough to hold the string. The address of that buffer is then stored in the pointer \textsf{string}. Of course, this means that the application has to \textsf{free(3)} the string once it's not required anymore. \paragraph{Line 56.} We don't need the context anymore, so we destroy it and thereby free all used resources. This does not affect \textsf{buffer} in any way because we used loan semantics. \paragraph{Lines 58-61.} Print the decoded values to the standard error stream for the user to take a look at them. \paragraph{Line 63.} Now that we don't need the contents of \textsf{string} anymore, we must return the buffer allocated in \textsf{xds\_decode()} to the system. \bigskip Like the encoding program described earlier, the source code to this program is included in the library distribution as \textsf{docs/decode.c}. You can compile and execute the program like this: \begin{quote} \begin{verbatim} $ gcc -I.. decode.c -o decode -L.. -lxds $ ./decode small, ms->big, ms->positive, ms->text, sizeof(ms->text)); } \end{verbatim} \end{quote} This engine takes the address of the ``mystruct'' structure from the stack and then uses \textsf{xds\_encode()} to handle all elements of ``mystruct'' separately --- which is fine, because these data types are supperted by XDS already. It is worth noting, though, that we refer to the other engines by name, meaning that these engines must be registered in ``xds'' by that name! What is very nice, though, is the fact that this encoding engine does not even need to know which engines are used to encode the actual values! If the user registeres the XDR engines under the appropriate names, ``mystruct'' will be encoded in XDR. If the user registeres the XML engines under the appropriate names, ``mystruct'' will be encoded in XML. Because of that property, we call such an engine a ``meta engine''. Of coures you need not necessarily implement an engine as \emph{meta} engine: Rather than going through \textsf{xds\_encode()}, it would be possible to execute the appropriate encoding engines directly. This had the advantage of not depending on those engines being registered at all, but it would make the custom engine depend on the elementary engines --- what is an unnecessary limitation. One more word about the engine syntax and semantics: As has been mentioned earlier, any function that adheres to the interface shown above is potentially an engine. These parameters have the following meaning: \begin{itemize} \item xds --- This is the XDS context that was originally provided to the \textsf{xds\_encode()} call, which in turn executed the engine. It may be used, for example, for executing \textsf{xds\_en\-code()} again like we did in our example engines. \item engine\_context --- The engine context can be used by the engine to store any type of internal information. The value the engine will receive must have been provided when the engine was registered by \textsf{xds\_register()}. Engines obviously may neglect this parameter if they don't need a context of their own --- all engines included in the distribution do so. \item buffer --- This parameter points to the buffer the encoded data should be written to. In decoding mode, ``buffer'' points to the encoded data, which should be decoded; the location where the results should be stored at can be found on the stack then. \item buffer\_size --- The number of bytes available in ``buffer''. In encoding mode, this means ``free space'', in decoding mode, ``buffer\_size'' determines how many bytes of encoded data are available in ``buffer'' for consumption. \item used\_buffer\_size --- This parameter points to a variable, which the callback must set before returning in order to let the framework know how many bytes it consumed from ``buffer''. A callback encoding, say, an int32 number into a 8 bytes text representation would set the used\_buffer\_size to 8: \begin{quote} \begin{verbatim} *used_buffer_size = 8; \end{verbatim} \end{quote} In encoding mode, this variable determines how many bytes the engine has written into ``buffer''; in decoding mode the variable determines how many bytes the engines has read from ``buffer''. \item args --- This pointer points to an initialized varadic argument. Use the standard C macro \textsf{va\_arg(3)} to fetch the actual data. \end{itemize} A callback may return any of the following return codes, as defined in \textsf{xds.h}: \begin{itemize} \item XDS\_OK --- No error. \item XDS\_ERR\_NO\_MEM --- Failed to allocate required memory. \item XDS\_ERR\_OVERFLOW --- The buffer is too small to hold all encoded data. The callback may set ``*used\_buffer\_size'' to the number of bytes it needs in ``buffer'', thereby giving the framework a hint by how many bytes it should enlarge the buffer before trying the engine again, but just leaving ``*used\_buffer\_size'' alone will work fine too, it may just be a bit less efficient in some cases. Obviously this return code does not make much sense in decoding mode. \item XDS\_ERR\_INVALID\_ARG --- Unexpected or incorrect parameters. \item XDS\_ERR\_TYPE\_MISMATCH --- This return code will be returned in decoding mode in case the decoding engine realizes that the data it is decoding does not fit what it is expecting. Not all encoding formats will allow to detect this at all. XDR, for example, does not. \item XDS\_ERR\_UNDERFLOW --- In decode mode, this error is be returned when an engine needs, say, 4 bytes of data in order to decode a value but ``buffer''/''buffer\_size'' provides less. \item XDS\_ERR\_UNKNOWN --- Any other reason to fail than those listed before. Catch all~\dots{} \end{itemize} Let's take a look at the corresponding decoding engine now: \begin{quote} \begin{verbatim} static int decode_mystruct(xds_t* xds, void* engine_context, void* buffer, size_t buffer_size, size_t* used_buffer_size, va_list* args) { struct mystruct* ms; size_t i; char* tmp; int rc; ms = va_arg(*args, struct mystruct*); rc = xds_decode(xds, "int32 int64 uint32 octetstream", &(ms->small), &(ms->big), &(ms->positive), &tmp, &i); if (rc == XDS_OK) { if (i == sizeof(ms->text)) memmove(ms->text, tmp, i); else rc = XDS_ERR_TYPE_MISMATCH; free(tmp); } return rc; } \end{verbatim} \end{quote} The engine simply calls \textsf{xds\_decode()} to handle the separate data types. The only complication is that the octet stream decoding engines return a pointer to \textsf{malloc(3)}ed buffer --- what is not what we need. Thus we have to manually copy the contents of that buffer into the right place in the structure and free the (now unused) buffer again. A complete example program encoding and decoding ``mystruct'' can be found at \textsf{docs/\-extended.c} in the distribution. \section{The XDS Framework} \label{xds} \subsection{xds\_t* xds\_init(xds\_mode\_t~\underline{mode});} This routine creates and initializes a context for use with the XDS library. The ``mode'' parameter may be either \textsf{XDS\_ENCODE} or \textsf{XDS\_DECODE}, depending on whether you want to encode or to decode data. If successful, \textsf{xds\_init()} returns a pointer to the XDS context structure. In case of failure, \textsf{xds\_init()} returns \textsf{NULL} and sets \textsf{errno} to ENOMEM (failed to allocate internal memory buffers) or EINVAL (``mode'' parameter was invalid). A context obtained from \textsf{xds\_init()} must be destroyed by \textsf{xds\_destroy()} when it is not needed any more. \subsection{void xds\_destroy(xds\_t*~\underline{xds});} \textsf{xds\_destroy()} will destroy an XDS context created by \textsf{xds\_init()}. Doing so will return all resources associated with this context --- most notably the memory used to buffer the results of encoding or decoding any values. A context may not be used after it has been destroyed. \subsection{int xds\_register(xds\_t*~\underline{xds}, const~char*~\underline{name}, xds\_engine\_t~\underline{engine}, void*~\underline{engine\_context});} This routine will register an engine in the provided XDS context. An ``engine'' is potentially any function that fullfils the following interface: \begin{quote} \begin{verbatim} int engine(xds_t* xds, void* engine_context, void* buffer, size_t buffer_size, size_t* used_buffer_size, va_list* args); \end{verbatim} \end{quote} By calling \textsf{xds\_register()}, the engine ``engine'' will be registered under the name ``name'' in the XDS context ``xds''. The last parameter ``engine\_context'' may be used as the user sees fit: It will be passed when the engine is actually called and may be used to implement an engine-specific context. Most engines will not need a context of their own, in which case \textsf{NULL} should be specified here. Please note that until the user calls \textsf{xds\_register()} for an XDS context he obtained from \textsf{xds\_init()}, no engines are registered for that context. Even the engines included in the library distribution are not registered automatically. For engine names, any combination of the characters ``a--z'', ``A--Z'', ``0--9'', ``-'', and ``\_'' may be used; anything else is not a legal engine name component. \textsf{xds\_register()} may return the following return codes: \textsf{XDS\_OK} (everything went fine; the engine is registered now), \textsf{XDS\_ERR\_INVALID\_ARG} (either ``xds'', ``name'', or ``engine'' are \textsf{NULL} or ``name'' contains illegal characters for an engine name), or \textsf{XDS\_ERR\_NO\_MEM} (failed to allocate internally required buffers). \subsection{int xds\_unregister(xds\_t*~\underline{xds}, const~char*~\underline{name});} \textsf{xds\_unregister()} will remove the engine ``name'' from XDS context ``xds''. The function will return \textsf{XDS\_OK} in case everything went fine, \textsf{XDS\_ERR\_UNKNOWN\_ENGINE} in case the engine ``name'' is not registered in ``xds'', or \textsf{XDS\_ERR\_INVALID\_ARG} if either ``xds'' or ``name'' are \textsf{NULL} or ``name'' contains illegal characters for an engine name. \subsection{int xds\_setbuffer(xds\_t*~\underline{xds}, xds\_scope\_t~\underline{flag}, void*~\underline{buffer}, size\_t~\underline{buffer\_len});} \begin{figure}[tbh] \begin{center} \includegraphics[width=\textwidth]{setbuffer-logic.eps} \caption{xds\_setbuffer() modes of operation} \label{setbuffer logic} \end{center} \end{figure} This routine allows the user to control XDS' buffer handling: Calling it will replace the buffer currently used in ``xds''. The address and size of that buffer are passed to \textsf{xds\_setbuffer()} via the ``buffer'' and ``buffer\_len'' parameters. The ``xds'' parameter determines for which XDS context the new buffer will be set. Furthermore, you can set ``flag'' to either \textsf{XDS\_GIFT} or \textsf{XDS\_LOAN}. \textsf{XDS\_GIFT} will tell XDS that the provided buffer is now owned by the library and that it may be resized by calling \textsf{realloc(3)}. Furthermore, the buffer is \textsf{free(3)}ed when ``xds'' is destroyed. If ``flag'' is \textsf{XDS\_GIFT} and ``buffer'' is \textsf{NULL}, \textsf{xds\_setbuffer()} will simply allocate a buffer of its own to be set in ``xds''. Please note that a buffer given to XDS as gift \emph{must} have been allocated using \textsf{malloc(3)} --- it may not live on the stack because XDS will try to free or to resize the buffer as it sees fit. Passing \textsf{XDS\_LOAN} via ``flag'' tells \textsf{xds\_setbuffer()} that the buffer is owned by the application and that XDS should not free nor resize the buffer in any case. In this mode, passing a buffer of \textsf{NULL} will result in an invalid-argument error. \subsection{int xds\_getbuffer(xds\_t*~\underline{xds}, xds\_scope\_t~\underline{flag}, void**~\underline{buffer}, size\_t*~\underline{buffer\_len});} This routine is the counterpart to \textsf{xds\_setbuffer()}: It will get the buffer currently used in the XDS context ``xds''. The address of that buffer is stored in the location ``buffer'' points to; the length of the buffer's content will be stored in the location ``buffer\_len'' points to. The ``flag'' argument may be set to either \textsf{XDS\_GIFT} or \textsf{XDS\_LOAN}. The first setting means that the buffer is now owned by the application and that XDS must not use it after this \textsf{xds\_getbuffer()} call anymore; the library will allocate a new internal buffer instead. Of course, this also means that the buffer will not be freed by \textsf{xds\_destroy()}; the application has to \textsf{free(3)} the buffer itself when it is not needed anymore. Setting ``flag'' to \textsf{XDS\_LOAN} tells XDS that the application just wishes to peek into the buffer and will not modify it. The buffer is still owned (and used) by XDS. Please note that the loaned address returned by \textsf{xds\_getbuffer()} may change after any other \textsf{xds\_xxx()} function call! The routine will return \textsf{XDS\_OK} (everything went fine) or \textsf{XDS\_ERR\_INVALID\_ARG} (``xds'', ``buffer'' or ``buffer\_len'' are \textsf{NULL} or ``flag'' is invalid) signifying success or failure respectively. Please note: It is perfectly legal for \textsf{xds\_getbuffer()} to return a buffer of \textsf{NULL} and a buffer length of 0! This happens when \textsf{xds\_getbuffer()} is called for an XDS context before a buffer has been allocated. \subsection{int xds\_vencode(xds\_t*~\underline{xds}, const~char*~\underline{fmt}, va\_list~\underline{args});} This routine will encode one or several values using the appropriate encoding engines registered in XDS context ``xds''. The parameter ``fmt'' contains a \textsf{sprintf(3)}-alike descriptions of the values to be encoded; the actual values are provided in the varadic parameter ``args''. The format for ``fmt'' is simple: Just provide the names of the engines to be used for encoding the appropriate value(s) in ``args''. Any non-legal engine-name character may be used as a delimiter. In order to encode two 32-bit integers followed by a 64-bit integer, the format string \begin{quote} \begin{verbatim} int32 int32 int64 \end{verbatim} \end{quote} could be used. In case you don't like the blank, use the colon instead: \begin{quote} \begin{verbatim*} int32:int32:int64 \end{verbatim*} \end{quote} Of course the names to be used here have to match to the names used to register the engines in ``xds'' earlier. Every time \textsf{xds\_vencode()} is called, it will append the encoded data at the end of the internal buffer stored in ``xds''. Thus, you can call \textsf{xds\_vencode()} several times in order to encode several values, but you'll still get all encoded values stored in one buffer. Calling \textsf{xds\_setbuffer()} or \textsf{xds\_getbuffer()} with gift semantics at any point during encoding will re-set the buffer to the beginning. All values that have been encoded into that buffer already will eventually be overwritten when \textsf{xds\_encode()} is called again. Hence: Don't call \textsf{xds\_setbuffer()} or \textsf{xds\_getbuffer()} unless you actually want to access the data stored in the buffer. Also, it should be noted that the data you have to provide for ``args'' depends entirely on what the deployed engines expect to find on the stack --- there is no ``standard'' on what should be put on the stack here. The XML and XDR engines included in the distribution will simply expect the value to be encoded to be found on the stack, but other engines may act differently. See section~\ref{meta engines} for an example of such an engine. \textsf{xds\_vencode()} will return any of the following return codes: \textsf{XDS\_OK} (everything worked fine), \textsf{XDS\_ERR\_NO\_MEM} (failed to allocate or to resize the internal buffer), \textsf{XDS\_ERR\_OVER\-FLOW} (the internal buffer is too small but is not owned by us), \textsf{XDS\_ERR\_INVALID\_ARG} (``xds'' or ``fmt'' are \textsf{NULL}), \textsf{XDS\_ERR\_UNKNOWN\_ENGINE} (an engine name specified in ``fmt'' is not registered in ``xds''), \textsf{XDS\_ERR\_INVALID\_MODE} (``xds'' is initialized in decode mode), or \textsf{XDS\_ERR\_UNKNOWN} (the engine returned an unspecified error). \subsection{int xds\_encode(xds\_t*~\underline{xds}, const~char*~\underline{fmt}, \dots{});} This routine is basically identical to \textsf{xds\_vencode()}, only that it uses a different prototype syntax. \subsection{int xds\_vdecode(xds\_t*~\underline{xds}, const~char*~\underline{fmt}, va\_list~\underline{args});} This routine is almost identical to \textsf{xds\_vencode()}: It expects an XDS context, a format string and a set of parameters for the engines, but \textsf{xds\_vdecode()} does not encode any data, it decodes the data back into the native format. The format string determines which engines are to be called by the framework in order to decode the values contained in the buffer. The values will then be stored at the locations found in the corresponding ``args'' entry. But please note that the exact behavior of the decoding engines is not specified! The XML and XDR engines included in this distribution expect a pointer to a location where to store the decoded value, but other engines may vary. \textsf{xds\_vdecode()} may return any of the following return codes: \textsf{XDS\_OK} (everything went fine), \textsf{XDS\_ERR\_INVALID\_ARG} (``xds'' or ``fmt'' are \textsf{NULL}), \textsf{XDS\_ERR\_TYPE\_MISMATCH} (the format string says the next value is of type $A$, but that's not what we found in the buffer), \textsf{XDS\_ERR\_UNKNOWN\_ENGINE} (an engine name specified in ``fmt'' is not registered in ``xds''), \textsf{XDS\_ERR\_INVALID\_MODE} (``xds'' has been initialized in encode mode), \textsf{XDS\_ERR\_UNDER\-FLOW} (an engine tried to read $n$ bytes from the buffer, but we don't have that much data left), or \textsf{XDS\_ERR\_UNKNOWN} (an engine returned an unspecified error). \subsection{int xds\_decode(xds\_t*~\underline{xds}, const~char*~\underline{fmt}, \dots{});} This routine is basically identical to \textsf{xds\_vdecode()}, only that it uses a different prototype syntax. \section{The XDR Engines} \label{xdr} \begin{tabular}{|c|c|c|c|} \hline \bf Function Name & \bf Expected ``args'' Datatype & \bf Input & \bf Output \\ \hline xdr\_encode\_uint32() & xds\_uint32\_t & 4 bytes & 4 bytes \\ xdr\_decode\_uint32() & xds\_uint32\_t* & 4 bytes & 4 bytes \\[1ex] xdr\_encode\_int32() & xds\_int32\_t & 4 bytes & 4 bytes \\ xdr\_decode\_int32() & xds\_int32\_t* & 4 bytes & 4 bytes \\[1ex] xdr\_encode\_uint64() & xds\_uint64\_t & 4 bytes & 4 bytes \\ xdr\_decode\_uint64() & xds\_uint64\_t* & 4 bytes & 4 bytes \\[1ex] xdr\_encode\_int64() & xds\_int64\_t & 4 bytes & 4 bytes \\ xdr\_decode\_int64() & xds\_int64\_t* & 4 bytes & 4 bytes \\[1ex] xdr\_encode\_float() & xds\_float\_t & 4 bytes & 4 bytes \\ xdr\_decode\_float() & xds\_float\_t* & 4 bytes & 4 bytes \\[1ex] xdr\_encode\_double() & xds\_double\_t & 8 bytes & 8 bytes \\ xdr\_decode\_double() & xds\_double\_t* & 8 bytes & 8 bytes \\[1ex] xdr\_encode\_octetstream() & void*, size\_t & variable & variable \\ xdr\_decode\_octetstream() & void**, size\_t* & variable & variable \\[1ex] xdr\_encode\_string() & char* & variable & variable \\ xdr\_decode\_string() & char** & variable & variable \\ \hline \end{tabular} \medskip Please note that the routines \textsf{xdr\_decode\_octetstream()} and \textsf{xdr\_decode\_string()} return a pointer to a buffer holding the decoded data. This buffer has been allocated with \textsf{malloc(3)} and must be \textsf{free(3)}ed by the application when it is not required anymore. All other callbacks write the decoded value into the location found on the stack, but these behave differently because the length of the decoded data is not known in advance and the application cannot provide a buffer that's guaranteed to suffice. \section{The XML Engines} \label{xml} \begin{tabular}{|c|c|c|c|} \hline \bf Function Name & \bf Expected ``args'' Datatype & \bf Input & \bf Output \\ \hline xml\_encode\_uint32() & xds\_uint32\_t & 4 bytes & 18--27 bytes \\ xml\_decode\_uint32() & xds\_uint32\_t* & 18--27 bytes & 4 bytes \\[1ex] xml\_encode\_int32() & xds\_int32\_t & 4 bytes & 16--26 bytes \\ xml\_decode\_int32() & xds\_int32\_t* & 16--26 bytes & 4 bytes \\[1ex] xml\_encode\_uint64() & xds\_uint64\_t & 8 bytes & 18--37 bytes \\ xml\_decode\_uint64() & xds\_uint64\_t* & 18--37 bytes & 8 bytes \\[1ex] xml\_encode\_int64() & xds\_int64\_t & 8 bytes & 16--36 bytes \\ xml\_decode\_int64() & xds\_int64\_t* & 16--36 bytes & 8 bytes \\[1ex] xml\_encode\_float() & xds\_float\_t & 4 bytes & variable \\ xml\_decode\_float() & xds\_float\_t* & variable & 4 bytes \\[1ex] xml\_encode\_double() & xds\_double\_t & 8 bytes & variable \\ xml\_decode\_double() & xds\_double\_t* & variable & 8 bytes \\[1ex] xml\_encode\_octetstream() & void*, size\_t & variable & variable \\ xml\_decode\_octetstream() & void**, size\_t* & variable & variable \\[1ex] xml\_encode\_string() & char* & variable & variable \\ xml\_decode\_string() & char** & variable & variable \\ \hline \end{tabular} \medskip Please note that the routines xml\_decode\_octetstream() and xml\_decode\_string() return a pointer to a buffer holding the decoded data. This buffer has been allocated with \textsf{malloc(3)} and must be \textsf{free(3)}ed by the application when it is not required anymore. All other callbacks write the decoded value into the location found on the stack, but these behave differently because the length of the decoded data is not known in advance and the application cannot provide a buffer that's guaranteed to suffice. \section{Frequently Asked Questions} \subsection{Why do we have separate encoding and decoding modes?} Some users complained about having to maintain separate XDS contexts for encoding and decoding. They wondered, why it is not possible to encode and decode with a single XDS context. The reason is that this limitatiton makes the XDS context structure and the programmer's API for XDS much simpler. If we were able to use a single context for encoding and decoding, we had to maintain \emph{two} lists of registered engines per XDS context: One set of encoding engines and one set of decoding engines. Consequently, the \textsf{xds\_register()} function would need to take an additional parameter, which determines whether you're registering an encoding or an decoding engine. All this is not necessary in the current design, because one list of registered engines suffices. Another important topic is buffer management. The buffer handling in encoding mode is subtly different from that in encoding mode: The XDS context contains a buffer, the size of that buffer and a kind of ``current position'' pointer. When an engine stores, say, 8 bytes of encoded data in the buffer, \textsf{xds\_vencode()} will increase the current position by 8 bytes --- the next encoding engine will append its encoded data at the end of the buffer. If the current position reaches the end of the buffer, the buffer is reallocated with an appropriately bigger size. In decoding mode, the same variables in the XDS context have a different meaning: Since the buffer is never going to be resized, the buffer size does not correspond to the size of the memory chunk that constitutes the buffer, it says how many bytes of information the buffer contains; it's the length of the contents. The current position is initialized to the beginning of the buffer and every time an engine claims to have decoded, say, 8 bytes from the buffer, the current position is increased by 8 bytes towards the end of the buffer. If the current position reaches the end of the buffer's contents, an \textsf{XDS\_UNDERFLOW} error is returned. Buffer handling is different in encoding and decoding mode in so far as that in encoding mode, the initial buffer is empty and the current position moves with the end of the content, determining where new data should be appended. In decoding mode, the initial buffer is filled and the current position wanders from the beginning to the end of the content. Thus, if an XDS context should be used for both encoding and decoding, the library had to manage two different buffers because the encoding and decoding buffers have different semantics. Thus, the \textsf{xds\_setbuffer()} and \textsf{xds\_getbuffer()} routines would need an additional parameter in order to set the two buffers independently. Considering all that, we found that the current design greatly reduces the complexity of the implementation and of the API while putting the user only through minimal inconvenience. \subsection{What are those xds\_int-something types good for?} \label{xds int stuff} The XDS library uses the data types \textsf{xds\_int32\_t}, etc.\ rather than \textsf{int}. This is necessary because we need to have a definive size for each data type. In ISO-C, though, the actual size of an \textsf{int} is undefined. In theory, the system header \textsf{sys/types.h} defines types with fixed sizes, but unfortunately the names of these data types vary from vendor to vendor. To solve that, we defined our own data types. The application programmer might want to take a look at the first few lines of the \textsf{xds.h} include file to see how the actual data types are mapped to the \textsf{xds\_xxx\_t} variant on your system. \subsection{Why do I have to register all the engines manually?} One idea that came up during the design of the API was to provide a way to register all elementary XML or XDR engines with a single function call, something like this: % \begin{quote} \begin{verbatim} xds = xds_init(XDS_ENCODE, XDS_XML); /* Use the XML engines. */ xds = xds_init(XDS_ENCODE, XDS_XDR); /* Use the XDR engines. */ \end{verbatim} \end{quote} % The advantage of this approach is that the application developer does not need to bother about registering some obscure functions like \textsf{xdr\_encode\_octetstream()}. We dismissed the idea nonetheless for the following reasons: \begin{itemize} \item Since the library is meant to be extensible, the \textsf{xds\_init()} has no (good) way of knowing which engines actually exist for an encoding scheme. Suppose someone writes a whole set of engines that implement the CORBA format, then he would not be able to register his engines without re-writing \textsf{xds\_init()}. \item On a similar notion, \textsf{xds\_init()} would not know about the meta engines required by the application developer. The call outlined above would only register the engines for the elementary data types; the meta engines do not even ``exist'' when the XDS library is compiled. \item This approach would make it hard to mix engines from different formats. If all engines are registered manually, the application programmer may chose to use the XDR format for encoding all kinds of integers, but to use the XML format for encoding strings, octet streams, or floating point numbers. \item If one routine would reference all engines of an encoding format, it meant that all engines of that format were linked into the binary once the application accesses that routine. It would not be possible to, say, register the engines dealing with integers without pulling the floating point engines into the program too --- even though nobody uses them. The author of this document wishes to remark, though, that this property of the library was later uh~\dots{} removed by the decision of the team leader to merge all engines into one source module per format. Sorry. \end{itemize} \begin{thebibliography}{xxx} \bibitem{xdr} RFC 1832: ``XDR: External Data Representation Standard'', R.~Srinivasan, August~1995 \bibitem{xml} XML-RPC Home Page: \textsf{http://www.xmlrpc.org/} \end{thebibliography} \end{document}