ossp-pkg/sio/BRAINSTORM.txt
1.4
Names:
o SIO = simple/streams/stacked I/O
o FIO = fast I/O
o IOL = IO Lite (achtung)
o LIO = Lite/Layered IO
o AIO = Abstracted I/O
Connectors/Plugs:
o Memory
SIO_PLUG *sio_plug_mem(uchar *buf, ulong len, uchar *(cb)(ulong));
- connect to preallocated buffer + callback function for more buffer allocs
- connect internally to auto-growing buffer(s)
o File
constructors:
- from fd
- from FILE
- from pathname
o Socket
- UDP
- TCP
o FILE*-2-SIO
o fd-2-SIO
o file-2-SIO
o TCP/socket-2-SIO
o UDP/socket-2-SIO
o Pipe/FIFO-2-SIO
o Null
Discard Plug
SIO *sio_plug_null(void)
Filters:
o Transparent; sio_pipe_trans
o Regex-Matching-Filter
o Regex-Subst-Filter
o MD5/SHA1/DES/IDEA
o Zlib/LZO
o SSL
IO Models (according to Stevens: Unix Network programming I, p. 144)
o blocking I/O
o nonblocking I/O (NO_HANG, NONBLK)
o I/O multiplexing (select and poll)
o signal driven I/O (SIGIO)
o asynchronous I/O (POSIX aio_xxx)
Data Structures
o Rings internal
o malloc(3) and mm_malloc(3) aware
Buffering-Modes:
o flush after timeout, close and exit() + explicit via flush()
o no buffering at all
o Chunking for HTTP Support!
o only read is buffered
o only write is buffered
o read & write is buffered
Buffering:
o filter koennen window/buffer size einstellen
o filter geben an ob sie shrinken/expandieren/gleichlassen oder random sind
o source connector kann eventuell Buffer vom User direkt nutzen
o umgekehrt kann auch user buffer wird von target connector genutzt werden
o application koennte auch bei write angeben, dasz buffer destroyed
werden darf und dasz er eventuell groesser ist als die daten,
die aus ihm geschrieben werden -> nuetzlich fuer expanding filters
bei der Ausgabe.
Optimierungen:
o mmap muss moeglich sein
o if target connector = socket & source connector = file => try to use sendfile(2)
o if target connector = file & source connector = file => try to use mmap(2)
o wenn I/O ohne filters dann no buffering oder nur wenn chunks zu klein
o wann es geht writev() und readv() nutzen, um Buffers zu koppeln beim I/O
o sockets haben fuer read/write() einen low water mark (siehe setsockopt)
o FreeBSD 4.0's accept_filter(9) mechanism to check for HTTP request
NOTES:
o Timeouts muessen generell unterstuetzt werden
o Error handling optional mit callbackfunction+void
o Callback functions for exceptional things
o Socket-Connector muss z.B. shutdown() erlauben
o man sollte meta-data (mtime, inode, etc.) von SIO connector rausfinden
koennen und setzen koennen
o Chunking muss moeglich sein (HTTP?)
o Read sollte lesen bis:
- number of bytes read
- found a char of class [a-z]
- found a string "foo"
- found a regex "..."
o man sollte auf jedem SIO seek()en koennen
o man sollte SIOs in FILEs umwandeln koennen???
o bei Socket Plugs muss shutdown() also eine Seite moeglich sein
Architecture:
o three types of objects (siehe SIO.fig):
- plugs (Stecker/Anschluesse) for socket, file, mem, etc.
- pipes (Rohre/Verbindungen)
- pipelines (the Kombination von plugs und pipes zu einer Einheit)
o buffered I/O: es gibt eine buffer-pipe, das ist alles
unbuffered I/O ist also eine Pipleline wo kein buffer-pipe dabei ist
Conversions:
- ASCII2EBCDIC and vice versa conversions
o SIO Disciplines for Virtual Filesystem Stuff
URLs as pathname open various network things, tarballs, etc.
file, http, ftp, extfs
?? vfs, libfetch, libcurl ??
Performance Gains:
- use sendfile()
- use TCP_CORK
- use ...
Filter classes (from Apache discussions):
1) content-generator (files, CGI, database, etc)
2) content-filter/munger/processor (SSI, PHP, etc)
3) content-encoding (gzip?)
4) digest/message processor? (mod_auth_digest)
5) transport-encoding (HTTP chunking)
6) socket-encoding (SSL)
Fuer Zero-Copy:
The data the user program writes must be page sized and start on a page
boundary in order for it to be run through the zero copy send code.
Ideas:
http://sourceforge.net/projects/paip
-----------------
API Functions:
sio_rc_t sio_attach (sio_t **sio, int fd);
sio_rc_t sio_deattach (sio_t *sio, int *fd);
sio_rc_t sio_setbuffer (sio_t *sio, size_t newsize, size_t *oldsize);
sio_rc_t sio_readvec (sio_t *sio, void **vec, size_t *veclen);
sio_rc_t sio_read (sio_t *sio, void *buf, size_t *buflen);
sio_rc_t sio_readline (sio_t *sio, void *buf, size_t *buflen);
sio_rc_t sio_readchar (sio_t *sio, char *c);
sio_rc_t sio_putback (sio_t *sio, void *buf, size_t buflen);
sio_rc_t sio_undo (sio_t *sio);
sio_rc_t sio_writevec (sio_t *sio, void **vec, size_t veclen);
sio_rc_t sio_write (sio_t *sio, void *buf, size_t buflen);
sio_rc_t sio_writestr (sio_t *sio, char *str);
sio_rc_t sio_writeline (sio_t *sio, void *buf, size_t buflen);
sio_rc_t sio_writechar (sio_t *sio, char c);
sio_rc_t sio_print (sio_t *sio, char *fmt, ...);
sio_rc_t sio_printv (sio_t *sio, char *fmt, va_list ap);
sio_rc_t sio_flush (sio_t *sio);
sio_rc_t sio_error (sio_t *sio, char **error);
sio_rc_t sio_read (sio_t *sio, sio_ioflags_t flags, ...);
sio_rc_t sio_write(sio_t *sio, sio_ioflags_t flags, ...);
SIO_CHR | SIO_STR | SIO_BUF | SIO_FMT | SIO_STRVEC | SIO_BUFVEC
type of objects: character, nul-terminated string or buffer+size or
format string based
SIO_MULT
multiple objects are passed in call
SIO_NULLEND
whether size of vector is indicated by NULL or given
SIO_COPY
objects are copied to library
(internal its SIO_GIFT after copy!)
SIO_GIFT
objects are gifted to library
SIO_LOAN
objects are just loaned to library
sio_read(sio, SIO_BUF, buf, buflen, &readlen);
sio_read(sio, SIO_LINE, buf, buflen, &readlen);
sio_write(sio, SIO_STR, "foo");
sio_write(sio, SIO_STR|SIO_MULT, "foo", "bar", NULL);
sio_write(sio, SIO_VEC|SIO_STR, vec, veclen);
sio_write(sio, SIO_VEC|SIO_STR|SIO_NULLEND, vec);
sio_write(sio, SIO_BUF, buf, buflen);
sio_write(sio, SIO_BUF|SIO_MULT, buf, buflen, buf2, buflen2, NULL);
sio_write(sio, SIO_VEC|SIO_BUF, vec, veclen);
sio_write(sio, SIO_VEC|SIO_BUF|SIO_NULLEND, vec);
sio_write(sio, SIO_STR|SIO_MULT, line, "\r\n", NULL);
sio_write(sio, SIO_CHR, c);
sio_write(sio, SIO_BUF, &c, 1);
sio_write(sio, SIO_FMT, "%c%s%S%b%B", c, cp, cpvec, cpvecsize, buf, bufsize, bufvec, bufvecsize);
API Comfort:
sio_writestr("..") -> sio_write(SIO_STR, "..");
API Standard:
sio_write(SIO_STR, char *x) -> sio_output(x, strlen(x));
sio_write(SIO_VEC, char *x) -> for... sio_output(x.ptr, x.len); done
API Basic:
sio_output
1. Was ist mit seekable fds (files!)?
seek, tell,
2. Top-level filtering and chaining?
tie, untie
-----------------
IDEA:
- SIO (Socket IO)
- BIO (Buffered/Filtered I/O)
- BA/BB (Buffer Aggregates, Bucket Brigades -- ACT)
-----------------
brigate ::= bucket *
bucket ::= <buf,len,type>
-----------------
/* SFIO: sfreserve?
* sfpool ?
*/
/*
* Data structures
*/
/* the general SIO API */
typedef struct sio_st sio_t;
/* I/O vector entry (similar to POSIX struct iovec) for sio_{read,write}v() */
typdef struct sio_iovec_st
char *iov_base; /* Base address. */
size_t iov_len; /* Length. */
} sio_iovec_t;
typedef long sio_off_t;
typedef sio_uint8_t;
typedef sio_uint16_t;
typedef sio_uint32_t;
#define SIO_SEEK_SET
#define SIO_SEEK_CUR
#define SIO_SEEK_END
/*
* Values
*/
#define SIO_EOF (-1)
/*
* Stream Disciplines
*/
sio_disc_t *sio_disc_null (void);
sio_disc_t *sio_disc_anon (void);
sio_disc_t *sio_disc_fd (int fd, ...);
sio_disc_t *sio_disc_socket (int fd, int type /*tcp,udp*/, ...);
sio_disc_t *sio_disc_pipe (int fd, ...);
sio_disc_t *sio_disc_file (FILE *fp, ...);
sio_disc_t *sio_disc_url (const char *url, ...);
/*
* Stream Handling
*/
sio_t *sio_new (sio_disc_t *disc);
sio_t *sio_dup (sio_t *sio);
int sio_free (sio_t *sio);
/*
* I/O Operations
*/
sio_size_t sio_read (sio_t *sio, void *buf, size_t bytes);
sio_size_t sio_write (sio_t *sio, void *buf, size_t bytes);
sio_size_t sio_writev (sio_t *sio, const sio_iovec_t *iov, int iovcnt);
sio_size_t sio_writev (sio_t *sio, const sio_iovec_t *iov, int iovcnt);
sio_off_t sio_seek (sio_t *sio, sio_off_t offset, int type);
sio_size_t sio_move (sio_t *siow, sio_t *sior, int n, int rsc);
int sio_getc (sio_t *sio);
int sio_putc (sio_t *sio, int c);
int sio_nputc (sio_t *sio, int c, sio_size_t n);
int sio_ungetc (sio_t *sio, int c);
char *sio_getr (sio_t *sio, int rsc, int type);
sio_size_t *sio_putr (sio_t *sio, const char *str, int rsc);
/*
* Data Formatting
*/
sio_site_t sio_printf (sio_t *sio, const char *fmt, ...);
sio_site_t sio_vprintf (sio_t *sio, const char *fmt, va_list ap);
sio_site_t sio_scanf (sio_t *sio, const char *fmt, ...);
sio_site_t sio_vscanf (sio_t *sio, const char *fmt, va_list ap);
/*
* Buffering & Synchronization
*/
int sio_sync (sio_t *sio);
int sio_purge (sio_t *sio);
int sio_poll (sio_poll_t *pl, sio_size_t pn, sio_time_t timeout);
int sio_eof (sio_t *sio);
/*
* Stream Control
*/
long sio_ctrl (sio_t *sio, int cmd, ...);
int sio_top (sio_t *sio);
int sio_push (sio_t *sio, sio_t *top);
int sio_pop (sio_t *sio);
int sio_swap (sio_t *sio1, sio_t *sio2);
###########################################################################
mlelstv sez:
sio pipe:
- two chains of handlers for reading and writing
- "bucket brigade"-buffers, degenerates to static buffer(ptr,len)
- handler degenerates to (handle,read(2),write(2)) ?
dispatching:
- sio_strategy(chain)
h = chain;
while (h) {
if h->call(sio) // reader or writer
h = h->next;
else
h = h->prev;
}
? eof/error propagation -> global flags, (byte offset ?)
- sio_write(buf)
push sio->writebuffer, buf;
sio_strategy(sio->writers);
buf = pop sio->writebuffer;
- sio_read(bound)
sio->readbuffer->append(buf);
sio_strategy(sio->readers);
pop sio->readbuffer;
buffer:
- bytestream(policy for chunks ?)
- drop bytestream
- append bytes to bytestream (ptr, len) == copy
- prepend bytes to bytestream (ptr, len) == copy
- append static buffer to bytestream
- splice bytestream into new bytestream (offset, length, dest)
- truncate bytestream (offset, length) == splice but drop data
- traverse chunks of bytestream forward (offset, length)
- traverse chunks of bytestream reverse (offset, length)
- splice last chunk from bytestream (convenience function)
- splice first chunk from bytestream (convenience function)
- flatten chunks of bytestream into target buffer (convenience function)
- garbage collection ? pullup ? seek cache ?
- read consistency while traversing ?
bytestream, chunks, buffer
chunk memory allocator needed
buffer memory allocator ?
---
stdio "emulation"
fixed buffer <-> bytestream
EOF handling
###########################################################################
mlelstv sez:
ideas for error/eof/urgent-data handling
-> multiple assembly lines per stream
- too big ?
- synchronization between "bands" required
-> "urgent" pointer similar to TCP ?
+ trivial implementation
- only single condition for multiple causes
-> in-band multiplexing of labeled streams
+ synchonization is implicit
+ any number of conditions
- needs support in AL (coalescing/merging)
