Index: ossp-pkg/l2/TODO RCS File: /v/ossp/cvs/ossp-pkg/l2/TODO,v rcsdiff -q -kk '-r1.43' '-r1.44' -u '/v/ossp/cvs/ossp-pkg/l2/TODO,v' 2>/dev/null --- TODO 2001/11/16 19:40:55 1.43 +++ TODO 2001/11/30 17:12:54 1.44 @@ -1,357 +1,185 @@ +OSSP L2 TODO +============ -OSSP L2 -======= +New Channels ---------------------------------------------- +- l2_ch_bofh does something cool but is undocumented and has its + source mangled/scrambled ;) + +- l2_ch_asynch spawns a new process and communicates with + the stemming channels via shared memory. Might be the + way to implement autoflush as well, or rather with its + own shared memory and sub-process. + +- l2_ch_smart is a smart debug buffer channel which flushes + only if errors occur. + +- l2_ch_repeat prints 'last message repeated 100 times.' + +- l2_ch_action is similar to pipe channel, but executes a given + command conditionally of the incoming priority or a regex. + -> Might be implemented by combining a pcre, filter, and pipe (ms) + +- l2_ch_proxy sniffs a tcp connection and dumps its traffic to + standard out or to the next channel, allowing for parsing + through an unknown protocol. + +- l2_ch_nntp for administration through news. + +- l2_ch_snmp logs a stream to an SNMP listener. + +Existing Channels ----------------------------------------- +- l2_ch_socket bind and udp parameters need work. +- l2_ch_socket consider taking a fd at configuration. + +- l2_ch_filter should have a PCRE_CASELESS option. + +- l2_ch_pcre is too large and might benefit from a + smaller set of pattern matching logic. + +- l2_ch_syslog should inherit socket logic to directly implement + syslog communication, allowing for remote syslog operations. +- l2_ch_syslog should be trimmed to necessary functionality, not + necessarily everything that the system syslog(3) implements. + +- l2_ch_irc has a problem pinging its host to stay alive. + +- l2_ch_prefix produces implicitly two log messages if a + following buffer channel would not accumulate them. +- l2_ch_prefix should have printf style without strftime. +- l2_ch_prefix should apply __FILE__, __LINE__, (__FUNCTION__) + as options in case we are building in custom striptease mode. + +- l2_ch_file should optionally support file locking via fcntl(3). + +- l2_ch_pipe needs an overhaul once l2_ch_async is implemented. + Also needs a review for dangling descriptors. During configure + only checks existance of command in non-shell mode. Remove + hard coded 256 in exec arguments. +- l2_ch_pipe should use a standardized url like prg:/path/to/program. +- l2_ch_pipe might consider taking a fd at configuration. + +- l2_ch_buffer buffer is duplicated when the process forks. This + means the buffer has to be flushed in advance or the content + is dumped twice. If the buffer would remember the pid of the + last writer, it could discard the contents of the buffer when + the pid changes. This is because the parent retains the pid + and the buffer content while the child changes the pid and + discards the content. Should be an optionial feature. + -> Can be immediately implemented (rse) +- l2_ch_buffer needs a L2_OK and L2_OK_PASS consistency check +- l2_ch_buffer should not use different param types in alarm and setitimer(2). +- l2_ch_buffer should incorporate multiplexer logic to properly flush + when multiple buffers are in use. Can be implemented via a environment + level timer and a lowest common denominator alarm function. For example, + tmr1 9secs; tmr2 6secs; envtmr expires every lcd(9,6) = 3secs +- l2_ch_buffer should not implement an autoflush, rather L2 should + have a multiplexed timer object to serve timer requests from + incoming callbacks. + +Library-wide changes -------------------------------------- +- Replace l2_sockmon with netcat or peep. +- Add nonreentrant log method like mm. +- Add striptease target to Makefile. +- Provide a C++ implementation also. +- l2tool needs an [addr:]port option +- Adjust license to ISC/MIT/BSD. +- API access through the LogManager logm; +- Jump to a callback in case of error, throw/exit. +- Offer the option to reopen the logfile on each write. +- Optimize log operations from log(..) {} + to log(....); + +- Implicit level decision should default to >=, + then follow with <= + = + ; + * + +- '\r\n' Uebersetzung + Bei Input \r und \n wegstrippen, und von jedem Output Channel + entweder \r\n (smtp) oder nur \n wieder anhaengen lassen. Oder + als Kanaloption, wobei \r, \r\n, \n, gleich String, oder gar nix. + +- Spec-facility + To debug an application, sometimes it's overkill to log everything + at DEBUG level. I see an improvement when an additional facility + can be specified. Example: DEBUG/LMTP but don't care about NNTP in + the lmtp2nntp program. Possibly could be implemented as a second mask. + -> Needs more consideration before implementation should start (rse) + +- Prelogging + L2 should log even before it is initialized. Maybe the log function + should buffer everything as long as a NULL l2-context is passed and + if ever a non-NULL context is passed every remembered message should + be logged afterwards or if destroy/flush is executed with a NULL- + context it should print the buffered stuff to stderr. + -> Can be immediately implemented, but one has to be carefully here. + I want to think about this a little bit more in-depth. (rse) + +- Config file + Might be used with the previous prelogging principle in mind. + Namely, a file that describes a set of channel specs for one + or many configurations. Path is searched in the following order: + 1. /etc/liblog.conf + 2. (in ., .., ../..) + 3. $HOME/.liblog.conf + +- hook_write methods should receive a null-termined string + instead of buf+size, because some channels like syslog + otherwise have to rebuffer the message and append the + null terminator character. -Spec-Parsing: -- location tracking -- target=remote has to be currently after host=xxx because of - single-configuration procedure +- An optional syslog(3) compatible API for converting syslog-only based + applications (like sendmail) to (restricted) liblog-based applications. + -> Des ist schon implimentiert in fakesyslog oder? (ms) -- prefix channel produces implicitly two log messages if - a following buffer channel would not accumulate them. Hmmm.. +Spec-parsing ---------------------------------------------- +- implement location tracking -- bugfix l2tool and after it works 100% remove l2_test - in favor of l2tool and add instead a test shell script - which calls l2tool with various input specifications. +- target=remote has to be currently after host=xxx because of + single-configuration procedure -Channel-Only Revamping: +- bugfix l2tool and after it works 100% remove l2_test in + favor of l2tool and add instead a test shell script + which calls l2tool with various input specifications. + +Channel-Only Revamping ------------------------------------ +- l2_objects update - syscall override ala OSSP SA in l2_env_t -- l2_objects.fig update - perhaps rename l2_env to l2_ctx and l2_channel_ to just l2_ +- API cleanup for open semantics -- Perhaps we should later also write an l2_ch_bofh.c which - something very cool but is undocumented and has its source - mangled/scrambled ;) - -- Problem: OpenPKG fakesyslog ist nasty, because it doesn't provide - logging to multiple files or filter out some messages. Additionally - the app has to be restarted in order to reopen the logfile which - is nasty for MTAs like Postfix in case of very high loads (because - they start again processign the queue from scratch). What we need - is a new L2-based libsyslog.a which sends the stuff via Unix Domain - socket to an L2 daemon which in turn logs to targets via L2. - -- "smart debug buffer channel": - extra debug buffer, flush only if error occurs - -- "repeat channel": "last message repeated 100 times" - -- "\r\n" Problematik: Vorschlag: Bei Input \r und \n wegstrippen, - und von jedem Output Channel entweder \r\n (smtp) oder nur \n - wieder anhaengen lassen. - -- thl: log facility - To debug an application, sometimes it's overkill to log everything at - DEBUG level. I see an improvement when an additional facility can be - specified. Example: DEBUG/LMTP but don't care about NNTP in the - lmtp2nntp program. Possibly could be implemented as a second mask. - -> needs more consideration before implementation should start (rse) - -- thl: buffer fork() awareness - When the process forks, the buffer is duplicated. Currently this means - the buffer has to be flushed in advance or the content is dumped twice. - If the buffer would remember the pid of the last writer, it could - discard the contents of the buffer when the pid changes. This is because - the parent retains the pid and the buffer content while the child - changes the pid and discards the content. Should be an optionial feature. - -> can be immediately implemented (rse) - -- thl: prelog - I want to log everything even things that happen before L2 is - initialized. Complicated, i know. Maybe the log function should buffer - everything as long as a NULL l2-context is passed and if ever a non-NULL - context is passed every remembered message should be logged afterwards - or if destroy/ flush is executed with a NULL-context it should print the - buffered stuff to stderr. - -> can be immediately implemented, but one has to be carefully here. - Seems like I want to think about this a little bit more in-depth. (rse) - -o Another great rse idea, make a proxy server channel that sniffs a tcp - connection and dumps its traffic to standard out. Optionally, it could dump - this to the next channel, allowing for parsing through an unknown protocol. - -RSE: -- channel API cleanup: open semantics -- bind and udp parameters in socket channel -- prefix channels: - - printf style without strftime; -- l2tool [addr:]port -- autoflush via shared memory and sub-process? -- NNTP channel -- SNMP trap channel -- perhaps replace too large PCRE stuff with smaller pattern - matching stuff -- file channel should optionally support file locking - via fcntl(3). -- idea of an asynch channel, that spawns a new process and - communicates with the stemming channels via shared memory - -Lunchtime: -- Correct DNS resolve blocking problem by using a funky asynchronous DNS lib. - This leads to easy reimplementation of the prefix channel (asynchronous) - -MS: -- netcat can probably replace l2_sockmon, remove from build and add netcat -- pipe channel may need a big overhaul if we redesign - it around the asynch channel principle -- review pipe handler for dangling descriptors -- configure only checks existance of command in non-shell mode -- find alternative to exec arguments which is hard coded to 256 -- signal handler chaining, save old signal handler and call it after our own -- consider adding options such as PCRE_CASELESS to filter channel -- implement "action" channel, can be based on pipe channel -- correct problem with multiple buffer channels using the timer - can be solved by running a signal handler at the environment level, then - approximating the (multiple) user-chosen timer intervals by setting the - environment timer to expire at the time of the least common denominator of - the configured timer intervals - ie: tmr1 9secs; tmr2 6secs; envtmr will expire every lcd(9,6) = 3secs - alarm and setitimer methods are using different param types! Fixit. - review newly introduced timer code for L2_OK and L2_OK_PASS inconsistency -- solve problem with buffer (timer), irc (ping), and buffer (autoflush), by - creating sleep/ping threads. The disadvantage is that we must depend on a - pth installation, and force the parent app to be multithreaded. - Alternatively, we can spawn a management process in l2_stream_create(), who - owns management resources globally available to all channels. Or write the - l2 mini-protocol :-( -- split all timer server code out of L2 and into new multiplexed timer object - probably must use callbacks, unless user signal space is occupied - -ISSUES ------- - -o hook_write's should perhaps receive a nul-termined string - instead of buf+size, because syslog else has to re-buffer it - in order to append the nul terminator character. - -o Stream Members: channels static array - - consider dynamic - -o buffer - user needs to know how a buffer object behaves in - relation to up/downstream channels. When does it - pass its data to the next channel, when does it - erase, what happens to its data when it is over - written or flushed... - -o buffer timer - How on earth to use a C-style exception handler to flush - our buffer? We can't directly call ANY function from an - exception state, so this might not work at all. - - Depends on SIGALARM, only one handler of which may exist. - A more robust implementation would not use such a precious - resource, and guard against signal collision with other channels. - -o syslog - many options need docu, and we should mention to - the user that more info is found in the man page - for syslog(), because after all that is what is - doing all the work in our implementation. Also, - can we really properly document these features - if they change from one system's syslog to the next? - -o errors - when a channel fails during an operation, how - does it report this? How should a user interpret - the error message or other data? Do we need more - accurate or detailed error messages in the channel - code? When a channel fails, does it continue - passing data on to downstream channels? Is it - corrupt data? - -o Syslog Kanal - - Trim down to what will be used, right now the - channel supports ALL functionality through syslog(3) - -BRAINSTORMING -------------- - -Braindump: -- debugging is special case of logging -- tracing is special case of debugging - -Channel Handler Configuration: -o l2_handler_null - - no configuration at all -o l2_handler_fd - - mode="unix|stdio" - - fd=int|FILE* -o l2_handler_file - - mode="unix|stdio" - - path=char* - - append="yes|no" -o l2_handler_pipe - - url="prg:/path/to/program" - - fd=int -o l2_handler_socket - - url="tcp://hostname:port" - - fd=int -o l2_handler_syslog - - ident=char* - - should have its own logic - and not use unix lib syslog() - thus able to write to a remote - syslog daemon -o l2_handler_filter - - pattern=char* -o l2_handler_prefix - - prefix=char* -o l2_handler_buffer - - size=size_t -o all output channels - - should they have downstream, or be true endpoints? -o l2_ch_socket - - write should handle partial send() - thus check the return of send -o l2_ch_buffer - - write() must implicitly flush() when incoming - data is larger than remaining buffer capacity -o l2_ch_action - - new action channnel could accept a regexp or L2_LEVEL as configuration, - and then run a callback function or exec a binary depending on incoming - l2 stream messages. For example, lmtp2nntp could conditionally run code - via the new action channel to monitor spam or alert an admin about a - flawed OR CURIOUs LMTP header. However, the action channel is also - appealling to other applications that don't run as daemons or at the - system level. - - l2_channel_config(l2Act, "regexp,run", "(nobody|cz)", "callback"); - - l2_channel_config(l2Act, "funcptr", pfnAddtcpwrap); - -License: -- ISC/MIT/BSD - -Sprache: -- C++ -- C - -Aufbau: -1. Layer C++ API log.hpp, log.cpp -2. Layer C API log.h log.c -3. Layer C Backend backend.h backend.c -- "make striptease" - -- optimierung: -log(..) -{ -} -: -log(....); -: - -API Levels: -- PANIC (-> LOG_EMERG) -- CRITICAL (-> LOG_CRIT) -- ERROR (-> LOG_ERR) -- WARNING -- NOTICE -- INFO -- TRACE (-> LOG_DEBUG) -- DEBUG (-> LOG_DEBUG) -- ALERT - -Level Entscheidungen: ->= (default) -<= -= -; -* - -Backend Channels: -1 Level -> N Channels -- file (append) -- program (stdin) -- syslog -- stderr/stdout -- null (discard, nicht nur /dev/null) -- filedescriptor (escape/ext) -- callback function - -Log Messages: -- raw -- optional prefixes (inclusive order): - string - facility - level - timestamp - pid - (tid) -- errno (like syslog %m) -- eigene %{foo}x mit callback function mit context -- automatisch: number -> string mapping (fuer error strings) -- __FILE__, __LINE__, (__FUNCTION__) - -Configuration: -- ueber C/C++ API -- zusaetzlich Config-File - 1. /etc/liblog.conf - 2. (in ., .., ../..) - 3. $HOME/.liblog.conf - -- !debug -> !code - -API C (ala MM): -- reentrant: log_xxx -- non-reentrant: Log_xxx - -Message Filtering/Masking: -- facility und/oder levels und/oder wildcard pattern - -API Using: -- C++: - LogManager logm; - logm.debug1("test"); - logm.configure(" - -- C: - log lh; - lh = log_init(LOG_CFGFILE|LOG_CFGPARENT|LOG_XXX|..., "foo" (=facility)); - log_configure(lh, "foo", LOG_WARN|LOG_LESSER, null); - log_cb(lh, "x", func, ctx); - int func(void *ctx, char *str, ...); - log_msg(lh, LOG_WARN, "..%{foo}x %s...%E..", cp); - log_dbg(lh, "..%{foo}x %s...%E..", cp); - log_kill(lh); - -- Buffered I/O: - fuer manche channels non-buffered (debug, errors) - fuer manche andere aber buffered (access log, performance) - loesung: I/O ueber callbacks (3x: open, write, close) z.B. RRDTool - -- Varargs: - log ist nur wrapper fuer vlog - -- Error Handling: - o log kein Return Code - o aber error callback function (dadrin in C++: throw, in C: exit) - -- Newline Handling: - option fuer channel: \r, \r\n, \n oder gleich string - und moeglichkeit gar nix (string="") - -- Perhaps: - optionally reopen logfile on each write - -- An optional syslog(3) compatible API for converting syslog-only based - applications (like sendmail) to (restricted) liblog-based applications. - -- Ein Wort noch zu variablen Argumentlisten in cpp-Makros: gcc - unterstützt dies in in der GNU- und der C99-Ausführung. Das heißt, - der "..." Parameter kann im Makro respektive über "args" und über - "__VA_ARGS__" angesprochen werden. Wichtig ist dabei, daß "..." - nicht leer sein -- also kein Argument enthalten -- darf, da sonst - der Präprocessor an einem eventuell vorhandenen Komma scheitert. - Dies kann beim gcc durch Voranstellen von "##" vor dem "__VA_ARGS__" - umgangen werden. Ouch. - Beide Erweiterungen sind derzeit nicht aktiv, wenn mit -ansi - compiliert wird. Explizit anschalten läßt sich die standard-konforme - Erweiterung über "-std=c9x", bzw. "-std=c99" bei neueren gccs. +Related Projects ------------------------------------------ +- Custom libsyslog + OpenPKG fakesyslog is nasty, because it doesn't provide logging + to multiple files or filter out some messages. Additionally the + app has to be restarted in order to reopen the logfile which is + nasty for MTAs like Postfix in case of very high loads (because + they start again processign the queue from scratch). What we need + is a new L2-based libsyslog.a which sends the stuff via Unix Domain + socket to an L2 daemon which in turn logs to targets via L2. + +Documentation --------------------------------------------- +- l2_ch_buffer + How the buffer object behaves in relation to up/downstream + channels. When does it pass its data to the next channel, + when does it erase, what happens to its data when it is + over written or flushed. + +- l2_ch_syslog + Describe the options, and mention that more info is found + in the man page for syslog(3). Identify features that + change from one system to the next. + +- l2 errors + Describe how an l2 channel reports errors when it fails during + an operation, and how a user should interpret the error message? + Describe whether a channel continues passing data on downstream + if it somehow fails, and if the data will be corrupt. -High-Level Configuration Interface +High-level configuration interface ================================== - Config-File (OSSP): - logging { channel 0 null ALL; @@ -373,11 +201,9 @@ log access 0; log stat 1; - }; Alternative: - log access { error: syslog; prefix -> { @@ -388,7 +214,6 @@ } Command-Line (lmtp2nntp): - $ lmtp2nntp -l 'INFO:' { prefix(prefix="%Y-%m-%d/%H:%M:%S %L") ->{ buffer(size=4k) @@ -398,3 +223,73 @@ error:syslog } +Brainstorming --------------------------------------------- +- Debugging can be implemented as a special case of logging +- Tracing can be implemented as a special case of debugging + + +*** *** *** *** *** *** Out of date *** *** *** *** *** *** +*** *** *** *** *** *** Out of date *** *** *** *** *** *** +*** *** *** *** *** *** Out of date *** *** *** *** *** *** + + +Backend channels ------------------------------------------ +- Level -> N Channels +- file (append) +- program (stdin) +- syslog +- stderr/stdout +- null (discard, nicht nur /dev/null) +- filedescriptor (escape/ext) +- callback function + +Aufbau ---------------------------------------------------- +Layer C++ API log.hpp, log.cpp +Layer C API log.h log.c +Layer C Backend backend.h backend.c + +- Ein Wort noch zu variablen Argumentlisten in cpp-Makros: gcc + unterstützt dies in in der GNU- und der C99-Ausführung. Das heißt, + der "..." Parameter kann im Makro respektive über "args" und über + "__VA_ARGS__" angesprochen werden. Wichtig ist dabei, daß "..." + nicht leer sein -- also kein Argument enthalten -- darf, da sonst + der Präprocessor an einem eventuell vorhandenen Komma scheitert. + Dies kann beim gcc durch Voranstellen von "##" vor dem "__VA_ARGS__" + umgangen werden. Ouch. Beide Erweiterungen sind derzeit nicht aktiv, + wenn mit -ansi compiliert wird. Explizit anschalten läßt sich die + standard-konforme Erweiterung über "-std=c9x", bzw. "-std=c99" bei + neueren gccs. + +Log Messages: +- raw +- optional prefixes (inclusive of order) + string + facility + level + timestamp + pid + (tid) +- errno (like syslog %m) +- eigene %{foo}x mit callback function mit context +- automatisch number -> string mapping (fuer error strings) +- !debug -> !code + +To be deleted --------------------------------------------- +- C: + log lh; + lh = log_init(LOG_CFGFILE|LOG_CFGPARENT|LOG_XXX|..., "foo" (=facility)); + log_configure(lh, "foo", LOG_WARN|LOG_LESSER, null); + log_cb(lh, "x", func, ctx); + int func(void *ctx, char *str, ...); + log_msg(lh, LOG_WARN, "..%{foo}x %s...%E..", cp); + log_dbg(lh, "..%{foo}x %s...%E..", cp); + log_kill(lh); + +- Buffered I/O: + fuer manche channels non-buffered (debug, errors) + fuer manche andere aber buffered (access log, performance) + loesung: I/O ueber callbacks (3x: open, write, close) z.B. RRDTool + +- Varargs: + log ist nur wrapper fuer vlog +