--- TODO 2001/11/16 19:40:55 1.43
+++ TODO 2001/11/30 17:12:54 1.44
@@ -1,357 +1,185 @@
+OSSP L2 TODO
+============
-OSSP L2
-=======
+New Channels ----------------------------------------------
+- l2_ch_bofh does something cool but is undocumented and has its
+ source mangled/scrambled ;)
+
+- l2_ch_asynch spawns a new process and communicates with
+ the stemming channels via shared memory. Might be the
+ way to implement autoflush as well, or rather with its
+ own shared memory and sub-process.
+
+- l2_ch_smart is a smart debug buffer channel which flushes
+ only if errors occur.
+
+- l2_ch_repeat prints 'last message repeated 100 times.'
+
+- l2_ch_action is similar to pipe channel, but executes a given
+ command conditionally of the incoming priority or a regex.
+ -> Might be implemented by combining a pcre, filter, and pipe (ms)
+
+- l2_ch_proxy sniffs a tcp connection and dumps its traffic to
+ standard out or to the next channel, allowing for parsing
+ through an unknown protocol.
+
+- l2_ch_nntp for administration through news.
+
+- l2_ch_snmp logs a stream to an SNMP listener.
+
+Existing Channels -----------------------------------------
+- l2_ch_socket bind and udp parameters need work.
+- l2_ch_socket consider taking a fd at configuration.
+
+- l2_ch_filter should have a PCRE_CASELESS option.
+
+- l2_ch_pcre is too large and might benefit from a
+ smaller set of pattern matching logic.
+
+- l2_ch_syslog should inherit socket logic to directly implement
+ syslog communication, allowing for remote syslog operations.
+- l2_ch_syslog should be trimmed to necessary functionality, not
+ necessarily everything that the system syslog(3) implements.
+
+- l2_ch_irc has a problem pinging its host to stay alive.
+
+- l2_ch_prefix produces implicitly two log messages if a
+ following buffer channel would not accumulate them.
+- l2_ch_prefix should have printf style without strftime.
+- l2_ch_prefix should apply __FILE__, __LINE__, (__FUNCTION__)
+ as options in case we are building in custom striptease mode.
+
+- l2_ch_file should optionally support file locking via fcntl(3).
+
+- l2_ch_pipe needs an overhaul once l2_ch_async is implemented.
+ Also needs a review for dangling descriptors. During configure
+ only checks existance of command in non-shell mode. Remove
+ hard coded 256 in exec arguments.
+- l2_ch_pipe should use a standardized url like prg:/path/to/program.
+- l2_ch_pipe might consider taking a fd at configuration.
+
+- l2_ch_buffer buffer is duplicated when the process forks. This
+ means the buffer has to be flushed in advance or the content
+ is dumped twice. If the buffer would remember the pid of the
+ last writer, it could discard the contents of the buffer when
+ the pid changes. This is because the parent retains the pid
+ and the buffer content while the child changes the pid and
+ discards the content. Should be an optionial feature.
+ -> Can be immediately implemented (rse)
+- l2_ch_buffer needs a L2_OK and L2_OK_PASS consistency check
+- l2_ch_buffer should not use different param types in alarm and setitimer(2).
+- l2_ch_buffer should incorporate multiplexer logic to properly flush
+ when multiple buffers are in use. Can be implemented via a environment
+ level timer and a lowest common denominator alarm function. For example,
+ tmr1 9secs; tmr2 6secs; envtmr expires every lcd(9,6) = 3secs
+- l2_ch_buffer should not implement an autoflush, rather L2 should
+ have a multiplexed timer object to serve timer requests from
+ incoming callbacks.
+
+Library-wide changes --------------------------------------
+- Replace l2_sockmon with netcat or peep.
+- Add nonreentrant log method like mm.
+- Add striptease target to Makefile.
+- Provide a C++ implementation also.
+- l2tool needs an [addr:]port option
+- Adjust license to ISC/MIT/BSD.
+- API access through the LogManager logm;
+- Jump to a callback in case of error, throw/exit.
+- Offer the option to reopen the logfile on each write.
+- Optimize log operations from log(..) {}
+ to log(....);
+
+- Implicit level decision should default to >=,
+ then follow with <=
+ =
+ ;
+ *
+
+- '\r\n' Uebersetzung
+ Bei Input \r und \n wegstrippen, und von jedem Output Channel
+ entweder \r\n (smtp) oder nur \n wieder anhaengen lassen. Oder
+ als Kanaloption, wobei \r, \r\n, \n, gleich String, oder gar nix.
+
+- Spec-facility
+ To debug an application, sometimes it's overkill to log everything
+ at DEBUG level. I see an improvement when an additional facility
+ can be specified. Example: DEBUG/LMTP but don't care about NNTP in
+ the lmtp2nntp program. Possibly could be implemented as a second mask.
+ -> Needs more consideration before implementation should start (rse)
+
+- Prelogging
+ L2 should log even before it is initialized. Maybe the log function
+ should buffer everything as long as a NULL l2-context is passed and
+ if ever a non-NULL context is passed every remembered message should
+ be logged afterwards or if destroy/flush is executed with a NULL-
+ context it should print the buffered stuff to stderr.
+ -> Can be immediately implemented, but one has to be carefully here.
+ I want to think about this a little bit more in-depth. (rse)
+
+- Config file
+ Might be used with the previous prelogging principle in mind.
+ Namely, a file that describes a set of channel specs for one
+ or many configurations. Path is searched in the following order:
+ 1. /etc/liblog.conf
+ 2. (in ., .., ../..)
+ 3. $HOME/.liblog.conf
+
+- hook_write methods should receive a null-termined string
+ instead of buf+size, because some channels like syslog
+ otherwise have to rebuffer the message and append the
+ null terminator character.
-Spec-Parsing:
-- location tracking
-- target=remote has to be currently after host=xxx because of
- single-configuration procedure
+- An optional syslog(3) compatible API for converting syslog-only based
+ applications (like sendmail) to (restricted) liblog-based applications.
+ -> Des ist schon implimentiert in fakesyslog oder? (ms)
-- prefix channel produces implicitly two log messages if
- a following buffer channel would not accumulate them. Hmmm..
+Spec-parsing ----------------------------------------------
+- implement location tracking
-- bugfix l2tool and after it works 100% remove l2_test
- in favor of l2tool and add instead a test shell script
- which calls l2tool with various input specifications.
+- target=remote has to be currently after host=xxx because of
+ single-configuration procedure
-Channel-Only Revamping:
+- bugfix l2tool and after it works 100% remove l2_test in
+ favor of l2tool and add instead a test shell script
+ which calls l2tool with various input specifications.
+
+Channel-Only Revamping ------------------------------------
+- l2_objects update
- syscall override ala OSSP SA in l2_env_t
-- l2_objects.fig update
- perhaps rename l2_env to l2_ctx and l2_channel_ to just l2_
+- API cleanup for open semantics
-- Perhaps we should later also write an l2_ch_bofh.c which
- something very cool but is undocumented and has its source
- mangled/scrambled ;)
-
-- Problem: OpenPKG fakesyslog ist nasty, because it doesn't provide
- logging to multiple files or filter out some messages. Additionally
- the app has to be restarted in order to reopen the logfile which
- is nasty for MTAs like Postfix in case of very high loads (because
- they start again processign the queue from scratch). What we need
- is a new L2-based libsyslog.a which sends the stuff via Unix Domain
- socket to an L2 daemon which in turn logs to targets via L2.
-
-- "smart debug buffer channel":
- extra debug buffer, flush only if error occurs
-
-- "repeat channel": "last message repeated 100 times"
-
-- "\r\n" Problematik: Vorschlag: Bei Input \r und \n wegstrippen,
- und von jedem Output Channel entweder \r\n (smtp) oder nur \n
- wieder anhaengen lassen.
-
-- thl: log facility
- To debug an application, sometimes it's overkill to log everything at
- DEBUG level. I see an improvement when an additional facility can be
- specified. Example: DEBUG/LMTP but don't care about NNTP in the
- lmtp2nntp program. Possibly could be implemented as a second mask.
- -> needs more consideration before implementation should start (rse)
-
-- thl: buffer fork() awareness
- When the process forks, the buffer is duplicated. Currently this means
- the buffer has to be flushed in advance or the content is dumped twice.
- If the buffer would remember the pid of the last writer, it could
- discard the contents of the buffer when the pid changes. This is because
- the parent retains the pid and the buffer content while the child
- changes the pid and discards the content. Should be an optionial feature.
- -> can be immediately implemented (rse)
-
-- thl: prelog
- I want to log everything even things that happen before L2 is
- initialized. Complicated, i know. Maybe the log function should buffer
- everything as long as a NULL l2-context is passed and if ever a non-NULL
- context is passed every remembered message should be logged afterwards
- or if destroy/ flush is executed with a NULL-context it should print the
- buffered stuff to stderr.
- -> can be immediately implemented, but one has to be carefully here.
- Seems like I want to think about this a little bit more in-depth. (rse)
-
-o Another great rse idea, make a proxy server channel that sniffs a tcp
- connection and dumps its traffic to standard out. Optionally, it could dump
- this to the next channel, allowing for parsing through an unknown protocol.
-
-RSE:
-- channel API cleanup: open semantics
-- bind and udp parameters in socket channel
-- prefix channels:
- - printf style without strftime;
-- l2tool [addr:]port
-- autoflush via shared memory and sub-process?
-- NNTP channel
-- SNMP trap channel
-- perhaps replace too large PCRE stuff with smaller pattern
- matching stuff
-- file channel should optionally support file locking
- via fcntl(3).
-- idea of an asynch channel, that spawns a new process and
- communicates with the stemming channels via shared memory
-
-Lunchtime:
-- Correct DNS resolve blocking problem by using a funky asynchronous DNS lib.
- This leads to easy reimplementation of the prefix channel (asynchronous)
-
-MS:
-- netcat can probably replace l2_sockmon, remove from build and add netcat
-- pipe channel may need a big overhaul if we redesign
- it around the asynch channel principle
-- review pipe handler for dangling descriptors
-- configure only checks existance of command in non-shell mode
-- find alternative to exec arguments which is hard coded to 256
-- signal handler chaining, save old signal handler and call it after our own
-- consider adding options such as PCRE_CASELESS to filter channel
-- implement "action" channel, can be based on pipe channel
-- correct problem with multiple buffer channels using the timer
- can be solved by running a signal handler at the environment level, then
- approximating the (multiple) user-chosen timer intervals by setting the
- environment timer to expire at the time of the least common denominator of
- the configured timer intervals
- ie: tmr1 9secs; tmr2 6secs; envtmr will expire every lcd(9,6) = 3secs
- alarm and setitimer methods are using different param types! Fixit.
- review newly introduced timer code for L2_OK and L2_OK_PASS inconsistency
-- solve problem with buffer (timer), irc (ping), and buffer (autoflush), by
- creating sleep/ping threads. The disadvantage is that we must depend on a
- pth installation, and force the parent app to be multithreaded.
- Alternatively, we can spawn a management process in l2_stream_create(), who
- owns management resources globally available to all channels. Or write the
- l2 mini-protocol :-(
-- split all timer server code out of L2 and into new multiplexed timer object
- probably must use callbacks, unless user signal space is occupied
-
-ISSUES
-------
-
-o hook_write's should perhaps receive a nul-termined string
- instead of buf+size, because syslog else has to re-buffer it
- in order to append the nul terminator character.
-
-o Stream Members: channels static array
- - consider dynamic
-
-o buffer
- user needs to know how a buffer object behaves in
- relation to up/downstream channels. When does it
- pass its data to the next channel, when does it
- erase, what happens to its data when it is over
- written or flushed...
-
-o buffer timer
- How on earth to use a C-style exception handler to flush
- our buffer? We can't directly call ANY function from an
- exception state, so this might not work at all.
-
- Depends on SIGALARM, only one handler of which may exist.
- A more robust implementation would not use such a precious
- resource, and guard against signal collision with other channels.
-
-o syslog
- many options need docu, and we should mention to
- the user that more info is found in the man page
- for syslog(), because after all that is what is
- doing all the work in our implementation. Also,
- can we really properly document these features
- if they change from one system's syslog to the next?
-
-o errors
- when a channel fails during an operation, how
- does it report this? How should a user interpret
- the error message or other data? Do we need more
- accurate or detailed error messages in the channel
- code? When a channel fails, does it continue
- passing data on to downstream channels? Is it
- corrupt data?
-
-o Syslog Kanal
- - Trim down to what will be used, right now the
- channel supports ALL functionality through syslog(3)
-
-BRAINSTORMING
--------------
-
-Braindump:
-- debugging is special case of logging
-- tracing is special case of debugging
-
-Channel Handler Configuration:
-o l2_handler_null
- - no configuration at all
-o l2_handler_fd
- - mode="unix|stdio"
- - fd=int|FILE*
-o l2_handler_file
- - mode="unix|stdio"
- - path=char*
- - append="yes|no"
-o l2_handler_pipe
- - url="prg:/path/to/program"
- - fd=int
-o l2_handler_socket
- - url="tcp://hostname:port"
- - fd=int
-o l2_handler_syslog
- - ident=char*
- - should have its own logic
- and not use unix lib syslog()
- thus able to write to a remote
- syslog daemon
-o l2_handler_filter
- - pattern=char*
-o l2_handler_prefix
- - prefix=char*
-o l2_handler_buffer
- - size=size_t
-o all output channels
- - should they have downstream, or be true endpoints?
-o l2_ch_socket
- - write should handle partial send()
- thus check the return of send
-o l2_ch_buffer
- - write() must implicitly flush() when incoming
- data is larger than remaining buffer capacity
-o l2_ch_action
- - new action channnel could accept a regexp or L2_LEVEL as configuration,
- and then run a callback function or exec a binary depending on incoming
- l2 stream messages. For example, lmtp2nntp could conditionally run code
- via the new action channel to monitor spam or alert an admin about a
- flawed OR CURIOUs LMTP header. However, the action channel is also
- appealling to other applications that don't run as daemons or at the
- system level.
- - l2_channel_config(l2Act, "regexp,run", "(nobody|cz)", "callback");
- - l2_channel_config(l2Act, "funcptr", pfnAddtcpwrap);
-
-License:
-- ISC/MIT/BSD
-
-Sprache:
-- C++
-- C
-
-Aufbau:
-1. Layer C++ API log.hpp, log.cpp
-2. Layer C API log.h log.c
-3. Layer C Backend backend.h backend.c
-- "make striptease"
-
-- optimierung:
-log(..)
-{
-}
-:
-log(....);
-:
-
-API Levels:
-- PANIC (-> LOG_EMERG)
-- CRITICAL (-> LOG_CRIT)
-- ERROR (-> LOG_ERR)
-- WARNING
-- NOTICE
-- INFO
-- TRACE (-> LOG_DEBUG)
-- DEBUG (-> LOG_DEBUG)
-- ALERT
-
-Level Entscheidungen:
->= (default)
-<=
-=
-;
-*
-
-Backend Channels:
-1 Level -> N Channels
-- file (append)
-- program (stdin)
-- syslog
-- stderr/stdout
-- null (discard, nicht nur /dev/null)
-- filedescriptor (escape/ext)
-- callback function
-
-Log Messages:
-- raw
-- optional prefixes (inclusive order):
- string
- facility
- level
- timestamp
- pid
- (tid)
-- errno (like syslog %m)
-- eigene %{foo}x mit callback function mit context
-- automatisch: number -> string mapping (fuer error strings)
-- __FILE__, __LINE__, (__FUNCTION__)
-
-Configuration:
-- ueber C/C++ API
-- zusaetzlich Config-File
- 1. /etc/liblog.conf
- 2. (in ., .., ../..)
- 3. $HOME/.liblog.conf
-
-- !debug -> !code
-
-API C (ala MM):
-- reentrant: log_xxx
-- non-reentrant: Log_xxx
-
-Message Filtering/Masking:
-- facility und/oder levels und/oder wildcard pattern
-
-API Using:
-- C++:
- LogManager logm;
- logm.debug1("test");
- logm.configure("
-
-- C:
- log lh;
- lh = log_init(LOG_CFGFILE|LOG_CFGPARENT|LOG_XXX|..., "foo" (=facility));
- log_configure(lh, "foo", LOG_WARN|LOG_LESSER, null);
- log_cb(lh, "x", func, ctx);
- int func(void *ctx, char *str, ...);
- log_msg(lh, LOG_WARN, "..%{foo}x %s...%E..", cp);
- log_dbg(lh, "..%{foo}x %s...%E..", cp);
- log_kill(lh);
-
-- Buffered I/O:
- fuer manche channels non-buffered (debug, errors)
- fuer manche andere aber buffered (access log, performance)
- loesung: I/O ueber callbacks (3x: open, write, close) z.B. RRDTool
-
-- Varargs:
- log ist nur wrapper fuer vlog
-
-- Error Handling:
- o log kein Return Code
- o aber error callback function (dadrin in C++: throw, in C: exit)
-
-- Newline Handling:
- option fuer channel: \r, \r\n, \n oder gleich string
- und moeglichkeit gar nix (string="")
-
-- Perhaps:
- optionally reopen logfile on each write
-
-- An optional syslog(3) compatible API for converting syslog-only based
- applications (like sendmail) to (restricted) liblog-based applications.
-
-- Ein Wort noch zu variablen Argumentlisten in cpp-Makros: gcc
- unterstützt dies in in der GNU- und der C99-Ausführung. Das heißt,
- der "..." Parameter kann im Makro respektive über "args" und über
- "__VA_ARGS__" angesprochen werden. Wichtig ist dabei, daß "..."
- nicht leer sein -- also kein Argument enthalten -- darf, da sonst
- der Präprocessor an einem eventuell vorhandenen Komma scheitert.
- Dies kann beim gcc durch Voranstellen von "##" vor dem "__VA_ARGS__"
- umgangen werden. Ouch.
- Beide Erweiterungen sind derzeit nicht aktiv, wenn mit -ansi
- compiliert wird. Explizit anschalten läßt sich die standard-konforme
- Erweiterung über "-std=c9x", bzw. "-std=c99" bei neueren gccs.
+Related Projects ------------------------------------------
+- Custom libsyslog
+ OpenPKG fakesyslog is nasty, because it doesn't provide logging
+ to multiple files or filter out some messages. Additionally the
+ app has to be restarted in order to reopen the logfile which is
+ nasty for MTAs like Postfix in case of very high loads (because
+ they start again processign the queue from scratch). What we need
+ is a new L2-based libsyslog.a which sends the stuff via Unix Domain
+ socket to an L2 daemon which in turn logs to targets via L2.
+
+Documentation ---------------------------------------------
+- l2_ch_buffer
+ How the buffer object behaves in relation to up/downstream
+ channels. When does it pass its data to the next channel,
+ when does it erase, what happens to its data when it is
+ over written or flushed.
+
+- l2_ch_syslog
+ Describe the options, and mention that more info is found
+ in the man page for syslog(3). Identify features that
+ change from one system to the next.
+
+- l2 errors
+ Describe how an l2 channel reports errors when it fails during
+ an operation, and how a user should interpret the error message?
+ Describe whether a channel continues passing data on downstream
+ if it somehow fails, and if the data will be corrupt.
-High-Level Configuration Interface
+High-level configuration interface
==================================
-
Config-File (OSSP):
-
logging {
channel 0 null ALL;
@@ -373,11 +201,9 @@
log access 0;
log stat 1;
-
};
Alternative:
-
log access {
error: syslog;
prefix -> {
@@ -388,7 +214,6 @@
}
Command-Line (lmtp2nntp):
-
$ lmtp2nntp -l 'INFO:'
{ prefix(prefix="%Y-%m-%d/%H:%M:%S %L")
->{ buffer(size=4k)
@@ -398,3 +223,73 @@
error:syslog
}
+Brainstorming ---------------------------------------------
+- Debugging can be implemented as a special case of logging
+- Tracing can be implemented as a special case of debugging
+
+
+*** *** *** *** *** *** Out of date *** *** *** *** *** ***
+*** *** *** *** *** *** Out of date *** *** *** *** *** ***
+*** *** *** *** *** *** Out of date *** *** *** *** *** ***
+
+
+Backend channels ------------------------------------------
+- Level -> N Channels
+- file (append)
+- program (stdin)
+- syslog
+- stderr/stdout
+- null (discard, nicht nur /dev/null)
+- filedescriptor (escape/ext)
+- callback function
+
+Aufbau ----------------------------------------------------
+Layer C++ API log.hpp, log.cpp
+Layer C API log.h log.c
+Layer C Backend backend.h backend.c
+
+- Ein Wort noch zu variablen Argumentlisten in cpp-Makros: gcc
+ unterstützt dies in in der GNU- und der C99-Ausführung. Das heißt,
+ der "..." Parameter kann im Makro respektive über "args" und über
+ "__VA_ARGS__" angesprochen werden. Wichtig ist dabei, daß "..."
+ nicht leer sein -- also kein Argument enthalten -- darf, da sonst
+ der Präprocessor an einem eventuell vorhandenen Komma scheitert.
+ Dies kann beim gcc durch Voranstellen von "##" vor dem "__VA_ARGS__"
+ umgangen werden. Ouch. Beide Erweiterungen sind derzeit nicht aktiv,
+ wenn mit -ansi compiliert wird. Explizit anschalten läßt sich die
+ standard-konforme Erweiterung über "-std=c9x", bzw. "-std=c99" bei
+ neueren gccs.
+
+Log Messages:
+- raw
+- optional prefixes (inclusive of order)
+ string
+ facility
+ level
+ timestamp
+ pid
+ (tid)
+- errno (like syslog %m)
+- eigene %{foo}x mit callback function mit context
+- automatisch number -> string mapping (fuer error strings)
+- !debug -> !code
+
+To be deleted ---------------------------------------------
+- C:
+ log lh;
+ lh = log_init(LOG_CFGFILE|LOG_CFGPARENT|LOG_XXX|..., "foo" (=facility));
+ log_configure(lh, "foo", LOG_WARN|LOG_LESSER, null);
+ log_cb(lh, "x", func, ctx);
+ int func(void *ctx, char *str, ...);
+ log_msg(lh, LOG_WARN, "..%{foo}x %s...%E..", cp);
+ log_dbg(lh, "..%{foo}x %s...%E..", cp);
+ log_kill(lh);
+
+- Buffered I/O:
+ fuer manche channels non-buffered (debug, errors)
+ fuer manche andere aber buffered (access log, performance)
+ loesung: I/O ueber callbacks (3x: open, write, close) z.B. RRDTool
+
+- Varargs:
+ log ist nur wrapper fuer vlog
+
|
