OSSP CVS Repository

ossp - Check-in [1394]
Not logged in
[Honeypot]  [Browse]  [Home]  [Login]  [Reports
[Search]  [Ticket]  [Timeline
  [Patchset]  [Tagging/Branching

Check-in Number: 1394
Date: 2001-Nov-30 18:12:54 (local)
2001-Nov-30 17:12:54 (UTC)
User:ms
Branch:
Comment: Reorganization and clarification.
Tickets:
Inspections:
Files:
ossp-pkg/l2/TODO      1.43 -> 1.44     241 inserted, 346 deleted

ossp-pkg/l2/TODO 1.43 -> 1.44

--- TODO 2001/11/16 19:40:55     1.43
+++ TODO 2001/11/30 17:12:54     1.44
@@ -1,357 +1,185 @@
+OSSP L2 TODO
+============
 
-OSSP L2
-=======
+New Channels ----------------------------------------------
+- l2_ch_bofh does something cool but is undocumented and has its
+    source mangled/scrambled ;)
+
+- l2_ch_asynch spawns a new process and communicates with
+    the stemming channels via shared memory. Might be the
+    way to implement autoflush as well, or rather with its
+    own shared memory and sub-process.
+
+- l2_ch_smart is a smart debug buffer channel which flushes
+    only if errors occur.
+
+- l2_ch_repeat prints 'last message repeated 100 times.'
+
+- l2_ch_action is similar to pipe channel, but executes a given
+    command conditionally of the incoming priority or a regex.
+    -> Might be implemented by combining a pcre, filter, and pipe (ms)
+
+- l2_ch_proxy sniffs a tcp connection and dumps its traffic to
+    standard out or to the next channel, allowing for parsing
+    through an unknown protocol.
+
+- l2_ch_nntp for administration through news.
+
+- l2_ch_snmp logs a stream to an SNMP listener.
+
+Existing Channels -----------------------------------------
+- l2_ch_socket bind and udp parameters need work.
+- l2_ch_socket consider taking a fd at configuration.
+
+- l2_ch_filter should have a PCRE_CASELESS option.
+
+- l2_ch_pcre is too large and might benefit from a
+    smaller set of pattern matching logic.
+
+- l2_ch_syslog should inherit socket logic to directly implement
+    syslog communication, allowing for remote syslog operations.
+- l2_ch_syslog should be trimmed to necessary functionality, not
+    necessarily everything that the system syslog(3) implements.
+
+- l2_ch_irc has a problem pinging its host to stay alive.
+
+- l2_ch_prefix produces implicitly two log messages if a
+    following buffer channel would not accumulate them.
+- l2_ch_prefix should have printf style without strftime.
+- l2_ch_prefix should apply __FILE__, __LINE__, (__FUNCTION__)
+    as options in case we are building in custom striptease mode.
+
+- l2_ch_file should optionally support file locking via fcntl(3).
+
+- l2_ch_pipe needs an overhaul once l2_ch_async is implemented.
+    Also needs a review for dangling descriptors. During configure
+    only checks existance of command in non-shell mode. Remove
+    hard coded 256 in exec arguments.
+- l2_ch_pipe should use a standardized url like prg:/path/to/program.
+- l2_ch_pipe might consider taking a fd at configuration.
+
+- l2_ch_buffer buffer is duplicated when the process forks. This
+    means the buffer has to be flushed in advance or the content
+    is dumped twice. If the buffer would remember the pid of the
+    last writer, it could discard the contents of the buffer when
+    the pid changes. This is because the parent retains the pid
+    and the buffer content while the child changes the pid and
+    discards the content. Should be an optionial feature.
+    -> Can be immediately implemented (rse)
+- l2_ch_buffer needs a L2_OK and L2_OK_PASS consistency check
+- l2_ch_buffer should not use different param types in alarm and setitimer(2).
+- l2_ch_buffer should incorporate multiplexer logic to properly flush
+    when multiple buffers are in use. Can be implemented via a environment
+    level timer and a lowest common denominator alarm function. For example,
+        tmr1 9secs; tmr2 6secs; envtmr expires every lcd(9,6) = 3secs
+- l2_ch_buffer should not implement an autoflush, rather L2 should
+    have a multiplexed timer object to serve timer requests from
+    incoming callbacks.
+
+Library-wide changes --------------------------------------
+- Replace l2_sockmon with netcat or peep.
+- Add nonreentrant log method like mm.
+- Add striptease target to Makefile.
+- Provide a C++ implementation also.
+- l2tool needs an [addr:]port option
+- Adjust license to ISC/MIT/BSD.
+- API access through the LogManager logm;
+- Jump to a callback in case of error, throw/exit.
+- Offer the option to reopen the logfile on each write.
+- Optimize log operations from log(..) {}
+    to log(....);
+
+- Implicit level decision should default to >=,
+    then follow with <=
+                      =
+                      ;
+                      *
+
+- '\r\n' Uebersetzung
+    Bei Input \r und \n wegstrippen, und von jedem Output Channel
+    entweder \r\n (smtp) oder nur \n wieder anhaengen lassen. Oder
+    als Kanaloption, wobei \r, \r\n, \n, gleich String, oder gar nix.
+
+- Spec-facility
+    To debug an application, sometimes it's overkill to log everything
+    at DEBUG level. I see an improvement when an additional facility
+    can be specified. Example: DEBUG/LMTP but don't care about NNTP in
+    the lmtp2nntp program. Possibly could be implemented as a second mask.
+    -> Needs more consideration before implementation should start (rse)
+
+- Prelogging
+    L2 should log even before it is initialized. Maybe the log function
+    should buffer everything as long as a NULL l2-context is passed and
+    if ever a non-NULL context is passed every remembered message should
+    be logged afterwards or if destroy/flush is executed with a NULL-
+    context it should print the buffered stuff to stderr.
+    -> Can be immediately implemented, but one has to be carefully here.
+       I want to think about this a little bit more in-depth. (rse)
+
+- Config file
+    Might be used with the previous prelogging principle in mind.
+    Namely, a file that describes a set of channel specs for one
+    or many configurations. Path is searched in the following order:
+      1. /etc/liblog.conf
+      2. (in ., .., ../..)
+      3. $HOME/.liblog.conf
+
+- hook_write methods should receive a null-termined string
+    instead of buf+size, because some channels like syslog
+    otherwise have to rebuffer the message and append the
+    null terminator character.
 
-Spec-Parsing:
-- location tracking
-- target=remote has to be currently after host=xxx because of
-  single-configuration procedure
+- An optional syslog(3) compatible API for converting syslog-only based
+  applications (like sendmail) to (restricted) liblog-based applications.
+  -> Des ist schon implimentiert in fakesyslog oder? (ms)
 
-- prefix channel produces implicitly two log messages if
-  a following buffer channel would not accumulate them. Hmmm..
+Spec-parsing ----------------------------------------------
+- implement location tracking
 
-- bugfix l2tool and after it works 100% remove l2_test 
-  in favor of l2tool and add instead a test shell script
-  which calls l2tool with various input specifications.
+- target=remote has to be currently after host=xxx because of
+    single-configuration procedure
 
-Channel-Only Revamping:
+- bugfix l2tool and after it works 100% remove l2_test in
+    favor of l2tool and add instead a test shell script
+    which calls l2tool with various input specifications.
+
+Channel-Only Revamping ------------------------------------
+- l2_objects update
 - syscall override ala OSSP SA in l2_env_t
-- l2_objects.fig update
 - perhaps rename l2_env to l2_ctx and l2_channel_ to just l2_
+- API cleanup for open semantics
 
-- Perhaps we should later also write an l2_ch_bofh.c which
-  something very cool but is undocumented and has its source
-  mangled/scrambled ;)
-
-- Problem: OpenPKG fakesyslog ist nasty, because it doesn't provide
-  logging to multiple files or filter out some messages. Additionally
-  the app has to be restarted in order to reopen the logfile which
-  is nasty for MTAs like Postfix in case of very high loads (because
-  they start again processign the queue from scratch). What we need
-  is a new L2-based libsyslog.a which sends the stuff via Unix Domain
-  socket to an L2 daemon which in turn logs to targets via L2.
-
-- "smart debug buffer channel": 
-   extra debug buffer, flush only if error occurs
-
-- "repeat channel": "last message repeated 100 times"
-
-- "\r\n" Problematik: Vorschlag: Bei Input \r und \n wegstrippen,
-  und von jedem Output Channel entweder \r\n (smtp) oder nur \n
-  wieder anhaengen lassen.
-
-- thl: log facility
-  To debug an application, sometimes it's overkill to log everything at
-  DEBUG level. I see an improvement when an additional facility can be
-  specified. Example: DEBUG/LMTP but don't care about NNTP in the
-  lmtp2nntp program. Possibly could be implemented as a second mask.
-  -> needs more consideration before implementation should start (rse)
-
-- thl: buffer fork() awareness
-  When the process forks, the buffer is duplicated. Currently this means
-  the buffer has to be flushed in advance or the content is dumped twice.
-  If the buffer would remember the pid of the last writer, it could
-  discard the contents of the buffer when the pid changes. This is because
-  the parent retains the pid and the buffer content while the child
-  changes the pid and discards the content. Should be an optionial feature.
-  -> can be immediately implemented (rse)
-
-- thl: prelog
-  I want to log everything even things that happen before L2 is
-  initialized. Complicated, i know. Maybe the log function should buffer
-  everything as long as a NULL l2-context is passed and if ever a non-NULL
-  context is passed every remembered message should be logged afterwards
-  or if destroy/ flush is executed with a NULL-context it should print the
-  buffered stuff to stderr.
-  -> can be immediately implemented, but one has to be carefully here.
-     Seems like I want to think about this a little bit more in-depth. (rse)
-
-o Another great rse idea, make a proxy server channel that sniffs a tcp
-  connection and dumps its traffic to standard out. Optionally, it could dump
-  this to the next channel, allowing for parsing through an unknown protocol.
-
-RSE:
-- channel API cleanup: open semantics
-- bind and udp parameters in socket channel
-- prefix channels:
-  - printf style without strftime;
-- l2tool [addr:]port
-- autoflush via shared memory and sub-process?
-- NNTP channel
-- SNMP trap channel
-- perhaps replace too large PCRE stuff with smaller pattern
-  matching stuff
-- file channel should optionally support file locking 
-  via fcntl(3).
-- idea of an asynch channel, that spawns a new process and
-  communicates with the stemming channels via shared memory
-
-Lunchtime:
-- Correct DNS resolve blocking problem by using a funky asynchronous DNS lib.
-  This leads to easy reimplementation of the prefix channel (asynchronous)
-
-MS:
-- netcat can probably replace l2_sockmon, remove from build and add netcat
-- pipe channel may need a big overhaul if we redesign
-  it around the asynch channel principle
-- review pipe handler for dangling descriptors
-- configure only checks existance of command in non-shell mode
-- find alternative to exec arguments which is hard coded to 256
-- signal handler chaining, save old signal handler and call it after our own
-- consider adding options such as PCRE_CASELESS to filter channel
-- implement "action" channel, can be based on pipe channel
-- correct problem with multiple buffer channels using the timer
-    can be solved by running a signal handler at the environment level, then
-    approximating the (multiple) user-chosen timer intervals by setting the
-    environment timer to expire at the time of the least common denominator of
-    the configured timer intervals
-      ie: tmr1 9secs; tmr2 6secs; envtmr will expire every lcd(9,6) = 3secs
-    alarm and setitimer methods are using different param types! Fixit.
-    review newly introduced timer code for L2_OK and L2_OK_PASS inconsistency
-- solve problem with buffer (timer), irc (ping), and buffer (autoflush), by
-  creating sleep/ping threads. The disadvantage is that we must depend on a
-  pth installation, and force the parent app to be multithreaded.
-  Alternatively, we can spawn a management process in l2_stream_create(), who
-  owns management resources globally available to all channels. Or write the
-  l2 mini-protocol :-(
-- split all timer server code out of L2 and into new multiplexed timer object
-    probably must use callbacks, unless user signal space is occupied
-
-ISSUES
-------
-
-o hook_write's should perhaps receive a nul-termined string
-  instead of buf+size, because syslog else has to re-buffer it
-  in order to append the nul terminator character.
-
-o Stream Members: channels static array
-  - consider dynamic
-
-o buffer
-  user needs to know how a buffer object behaves in
-  relation to up/downstream channels. When does it
-  pass its data to the next channel, when does it
-  erase, what happens to its data when it is over
-  written or flushed...
-
-o buffer timer
-  How on earth to use a C-style exception handler to flush
-  our buffer? We can't directly call ANY function from an
-  exception state, so this might not work at all.
-
-  Depends on SIGALARM, only one handler of which may exist.
-  A more robust implementation would not use such a precious
-  resource, and guard against signal collision with other channels.
-
-o syslog
-  many options need docu, and we should mention to
-  the user that more info is found in the man page
-  for syslog(), because after all that is what is
-  doing all the work in our implementation. Also,
-  can we really properly document these features
-  if they change from one system's syslog to the next?
-
-o errors
-  when a channel fails during an operation, how
-  does it report this? How should a user interpret
-  the error message or other data? Do we need more
-  accurate or detailed error messages in the channel
-  code? When a channel fails, does it continue
-  passing data on to downstream channels? Is it
-  corrupt data?
-
-o Syslog Kanal
-  - Trim down to what will be used, right now the
-    channel supports ALL functionality through syslog(3)
-
-BRAINSTORMING
--------------
-
-Braindump:
-- debugging is special case of logging
-- tracing is special case of debugging
-
-Channel Handler Configuration:
-o l2_handler_null
-  - no configuration at all
-o l2_handler_fd
-  - mode="unix|stdio"
-  - fd=int|FILE*
-o l2_handler_file
-  - mode="unix|stdio"
-  - path=char*
-  - append="yes|no"
-o l2_handler_pipe
-  - url="prg:/path/to/program"
-  - fd=int
-o l2_handler_socket
-  - url="tcp://hostname:port"
-  - fd=int
-o l2_handler_syslog
-  - ident=char*
-  - should have its own logic
-    and not use unix lib syslog()
-    thus able to write to a remote
-    syslog daemon
-o l2_handler_filter
-  - pattern=char*
-o l2_handler_prefix
-  - prefix=char*
-o l2_handler_buffer
-  - size=size_t
-o all output channels
-  - should they have downstream, or be true endpoints?
-o l2_ch_socket
-  - write should handle partial send()
-    thus check the return of send
-o l2_ch_buffer
-  - write() must implicitly flush() when incoming
-    data is larger than remaining buffer capacity
-o l2_ch_action
-  - new action channnel could accept a regexp or L2_LEVEL as configuration,
-    and then run a callback function or exec a binary depending on incoming
-    l2 stream messages. For example, lmtp2nntp could conditionally run code
-    via the new action channel to monitor spam or alert an admin about a
-    flawed OR CURIOUs LMTP header. However, the action channel is also
-    appealling to other applications that don't run as daemons or at the
-    system level.
-  - l2_channel_config(l2Act, "regexp,run", "(nobody|cz)", "callback");
-  - l2_channel_config(l2Act, "funcptr", pfnAddtcpwrap);
-
-License:
-- ISC/MIT/BSD
-
-Sprache:
-- C++
-- C
-
-Aufbau:
-1. Layer C++ API      log.hpp,  log.cpp
-2. Layer C   API      log.h     log.c
-3. Layer C   Backend  backend.h backend.c
-- "make striptease"
-
-- optimierung:
-log(..)
-{
-}
-:
-log(....);
-:
-
-API Levels:
-- PANIC    (-> LOG_EMERG)
-- CRITICAL (-> LOG_CRIT)
-- ERROR    (-> LOG_ERR)
-- WARNING
-- NOTICE
-- INFO
-- TRACE    (-> LOG_DEBUG)
-- DEBUG    (-> LOG_DEBUG)
-- ALERT
-
-Level Entscheidungen:
->= (default)
-<=
-=
-;
-*
-
-Backend Channels:
-1 Level -> N Channels
-- file (append)
-- program (stdin)
-- syslog
-- stderr/stdout
-- null (discard, nicht nur /dev/null)
-- filedescriptor (escape/ext)
-- callback function
-
-Log Messages:
-- raw
-- optional prefixes (inclusive order):
-  string
-  facility
-  level
-  timestamp
-  pid
-  (tid)
-- errno (like syslog %m)
-- eigene %{foo}x mit callback function mit context
-- automatisch: number -> string mapping (fuer error strings)
-- __FILE__, __LINE__, (__FUNCTION__)
-
-Configuration:
-- ueber C/C++ API
-- zusaetzlich Config-File
-  1. /etc/liblog.conf
-  2. (in ., .., ../..)
-  3. $HOME/.liblog.conf
-
-- !debug -> !code
-
-API C (ala MM):
-- reentrant: log_xxx
-- non-reentrant: Log_xxx
-
-Message Filtering/Masking:
-- facility und/oder levels und/oder wildcard pattern
-
-API Using:
-- C++:
-  LogManager logm;
-  logm.debug1("test");
-  logm.configure("
-
-- C:
-  log lh;
-  lh = log_init(LOG_CFGFILE|LOG_CFGPARENT|LOG_XXX|..., "foo" (=facility));
-  log_configure(lh, "foo", LOG_WARN|LOG_LESSER, null);
-  log_cb(lh, "x", func, ctx);
-  int func(void *ctx, char *str, ...);
-  log_msg(lh, LOG_WARN, "..%{foo}x %s...%E..", cp);
-  log_dbg(lh, "..%{foo}x %s...%E..", cp);
-  log_kill(lh);
-
-- Buffered I/O:
-  fuer manche channels non-buffered (debug, errors)
-  fuer manche andere aber buffered (access log, performance)
-  loesung: I/O ueber callbacks (3x: open, write, close) z.B. RRDTool
-
-- Varargs:
-  log ist nur wrapper fuer vlog
-
-- Error Handling:
-  o log kein Return Code
-  o aber error callback function (dadrin in C++: throw, in C: exit)
-
-- Newline Handling:
-  option fuer channel: \r, \r\n, \n oder gleich string
-  und moeglichkeit gar nix (string="")
-
-- Perhaps:
-  optionally reopen logfile on each write
-
-- An optional syslog(3) compatible API for converting syslog-only based
-  applications (like sendmail) to (restricted) liblog-based applications.
-
-- Ein Wort noch zu variablen Argumentlisten in cpp-Makros: gcc
-  unterstützt dies in in der GNU- und der C99-Ausführung. Das heißt,
-  der "..." Parameter kann im Makro respektive über "args" und über
-  "__VA_ARGS__" angesprochen werden. Wichtig ist dabei, daß "..."
-  nicht leer sein -- also kein Argument enthalten -- darf, da sonst
-  der Präprocessor an einem eventuell vorhandenen Komma scheitert.
-  Dies kann beim gcc durch Voranstellen von "##" vor dem "__VA_ARGS__"
-  umgangen werden. Ouch.
-  Beide Erweiterungen sind derzeit nicht aktiv, wenn mit -ansi
-  compiliert wird. Explizit anschalten läßt sich die standard-konforme
-  Erweiterung über "-std=c9x", bzw. "-std=c99" bei neueren gccs.
+Related Projects ------------------------------------------
+- Custom libsyslog
+    OpenPKG fakesyslog is nasty, because it doesn't provide logging
+    to multiple files or filter out some messages. Additionally the
+    app has to be restarted in order to reopen the logfile which is
+    nasty for MTAs like Postfix in case of very high loads (because
+    they start again processign the queue from scratch). What we need
+    is a new L2-based libsyslog.a which sends the stuff via Unix Domain
+    socket to an L2 daemon which in turn logs to targets via L2.
+
+Documentation ---------------------------------------------
+- l2_ch_buffer
+    How the buffer object behaves in relation to up/downstream
+    channels. When does it pass its data to the next channel,
+    when does it erase, what happens to its data when it is
+    over written or flushed.
+
+- l2_ch_syslog
+    Describe the options, and mention that more info is found
+    in the man page for syslog(3). Identify features that
+    change from one system to the next.
+
+- l2 errors
+    Describe how an l2 channel reports errors when it fails during
+    an operation, and how a user should interpret the error message?
+    Describe whether a channel continues passing data on downstream
+    if it somehow fails, and if the data will be corrupt.
 
-High-Level Configuration Interface
+High-level configuration interface
 ==================================
-
 Config-File (OSSP):
-
 logging {
     channel 0 null ALL; 
 
@@ -373,11 +201,9 @@
 
     log access 0;
     log stat 1;
-
 };
 
 Alternative: 
-
 log access {
     error: syslog;
     prefix -> { 
@@ -388,7 +214,6 @@
 }
 
 Command-Line (lmtp2nntp):
-
 $ lmtp2nntp -l 'INFO:'
   { prefix(prefix="%Y-%m-%d/%H:%M:%S %L")
     ->{ buffer(size=4k)
@@ -398,3 +223,73 @@
     error:syslog
   }
 
+Brainstorming ---------------------------------------------
+- Debugging can be implemented as a special case of logging
+- Tracing can be implemented as a special case of debugging
+
+
+*** *** *** *** *** *** Out of date *** *** *** *** *** ***
+*** *** *** *** *** *** Out of date *** *** *** *** *** ***
+*** *** *** *** *** *** Out of date *** *** *** *** *** ***
+
+
+Backend channels ------------------------------------------
+- Level -> N Channels
+- file (append)
+- program (stdin)
+- syslog
+- stderr/stdout
+- null (discard, nicht nur /dev/null)
+- filedescriptor (escape/ext)
+- callback function
+
+Aufbau ----------------------------------------------------
+Layer C++ API      log.hpp,  log.cpp
+Layer C   API      log.h     log.c
+Layer C   Backend  backend.h backend.c
+
+- Ein Wort noch zu variablen Argumentlisten in cpp-Makros: gcc
+  unterstützt dies in in der GNU- und der C99-Ausführung. Das heißt,
+  der "..." Parameter kann im Makro respektive über "args" und über
+  "__VA_ARGS__" angesprochen werden. Wichtig ist dabei, daß "..."
+  nicht leer sein -- also kein Argument enthalten -- darf, da sonst
+  der Präprocessor an einem eventuell vorhandenen Komma scheitert.
+  Dies kann beim gcc durch Voranstellen von "##" vor dem "__VA_ARGS__"
+  umgangen werden. Ouch. Beide Erweiterungen sind derzeit nicht aktiv,
+  wenn mit -ansi compiliert wird. Explizit anschalten läßt sich die
+  standard-konforme Erweiterung über "-std=c9x", bzw. "-std=c99" bei
+  neueren gccs.
+
+Log Messages:
+- raw
+- optional prefixes (inclusive of order)
+    string
+    facility
+    level
+    timestamp
+    pid
+    (tid)
+- errno (like syslog %m)
+- eigene %{foo}x mit callback function mit context
+- automatisch number -> string mapping (fuer error strings)
+- !debug -> !code
+
+To be deleted ---------------------------------------------
+- C:
+  log lh;
+  lh = log_init(LOG_CFGFILE|LOG_CFGPARENT|LOG_XXX|..., "foo" (=facility));
+  log_configure(lh, "foo", LOG_WARN|LOG_LESSER, null);
+  log_cb(lh, "x", func, ctx);
+  int func(void *ctx, char *str, ...);
+  log_msg(lh, LOG_WARN, "..%{foo}x %s...%E..", cp);
+  log_dbg(lh, "..%{foo}x %s...%E..", cp);
+  log_kill(lh);
+
+- Buffered I/O:
+  fuer manche channels non-buffered (debug, errors)
+  fuer manche andere aber buffered (access log, performance)
+  loesung: I/O ueber callbacks (3x: open, write, close) z.B. RRDTool
+
+- Varargs:
+  log ist nur wrapper fuer vlog
+

CVSTrac 2.0.1