OSSP CVS Repository

ossp - ossp-pkg/l2/TODO 1.29
Not logged in
[Honeypot]  [Browse]  [Directory]  [Home]  [Login
[Reports]  [Search]  [Ticket]  [Timeline
  [Raw

ossp-pkg/l2/TODO 1.29

OSSP L2
=======

- thl: log facility
  To debug an application, sometimes it's overkill to log everything at
  DEBUG level. I see an improvement when an additional facility can be
  specified. Example: DEBUG/LMTP but don't care about NNTP in the
  lmtp2nntp program. Possibly could be implemented as a second mask.
  -> needs more consideration before implementation should start (rse)

- thl: buffer fork() awareness
  When the process forks, the buffer is duplicated. Currently this means
  the buffer has to be flushed in advance or the content is dumped twice.
  If the buffer would remember the pid of the last writer, it could
  discard the contents of the buffer when the pid changes. This is because
  the parent retains the pid and the buffer content while the child
  changes the pid and discards the content. Should be an optionial feature.
  -> can be immediately implemented (rse)

- thl: prelog
  I want to log everything even things that happen before L2 is
  initialized. Complicated, i know. Maybe the log function should buffer
  everything as long as a NULL l2-context is passed and if ever a non-NULL
  context is passed every remembered message should be logged afterwards
  or if destroy/ flush is executed with a NULL-context it should print the
  buffered stuff to stderr.
  -> can be immediately implemented, but one has to be carefully here.
     Seems like I want to think about this a little bit more in-depth. (rse)

RSE:
- channel API cleanup: open semantics
- bind and udp parameters in socket channel
- socket channel convert to use SA library
- prefix channels:
  - printf style without strftime;
- l2tool [addr:]port
- autoflush via shared memory and sub-process?
- IRC channel
- NNTP channel
- SNMP trap channel
- perhaps generalize channels and get rid of stream
  by putting formatters into external object referenced
  by channels and by providing a T-channel at the top.
- perhaps replace too large PCRE stuff with smaller pattern
  matching stuff
- file channel should optionally support file locking 
  via fcntl(3).
- syslog channel should support direct remote logging
- implement "filter" channel

MS:
- implement pipe channel
- review pipe handler for dangling descriptors
- configure only checks existance of command in non-shell mode
- find alternative to exec arguments which is hard coded to 256
- signal handler chaining, save old signal handler and call it after our own
- consider adding options such as PCRE_CASELESS to filter channel

ISSUES
------

o hook_write's should perhaps receive a nul-termined string
  instead of buf+size, because syslog else has to re-buffer it
  in order to append the nul terminator character.

o Stream Members: channels static array
  - consider dynamic

o buffer
  user needs to know how a buffer object behaves in
  relation to up/downstream channels. When does it
  pass its data to the next channel, when does it
  erase, what happens to its data when it is over
  written or flushed...

o buffer timer
  How on earth to use a C-style exception handler to flush
  our buffer? We can't directly call ANY function from an
  exception state, so this might not work at all.

  Depends on SIGALARM, only one handler of which may exist.
  A more robust implementation would not use such a precious
  resource, and guard against signal collision with other channels.

o syslog
  many options need docu, and we should mention to
  the user that more info is found in the man page
  for syslog(), because after all that is what is
  doing all the work in our implementation. Also,
  can we really properly document these features
  if they change from one system's syslog to the next?

o errors
  when a channel fails during an operation, how
  does it report this? How should a user interpret
  the error message or other data? Do we need more
  accurate or detailed error messages in the channel
  code? When a channel fails, does it continue
  passing data on to downstream channels? Is it
  corrupt data?

o Syslog Kanal
  - Trim down to what will be used, right now the
    channel supports ALL functionality through syslog(3)

BRAINSTORMING
-------------

Braindump:
- debugging is special case of logging
- tracing is special case of debugging

Channel Handler Configuration:
o l2_handler_null
  - no configuration at all
o l2_handler_fd
  - mode="unix|stdio"
  - fd=int|FILE*
o l2_handler_file
  - mode="unix|stdio"
  - path=char*
  - append="yes|no"
o l2_handler_pipe
  - url="prg:/path/to/program"
  - fd=int
o l2_handler_socket
  - url="tcp://hostname:port"
  - fd=int
o l2_handler_syslog
  - ident=char*
  - should have its own logic
    and not use unix lib syslog()
    thus able to write to a remote
    syslog daemon
o l2_handler_filter
  - pattern=char*
o l2_handler_prefix
  - prefix=char*
o l2_handler_buffer
  - size=size_t
o all output channels
  - should they have downstream, or be true endpoints?
o l2_ch_socket
  - write should handle partial send()
    thus check the return of send
o l2_ch_buffer
  - write() must implicitly flush() when incoming
    data is larger than remaining buffer capacity

License:
- ISC/MIT/BSD

Sprache:
- C++
- C

Aufbau:
1. Layer C++ API      log.hpp,  log.cpp
2. Layer C   API      log.h     log.c
3. Layer C   Backend  backend.h backend.c
- "make striptease"

- optimierung:
log(..)
{
}
:
log(....);
:

API Levels:
- PANIC    (-> LOG_EMERG)
- CRITICAL (-> LOG_CRIT)
- ERROR    (-> LOG_ERR)
- WARNING
- NOTICE
- INFO
- TRACE    (-> LOG_DEBUG)
- DEBUG    (-> LOG_DEBUG)
- ALERT

Level Entscheidungen:
>= (default)
<=
=
;
*

Backend Channels:
1 Level -> N Channels
- file (append)
- program (stdin)
- syslog
- stderr/stdout
- null (discard, nicht nur /dev/null)
- filedescriptor (escape/ext)
- callback function

Log Messages:
- raw
- optional prefixes (inclusive order):
  string
  facility
  level
  timestamp
  pid
  (tid)
- errno (like syslog %m)
- eigene %{foo}x mit callback function mit context
- automatisch: number -> string mapping (fuer error strings)
- __FILE__, __LINE__, (__FUNCTION__)

Configuration:
- ueber C/C++ API
- zusaetzlich Config-File
  1. /etc/liblog.conf
  2. (in ., .., ../..)
  3. $HOME/.liblog.conf

- !debug -> !code

API C (ala MM):
- reentrant: log_xxx
- non-reentrant: Log_xxx

Message Filtering/Masking:
- facility und/oder levels und/oder wildcard pattern

API Using:
- C++:
  LogManager logm;
  logm.debug1("test");
  logm.configure("

- C:
  log lh;
  lh = log_init(LOG_CFGFILE|LOG_CFGPARENT|LOG_XXX|..., "foo" (=facility));
  log_configure(lh, "foo", LOG_WARN|LOG_LESSER, null);
  log_cb(lh, "x", func, ctx);
  int func(void *ctx, char *str, ...);
  log_msg(lh, LOG_WARN, "..%{foo}x %s...%E..", cp);
  log_dbg(lh, "..%{foo}x %s...%E..", cp);
  log_kill(lh);

- Buffered I/O:
  fuer manche channels non-buffered (debug, errors)
  fuer manche andere aber buffered (access log, performance)
  loesung: I/O ueber callbacks (3x: open, write, close) z.B. RRDTool

- Varargs:
  log ist nur wrapper fuer vlog

- Error Handling:
  o log kein Return Code
  o aber error callback function (dadrin in C++: throw, in C: exit)

- Newline Handling:
  option fuer channel: \r, \r\n, \n oder gleich string
  und moeglichkeit gar nix (string="")

- Perhaps:
  optionally reopen logfile on each write

- An optional syslog(3) compatible API for converting syslog-only based
  applications (like sendmail) to (restricted) liblog-based applications.

- Ein Wort noch zu variablen Argumentlisten in cpp-Makros: gcc
  unterstützt dies in in der GNU- und der C99-Ausführung. Das heißt,
  der "..." Parameter kann im Makro respektive über "args" und über
  "__VA_ARGS__" angesprochen werden. Wichtig ist dabei, daß "..."
  nicht leer sein -- also kein Argument enthalten -- darf, da sonst
  der Präprocessor an einem eventuell vorhandenen Komma scheitert.
  Dies kann beim gcc durch Voranstellen von "##" vor dem "__VA_ARGS__"
  umgangen werden. Ouch.
  Beide Erweiterungen sind derzeit nicht aktiv, wenn mit -ansi
  compiliert wird. Explizit anschalten läßt sich die standard-konforme
  Erweiterung über "-std=c9x", bzw. "-std=c99" bei neueren gccs.

High-Level Configuration Interface
==================================

Config-File (OSSP):

logging {
    channel 0 null ALL; 

    channel 1 prefix CUSTOM1 { prefix="%Y-%m-%d/%H:%M:%S" };
    channel 2 buffer { size=8k };
    channel 3 file { path=/path/to/ossp.stat.log; mode=0644 };

    channel 4 prefix wmask=DEBUG { prefix="%Y-%m-%d/%H:%M:%S %L" };

    channel 5 buffer { size=4k };
    channel 6 file { path=/path/to/ossp.error.log; mode=0644 };

    channel 7 syslog wmask=ERROR { peer=remote; host=loghost };

    channel 8 filter { regex="(memory|foo)" };
    channel 9 smtp { host=loghost; rcpt="rse@engelschall.com" };

    stream { 0->(4,7); 4->(5,7); 5->6; 8->9; 1->2->3 };

    log access 0;
    log stat 1;

};

Alternative: 

log access {
    error: syslog;
    prefix -> { 
        buffer(size=4k) -> 
            file(path=/path/to/log, mode=0644);
        panic: smtp(host=en1, rcpt=rse@engelschall.com);
    }
}

Command-Line (lmtp2nntp):

$ lmtp2nntp -l 'INFO:'
  { prefix(prefix="%Y-%m-%d/%H:%M:%S %L")
    ->{ buffer(size=4k)
        ->file(path=/path/to/log,mode=0644);
        smtp(host=en1,rcpt=rse@engelschall.com)
      };
    error:syslog
  }

Grammar
=======

hierarchy      : stream 
               | '{' stream_list '}'
               ; 

stream_list    : hierarchy 
               | hierarchy ';' stream_list
               ;

stream         : channel
               | channel "->" hierarchy
               ;

channel        : channel_mask ':' channel_spec
               | channel_spec
               ;

channel_mask   : T_ID_LEVEL
               | '(' channel_levels ')'
               ;

channel_levels : T_ID_LEVEL 
               | T_ID_LEVEL '|' channel_mask_list
               ;

channel_spec   : T_ID_CHANNEL
               | T_ID_CHANNEL '(' channel_params ')'
               ;

channel_params : channel_param
               | channel_param ',' channel_params
               ;
        
channel_param  : T_ID_PARAM '=' T_STRING
               ;
 

CVSTrac 2.0.1