ossp-pkg/l2/TODO
1.29
OSSP L2
=======
- thl: log facility
To debug an application, sometimes it's overkill to log everything at
DEBUG level. I see an improvement when an additional facility can be
specified. Example: DEBUG/LMTP but don't care about NNTP in the
lmtp2nntp program. Possibly could be implemented as a second mask.
-> needs more consideration before implementation should start (rse)
- thl: buffer fork() awareness
When the process forks, the buffer is duplicated. Currently this means
the buffer has to be flushed in advance or the content is dumped twice.
If the buffer would remember the pid of the last writer, it could
discard the contents of the buffer when the pid changes. This is because
the parent retains the pid and the buffer content while the child
changes the pid and discards the content. Should be an optionial feature.
-> can be immediately implemented (rse)
- thl: prelog
I want to log everything even things that happen before L2 is
initialized. Complicated, i know. Maybe the log function should buffer
everything as long as a NULL l2-context is passed and if ever a non-NULL
context is passed every remembered message should be logged afterwards
or if destroy/ flush is executed with a NULL-context it should print the
buffered stuff to stderr.
-> can be immediately implemented, but one has to be carefully here.
Seems like I want to think about this a little bit more in-depth. (rse)
RSE:
- channel API cleanup: open semantics
- bind and udp parameters in socket channel
- socket channel convert to use SA library
- prefix channels:
- printf style without strftime;
- l2tool [addr:]port
- autoflush via shared memory and sub-process?
- IRC channel
- NNTP channel
- SNMP trap channel
- perhaps generalize channels and get rid of stream
by putting formatters into external object referenced
by channels and by providing a T-channel at the top.
- perhaps replace too large PCRE stuff with smaller pattern
matching stuff
- file channel should optionally support file locking
via fcntl(3).
- syslog channel should support direct remote logging
- implement "filter" channel
MS:
- implement pipe channel
- review pipe handler for dangling descriptors
- configure only checks existance of command in non-shell mode
- find alternative to exec arguments which is hard coded to 256
- signal handler chaining, save old signal handler and call it after our own
- consider adding options such as PCRE_CASELESS to filter channel
ISSUES
------
o hook_write's should perhaps receive a nul-termined string
instead of buf+size, because syslog else has to re-buffer it
in order to append the nul terminator character.
o Stream Members: channels static array
- consider dynamic
o buffer
user needs to know how a buffer object behaves in
relation to up/downstream channels. When does it
pass its data to the next channel, when does it
erase, what happens to its data when it is over
written or flushed...
o buffer timer
How on earth to use a C-style exception handler to flush
our buffer? We can't directly call ANY function from an
exception state, so this might not work at all.
Depends on SIGALARM, only one handler of which may exist.
A more robust implementation would not use such a precious
resource, and guard against signal collision with other channels.
o syslog
many options need docu, and we should mention to
the user that more info is found in the man page
for syslog(), because after all that is what is
doing all the work in our implementation. Also,
can we really properly document these features
if they change from one system's syslog to the next?
o errors
when a channel fails during an operation, how
does it report this? How should a user interpret
the error message or other data? Do we need more
accurate or detailed error messages in the channel
code? When a channel fails, does it continue
passing data on to downstream channels? Is it
corrupt data?
o Syslog Kanal
- Trim down to what will be used, right now the
channel supports ALL functionality through syslog(3)
BRAINSTORMING
-------------
Braindump:
- debugging is special case of logging
- tracing is special case of debugging
Channel Handler Configuration:
o l2_handler_null
- no configuration at all
o l2_handler_fd
- mode="unix|stdio"
- fd=int|FILE*
o l2_handler_file
- mode="unix|stdio"
- path=char*
- append="yes|no"
o l2_handler_pipe
- url="prg:/path/to/program"
- fd=int
o l2_handler_socket
- url="tcp://hostname:port"
- fd=int
o l2_handler_syslog
- ident=char*
- should have its own logic
and not use unix lib syslog()
thus able to write to a remote
syslog daemon
o l2_handler_filter
- pattern=char*
o l2_handler_prefix
- prefix=char*
o l2_handler_buffer
- size=size_t
o all output channels
- should they have downstream, or be true endpoints?
o l2_ch_socket
- write should handle partial send()
thus check the return of send
o l2_ch_buffer
- write() must implicitly flush() when incoming
data is larger than remaining buffer capacity
License:
- ISC/MIT/BSD
Sprache:
- C++
- C
Aufbau:
1. Layer C++ API log.hpp, log.cpp
2. Layer C API log.h log.c
3. Layer C Backend backend.h backend.c
- "make striptease"
- optimierung:
log(..)
{
}
:
log(....);
:
API Levels:
- PANIC (-> LOG_EMERG)
- CRITICAL (-> LOG_CRIT)
- ERROR (-> LOG_ERR)
- WARNING
- NOTICE
- INFO
- TRACE (-> LOG_DEBUG)
- DEBUG (-> LOG_DEBUG)
- ALERT
Level Entscheidungen:
>= (default)
<=
=
;
*
Backend Channels:
1 Level -> N Channels
- file (append)
- program (stdin)
- syslog
- stderr/stdout
- null (discard, nicht nur /dev/null)
- filedescriptor (escape/ext)
- callback function
Log Messages:
- raw
- optional prefixes (inclusive order):
string
facility
level
timestamp
pid
(tid)
- errno (like syslog %m)
- eigene %{foo}x mit callback function mit context
- automatisch: number -> string mapping (fuer error strings)
- __FILE__, __LINE__, (__FUNCTION__)
Configuration:
- ueber C/C++ API
- zusaetzlich Config-File
1. /etc/liblog.conf
2. (in ., .., ../..)
3. $HOME/.liblog.conf
- !debug -> !code
API C (ala MM):
- reentrant: log_xxx
- non-reentrant: Log_xxx
Message Filtering/Masking:
- facility und/oder levels und/oder wildcard pattern
API Using:
- C++:
LogManager logm;
logm.debug1("test");
logm.configure("
- C:
log lh;
lh = log_init(LOG_CFGFILE|LOG_CFGPARENT|LOG_XXX|..., "foo" (=facility));
log_configure(lh, "foo", LOG_WARN|LOG_LESSER, null);
log_cb(lh, "x", func, ctx);
int func(void *ctx, char *str, ...);
log_msg(lh, LOG_WARN, "..%{foo}x %s...%E..", cp);
log_dbg(lh, "..%{foo}x %s...%E..", cp);
log_kill(lh);
- Buffered I/O:
fuer manche channels non-buffered (debug, errors)
fuer manche andere aber buffered (access log, performance)
loesung: I/O ueber callbacks (3x: open, write, close) z.B. RRDTool
- Varargs:
log ist nur wrapper fuer vlog
- Error Handling:
o log kein Return Code
o aber error callback function (dadrin in C++: throw, in C: exit)
- Newline Handling:
option fuer channel: \r, \r\n, \n oder gleich string
und moeglichkeit gar nix (string="")
- Perhaps:
optionally reopen logfile on each write
- An optional syslog(3) compatible API for converting syslog-only based
applications (like sendmail) to (restricted) liblog-based applications.
- Ein Wort noch zu variablen Argumentlisten in cpp-Makros: gcc
unterstützt dies in in der GNU- und der C99-Ausführung. Das heißt,
der "..." Parameter kann im Makro respektive über "args" und über
"__VA_ARGS__" angesprochen werden. Wichtig ist dabei, daß "..."
nicht leer sein -- also kein Argument enthalten -- darf, da sonst
der Präprocessor an einem eventuell vorhandenen Komma scheitert.
Dies kann beim gcc durch Voranstellen von "##" vor dem "__VA_ARGS__"
umgangen werden. Ouch.
Beide Erweiterungen sind derzeit nicht aktiv, wenn mit -ansi
compiliert wird. Explizit anschalten läßt sich die standard-konforme
Erweiterung über "-std=c9x", bzw. "-std=c99" bei neueren gccs.
High-Level Configuration Interface
==================================
Config-File (OSSP):
logging {
channel 0 null ALL;
channel 1 prefix CUSTOM1 { prefix="%Y-%m-%d/%H:%M:%S" };
channel 2 buffer { size=8k };
channel 3 file { path=/path/to/ossp.stat.log; mode=0644 };
channel 4 prefix wmask=DEBUG { prefix="%Y-%m-%d/%H:%M:%S %L" };
channel 5 buffer { size=4k };
channel 6 file { path=/path/to/ossp.error.log; mode=0644 };
channel 7 syslog wmask=ERROR { peer=remote; host=loghost };
channel 8 filter { regex="(memory|foo)" };
channel 9 smtp { host=loghost; rcpt="rse@engelschall.com" };
stream { 0->(4,7); 4->(5,7); 5->6; 8->9; 1->2->3 };
log access 0;
log stat 1;
};
Alternative:
log access {
error: syslog;
prefix -> {
buffer(size=4k) ->
file(path=/path/to/log, mode=0644);
panic: smtp(host=en1, rcpt=rse@engelschall.com);
}
}
Command-Line (lmtp2nntp):
$ lmtp2nntp -l 'INFO:'
{ prefix(prefix="%Y-%m-%d/%H:%M:%S %L")
->{ buffer(size=4k)
->file(path=/path/to/log,mode=0644);
smtp(host=en1,rcpt=rse@engelschall.com)
};
error:syslog
}
Grammar
=======
hierarchy : stream
| '{' stream_list '}'
;
stream_list : hierarchy
| hierarchy ';' stream_list
;
stream : channel
| channel "->" hierarchy
;
channel : channel_mask ':' channel_spec
| channel_spec
;
channel_mask : T_ID_LEVEL
| '(' channel_levels ')'
;
channel_levels : T_ID_LEVEL
| T_ID_LEVEL '|' channel_mask_list
;
channel_spec : T_ID_CHANNEL
| T_ID_CHANNEL '(' channel_params ')'
;
channel_params : channel_param
| channel_param ',' channel_params
;
channel_param : T_ID_PARAM '=' T_STRING
;