OSSP L2 ======= - thl: log facility To debug an application, sometimes it's overkill to log everything at DEBUG level. I see an improvement when an additional facility can be specified. Example: DEBUG/LMTP but don't care about NNTP in the lmtp2nntp program. Possibly could be implemented as a second mask. -> needs more consideration before implementation should start (rse) - thl: buffer fork() awareness When the process forks, the buffer is duplicated. Currently this means the buffer has to be flushed in advance or the content is dumped twice. If the buffer would remember the pid of the last writer, it could discard the contents of the buffer when the pid changes. This is because the parent retains the pid and the buffer content while the child changes the pid and discards the content. Should be an optionial feature. -> can be immediately implemented (rse) - thl: prelog I want to log everything even things that happen before L2 is initialized. Complicated, i know. Maybe the log function should buffer everything as long as a NULL l2-context is passed and if ever a non-NULL context is passed every remembered message should be logged afterwards or if destroy/ flush is executed with a NULL-context it should print the buffered stuff to stderr. -> can be immediately implemented, but one has to be carefully here. Seems like I want to think about this a little bit more in-depth. (rse) RSE: - channel API cleanup: open semantics - bind and udp parameters in socket channel - socket channel convert to use SA library - prefix channels: - printf style without strftime; - l2tool [addr:]port - autoflush via shared memory and sub-process? - IRC channel - NNTP channel - SNMP trap channel - perhaps generalize channels and get rid of stream by putting formatters into external object referenced by channels and by providing a T-channel at the top. - perhaps replace too large PCRE stuff with smaller pattern matching stuff - file channel should optionally support file locking via fcntl(3). - syslog channel should support direct remote logging - implement "filter" channel MS: - implement pipe channel - review pipe handler for dangling descriptors - configure only checks existance of command in non-shell mode - find alternative to exec arguments which is hard coded to 256 - signal handler chaining, save old signal handler and call it after our own - consider adding options such as PCRE_CASELESS to filter channel ISSUES ------ o hook_write's should perhaps receive a nul-termined string instead of buf+size, because syslog else has to re-buffer it in order to append the nul terminator character. o Stream Members: channels static array - consider dynamic o buffer user needs to know how a buffer object behaves in relation to up/downstream channels. When does it pass its data to the next channel, when does it erase, what happens to its data when it is over written or flushed... o buffer timer How on earth to use a C-style exception handler to flush our buffer? We can't directly call ANY function from an exception state, so this might not work at all. Depends on SIGALARM, only one handler of which may exist. A more robust implementation would not use such a precious resource, and guard against signal collision with other channels. o syslog many options need docu, and we should mention to the user that more info is found in the man page for syslog(), because after all that is what is doing all the work in our implementation. Also, can we really properly document these features if they change from one system's syslog to the next? o errors when a channel fails during an operation, how does it report this? How should a user interpret the error message or other data? Do we need more accurate or detailed error messages in the channel code? When a channel fails, does it continue passing data on to downstream channels? Is it corrupt data? o Syslog Kanal - Trim down to what will be used, right now the channel supports ALL functionality through syslog(3) BRAINSTORMING ------------- Braindump: - debugging is special case of logging - tracing is special case of debugging Channel Handler Configuration: o l2_handler_null - no configuration at all o l2_handler_fd - mode="unix|stdio" - fd=int|FILE* o l2_handler_file - mode="unix|stdio" - path=char* - append="yes|no" o l2_handler_pipe - url="prg:/path/to/program" - fd=int o l2_handler_socket - url="tcp://hostname:port" - fd=int o l2_handler_syslog - ident=char* - should have its own logic and not use unix lib syslog() thus able to write to a remote syslog daemon o l2_handler_filter - pattern=char* o l2_handler_prefix - prefix=char* o l2_handler_buffer - size=size_t o all output channels - should they have downstream, or be true endpoints? o l2_ch_socket - write should handle partial send() thus check the return of send o l2_ch_buffer - write() must implicitly flush() when incoming data is larger than remaining buffer capacity License: - ISC/MIT/BSD Sprache: - C++ - C Aufbau: 1. Layer C++ API log.hpp, log.cpp 2. Layer C API log.h log.c 3. Layer C Backend backend.h backend.c - "make striptease" - optimierung: log(..) { } : log(....); : API Levels: - PANIC (-> LOG_EMERG) - CRITICAL (-> LOG_CRIT) - ERROR (-> LOG_ERR) - WARNING - NOTICE - INFO - TRACE (-> LOG_DEBUG) - DEBUG (-> LOG_DEBUG) - ALERT Level Entscheidungen: >= (default) <= = ; * Backend Channels: 1 Level -> N Channels - file (append) - program (stdin) - syslog - stderr/stdout - null (discard, nicht nur /dev/null) - filedescriptor (escape/ext) - callback function Log Messages: - raw - optional prefixes (inclusive order): string facility level timestamp pid (tid) - errno (like syslog %m) - eigene %{foo}x mit callback function mit context - automatisch: number -> string mapping (fuer error strings) - __FILE__, __LINE__, (__FUNCTION__) Configuration: - ueber C/C++ API - zusaetzlich Config-File 1. /etc/liblog.conf 2. (in ., .., ../..) 3. $HOME/.liblog.conf - !debug -> !code API C (ala MM): - reentrant: log_xxx - non-reentrant: Log_xxx Message Filtering/Masking: - facility und/oder levels und/oder wildcard pattern API Using: - C++: LogManager logm; logm.debug1("test"); logm.configure(" - C: log lh; lh = log_init(LOG_CFGFILE|LOG_CFGPARENT|LOG_XXX|..., "foo" (=facility)); log_configure(lh, "foo", LOG_WARN|LOG_LESSER, null); log_cb(lh, "x", func, ctx); int func(void *ctx, char *str, ...); log_msg(lh, LOG_WARN, "..%{foo}x %s...%E..", cp); log_dbg(lh, "..%{foo}x %s...%E..", cp); log_kill(lh); - Buffered I/O: fuer manche channels non-buffered (debug, errors) fuer manche andere aber buffered (access log, performance) loesung: I/O ueber callbacks (3x: open, write, close) z.B. RRDTool - Varargs: log ist nur wrapper fuer vlog - Error Handling: o log kein Return Code o aber error callback function (dadrin in C++: throw, in C: exit) - Newline Handling: option fuer channel: \r, \r\n, \n oder gleich string und moeglichkeit gar nix (string="") - Perhaps: optionally reopen logfile on each write - An optional syslog(3) compatible API for converting syslog-only based applications (like sendmail) to (restricted) liblog-based applications. - Ein Wort noch zu variablen Argumentlisten in cpp-Makros: gcc unterstützt dies in in der GNU- und der C99-Ausführung. Das heißt, der "..." Parameter kann im Makro respektive über "args" und über "__VA_ARGS__" angesprochen werden. Wichtig ist dabei, daß "..." nicht leer sein -- also kein Argument enthalten -- darf, da sonst der Präprocessor an einem eventuell vorhandenen Komma scheitert. Dies kann beim gcc durch Voranstellen von "##" vor dem "__VA_ARGS__" umgangen werden. Ouch. Beide Erweiterungen sind derzeit nicht aktiv, wenn mit -ansi compiliert wird. Explizit anschalten läßt sich die standard-konforme Erweiterung über "-std=c9x", bzw. "-std=c99" bei neueren gccs. High-Level Configuration Interface ================================== Config-File (OSSP): logging { channel 0 null ALL; channel 1 prefix CUSTOM1 { prefix="%Y-%m-%d/%H:%M:%S" }; channel 2 buffer { size=8k }; channel 3 file { path=/path/to/ossp.stat.log; mode=0644 }; channel 4 prefix wmask=DEBUG { prefix="%Y-%m-%d/%H:%M:%S %L" }; channel 5 buffer { size=4k }; channel 6 file { path=/path/to/ossp.error.log; mode=0644 }; channel 7 syslog wmask=ERROR { peer=remote; host=loghost }; channel 8 filter { regex="(memory|foo)" }; channel 9 smtp { host=loghost; rcpt="rse@engelschall.com" }; stream { 0->(4,7); 4->(5,7); 5->6; 8->9; 1->2->3 }; log access 0; log stat 1; }; Alternative: log access { error: syslog; prefix -> { buffer(size=4k) -> file(path=/path/to/log, mode=0644); panic: smtp(host=en1, rcpt=rse@engelschall.com); } } Command-Line (lmtp2nntp): $ lmtp2nntp -l 'INFO:' { prefix(prefix="%Y-%m-%d/%H:%M:%S %L") ->{ buffer(size=4k) ->file(path=/path/to/log,mode=0644); smtp(host=en1,rcpt=rse@engelschall.com) }; error:syslog } Grammar ======= hierarchy : stream | '{' stream_list '}' ; stream_list : hierarchy | hierarchy ';' stream_list ; stream : channel | channel "->" hierarchy ; channel : channel_mask ':' channel_spec | channel_spec ; channel_mask : T_ID_LEVEL | '(' channel_levels ')' ; channel_levels : T_ID_LEVEL | T_ID_LEVEL '|' channel_mask_list ; channel_spec : T_ID_CHANNEL | T_ID_CHANNEL '(' channel_params ')' ; channel_params : channel_param | channel_param ',' channel_params ; channel_param : T_ID_PARAM '=' T_STRING ;