head 1.62; access; symbols L2_0_9_13:1.62 FSL_1_7_0:1.62 L2_0_9_12:1.62 LMTP2NNTP_1_4_1:1.62 LMTP2NNTP_1_4_0:1.62 FSL_1_6_1:1.62 L2_0_9_11:1.62 FSL_1_6_0:1.62 FSL_1_6b2:1.62 L2_0_9_10:1.62 FSL_1_6b1:1.62 L2_0_9_9:1.62 LMTP2NNTP_1_3_0:1.62 LMTP2NNTP_1_3b2:1.62 LMTP2NNTP_1_3b1:1.62 LMTP2NNTP_1_3a3:1.62 FSL_1_5_0:1.62 LMTP2NNTP_1_3a2:1.62 FSL_1_5a3:1.62 LMTP2NNTP_1_3a1:1.62 FSL_1_5a2:1.62 L2_0_9_8:1.62 FSL_1_5a1:1.62 L2_0_9_7:1.62 L2_0_9_6:1.62 FSL_1_4_0:1.62 FSL_1_4b1:1.62 L2_0_9_5:1.62 FSL_1_4a1:1.60 FSL_1_3_0:1.60 FSL_1_3b1:1.60 L2_0_9_4:1.60 FSL_1_2_1:1.60 L2_0_9_3:1.60 FSL_1_2_0:1.60 L2_0_9_2:1.60 FSL_1_1_0:1.60 FSL_1_1b1:1.60 WORKOFF:1.60.0.2 WORKOFF_BP:1.60 FSL_1_0_8:1.59 LMTP2NNTP_1_2_0:1.59 LMTP2NNTP_1_2b4:1.59 LMTP2NNTP_1_2b3:1.59 LMTP2NNTP_1_2b2:1.59 LMTP2NNTP_1_2b1:1.59 LMTP2NNTP_1_2a8:1.59 LMTP2NNTP_1_2a7:1.59 FSL_1_0_7:1.59 FSL_1_0_6:1.57 FSL_1_0_5:1.57 FSL_1_0_4:1.57 L2_0_9_1:1.57 FSL_1_0_3:1.57 LMTP2NNTP_1_2a6:1.57 FSL_1_0_2:1.57 FSL_1_0_1:1.57 FSL_1_0_0:1.57 FSL_0_9_0:1.57 L2_0_9_0:1.57 FSL_0_1_12:1.57 FSL_0_1_11:1.57 FSL_0_1_10:1.57 FSL_0_1_9:1.56 FSL_0_1_8:1.56 FSL_0_1_7:1.56 FSL_0_1_6:1.56 FSL_0_1_5:1.56 FSL_0_1_1:1.55 LMTP2NNTP_1_2a5:1.55 LMTP2NNTP_1_2a4:1.54 LMTP2NNTP_1_2a3:1.54 LMTP2NNTP_1_2a1:1.47 LMTP2NNTP_1_1_1:1.30 LMTP2NNTP_1_1_0:1.30 LMTP2NNTP_1_1b4:1.30 LMTP2NNTP_1_1b3:1.30 L2_CHANNEL_ONLY_REVAMPING_BEFORE:1.35 LMTP2NNTP_1_1b2:1.30 LMTP2NNTP_1_1b1:1.29 L2_0_1_0:1.19 START_MICHAEL:1.7 L2_INITIAL:1.1.1.1 OSSP:1.1.1; locks; strict; comment @# @; 1.62 date 2003.11.06.15.31.47; author thl; state Exp; branches; next 1.61; 1.61 date 2003.11.06.12.58.21; author thl; state Exp; branches; next 1.60; 1.60 date 2003.03.24.15.16.00; author thl; state Exp; branches; next 1.59; 1.59 date 2003.01.27.16.01.35; author thl; state Exp; branches; next 1.58; 1.58 date 2003.01.06.11.41.51; author rse; state Exp; branches; next 1.57; 1.57 date 2002.07.29.11.45.18; author thl; state Exp; branches; next 1.56; 1.56 date 2002.07.25.07.50.27; author thl; state Exp; branches; next 1.55; 1.55 date 2002.05.28.12.19.47; author thl; state Exp; branches; next 1.54; 1.54 date 2002.01.03.16.21.20; author rse; state Exp; branches; next 1.53; 1.53 date 2002.01.03.15.23.19; author rse; state Exp; branches; next 1.52; 1.52 date 2002.01.03.11.54.25; author rse; state Exp; branches; next 1.51; 1.51 date 2001.12.22.22.25.34; author rse; state Exp; branches; next 1.50; 1.50 date 2001.12.22.22.24.40; author rse; state Exp; branches; next 1.49; 1.49 date 2001.12.22.22.23.43; author rse; state Exp; branches; next 1.48; 1.48 date 2001.12.14.12.42.41; author rse; state Exp; branches; next 1.47; 1.47 date 2001.12.08.20.50.36; author rse; state Exp; branches; next 1.46; 1.46 date 2001.12.04.08.28.05; author rse; state Exp; branches; next 1.45; 1.45 date 2001.12.03.10.53.19; author ms; state Exp; branches; next 1.44; 1.44 date 2001.11.30.17.12.54; author ms; state Exp; branches; next 1.43; 1.43 date 2001.11.16.19.40.55; author ms; state Exp; branches; next 1.42; 1.42 date 2001.11.10.14.04.05; author rse; state Exp; branches; next 1.41; 1.41 date 2001.11.08.10.03.47; author rse; state Exp; branches; next 1.40; 1.40 date 2001.11.07.16.25.56; author rse; state Exp; branches; next 1.39; 1.39 date 2001.11.07.16.17.09; author rse; state Exp; branches; next 1.38; 1.38 date 2001.11.07.09.28.45; author ms; state Exp; branches; next 1.37; 1.37 date 2001.11.04.13.58.56; author rse; state Exp; branches; next 1.36; 1.36 date 2001.11.04.13.21.17; author rse; state Exp; branches; next 1.35; 1.35 date 2001.10.26.10.54.26; author rse; state Exp; branches; next 1.34; 1.34 date 2001.10.23.12.33.28; author ms; state Exp; branches; next 1.33; 1.33 date 2001.10.23.10.05.53; author ms; state Exp; branches; next 1.32; 1.32 date 2001.10.22.15.25.37; author rse; state Exp; branches; next 1.31; 1.31 date 2001.10.19.16.32.05; author ms; state Exp; branches; next 1.30; 1.30 date 2001.10.14.06.21.51; author ms; state Exp; branches; next 1.29; 1.29 date 2001.10.12.12.17.50; author rse; state Exp; branches; next 1.28; 1.28 date 2001.10.05.10.40.17; author ms; state Exp; branches; next 1.27; 1.27 date 2001.09.28.14.30.49; author ms; state Exp; branches; next 1.26; 1.26 date 2001.09.26.18.16.13; author rse; state Exp; branches; next 1.25; 1.25 date 2001.09.21.17.29.47; author ms; state Exp; branches; next 1.24; 1.24 date 2001.09.20.16.26.56; author ms; state Exp; branches; next 1.23; 1.23 date 2001.09.18.14.38.51; author ms; state Exp; branches; next 1.22; 1.22 date 2001.09.14.13.27.17; author rse; state Exp; branches; next 1.21; 1.21 date 2001.09.14.07.44.34; author rse; state Exp; branches; next 1.20; 1.20 date 2001.09.13.19.21.21; author rse; state Exp; branches; next 1.19; 1.19 date 2001.09.12.18.43.38; author rse; state Exp; branches; next 1.18; 1.18 date 2001.09.11.12.55.16; author rse; state Exp; branches; next 1.17; 1.17 date 2001.09.11.12.32.02; author rse; state Exp; branches; next 1.16; 1.16 date 2001.09.11.10.17.23; author ms; state Exp; branches; next 1.15; 1.15 date 2001.09.09.15.59.46; author rse; state Exp; branches; next 1.14; 1.14 date 2001.09.06.14.37.53; author rse; state Exp; branches; next 1.13; 1.13 date 2001.09.03.12.16.44; author rse; state Exp; branches; next 1.12; 1.12 date 2001.09.02.15.32.21; author ms; state Exp; branches; next 1.11; 1.11 date 2001.08.26.12.59.55; author ms; state Exp; branches; next 1.10; 1.10 date 2001.08.24.15.15.22; author ms; state Exp; branches; next 1.9; 1.9 date 2001.08.22.18.09.17; author ms; state Exp; branches; next 1.8; 1.8 date 2001.08.21.13.00.57; author ms; state Exp; branches; next 1.7; 1.7 date 2001.08.15.07.41.54; author rse; state Exp; branches; next 1.6; 1.6 date 2001.05.26.08.07.55; author rse; state Exp; branches; next 1.5; 1.5 date 2001.05.26.08.06.26; author rse; state Exp; branches; next 1.4; 1.4 date 2001.05.23.21.44.32; author rse; state Exp; branches; next 1.3; 1.3 date 2001.05.19.10.23.32; author rse; state Exp; branches; next 1.2; 1.2 date 2001.05.12.09.27.23; author rse; state Exp; branches; next 1.1; 1.1 date 2001.05.10.19.46.01; author rse; state Exp; branches 1.1.1.1; next ; 1.1.1.1 date 2001.05.10.19.46.01; author rse; state Exp; branches; next ; desc @@ 1.62 log @fix PR#23: l2 does not terminate a message when it contains a newline @ text @ _ ___ ____ ____ ____ _ ____ |_|_ _ / _ \/ ___/ ___|| _ \ | |___ \ _|_||_| | | | \___ \___ \| |_) | | | __) | |_||_|_| | |_| |___) |__) | __/ | |/ __/ |_|_|_| \___/|____/____/|_| |_|_____| OSSP l2 -- Flexible Logging TODO ==== 20030611 thl Fix Problem where l2 is not thread safe when multiple threads use the same env as is true with with fsl. http://cvs.ossp.org/tktview?tn=24 20030324 thl The sanity checking in the l2_ch_syslog.c:hook_configure() implementation causes an artifical sensibility to the order of parameters when a l2spec is used. A possible solution could be to relocate the sanity checks into hook_open(). The following two lines show simple examples, the first one works, the second doesn't. $ echo "hello" | l2tool 'syslog(facility=mail, ident=info, remotehost="127.0.0.1", remoteport="514", target="remote")' #works $ echo "hello" | l2tool 'syslog(facility=mail, ident=info, target="remote", remotehost="127.0.0.1", remoteport="514")' #error 20030127 thl Fixed a bug in l2_channel.c:l2_channel_destroy() where chD was passing a pointer to l2_channel_downstream() to find a sibling of a previously free(3)d node. This caused a bus error on FreeBSD5 where free(3) filled the free area with nonzero data. All other implementations seem to keep the data and l2_channel_downstream() traversed through still valid pointers in unmalloc(3)ed areas. This caused openssh using fsl v1.0.6 to fail on FreeBSD5. The fix was implemented in l2_channel_destroy() through look-ahead. This is once again a basic iteration problem which can catch us in other places as well. This needs to be verified. Structure of channels and documentation [thl]. It should be possible to drag the documentation out of a channel's source code. Everything else is error prone and a documentation nightmare. Currently, most (noop and null doesn't) channels have a code segment that looks like /* feed and call generic parameter parsing engine */ L2_PARAM_SET(pa[0], size, INT, &cfg->bufsize); L2_PARAM_SET(pa[1], interval, INT, &cfg->bufinterval); L2_PARAM_SET(pa[2], levelflush, INT, &cfg->levelflush); L2_PARAM_END(pa[3]); l2_channel_env(ch, &env); I recommend the L2_PARAM_SET macro should be modified to - set the default value of each paramater, if any exists - force the internal structure name to follow the revealed name - declare a channel to be a filter or output - declare parameters optional or mandatory Also i recommend an organizational enforcement of putting the channel description before such a code segment and a short description (i.e. unit, 0=deactivate) of each parameter/value just behind the macro. This could change the example above to look something like: /* * The buffer channel buffers messages poured in from upper channels. * It flushes the messages down to the lower channels when the buffered * space exceeds the buffer size, when a given time interval is reached * (0 disables this feature) or when a newly arrived message has a * level that matches the levelflush mask. * /* feed and call generic parameter parsing engine */ L2_PARAM_SET(pa[0], size, INT, 4096); /* o: bytes */ L2_PARAM_SET(pa[1], interval, INT, 0 ); /* o: sec, 0=disable */ L2_PARAM_SET(pa[2], levelflush, INT, 0 ); /* o: levelmask */ L2_PARAM_END(pa[3]); l2_channel_env(ch, &env); /**/ Neh idea as thl discussed with rse a few minutes ago - l2 should calculate the global logging mask automatically based on the OR'ing the masks of all output channels. This should increase performance as useless requests are captured at an early stage. - l2 should provide a function to set the logging masks including the global logging mask manually. This would require a mechanism to re-calculate the maks at any time when manually set to 'auto'. - l2 should provide a function to get the logging masks including the global logging mask. This could be used by the caller to check if it is worth calling the actual log and allow him to bypass wholly code blocks which exist for logging purposes only. Next steps: - libl2syslog: OpenPKG fakesyslog is nasty, because it doesn't provide logging to multiple files or filter out some messages. Additionally the application has to be restarted in order to reopen the logfile which is nasty for MTAs like Postfix in case of very high loads (because they start again processing the queue from scratch). What we need is a new L2-based libl2syslog which maps from syslog(3) API to l2(3) API. We later can add the reopen feature in L2 or sends the messages via Unix Domain socket to an L2 daemon which in turn logs to targets via l2tool, etc. - signal and process handling (l2_env*) - asynchronous channel (l2_ch_async.c) - manual page (l2.pod) New Channels ---------------------------------------------- - l2_ch_rotfile for a cronolog-style file writing, i.e., it changes the file according to a strftime-based filename expansion. - l2_ch_bofh does something cool but is undocumented and has its source mangled/scrambled ;) - l2_ch_asynch spawns a new process and communicates with the stemming channels via shared memory. Might be the way to implement autoflush as well, or rather with its own shared memory and sub-process. - l2_ch_smart is a smart debug buffer channel which flushes only if errors occur. - l2_ch_repeat prints 'last message repeated 100 times.' - l2_ch_action is similar to pipe channel, but executes a given command conditionally of the incoming priority or a regex. -> Might be implemented by combining a pcre, filter, and pipe (ms) - l2_ch_proxy sniffs a tcp connection and dumps its traffic to standard out or to the next channel, allowing for parsing through an unknown protocol. - l2_ch_nntp for logging messages to NNTP (news) service. support post and feed modes - l2_ch_snmp logs a stream to an SNMP listener. - ls_ch_trigger pulls the trigger to flush downstream if a certain critera is met - l2_ch_ring keeps messages in a ring buffer until someone pulls the trigger (backtrace) Existing Channels ----------------------------------------- - l2_ch_socket bind and udp parameters need work. (RSE: why and what?) - l2_ch_socket consider taking a fd at configuration. - l2_ch_pcre is too large and might benefit from a smaller set of pattern matching logic. - l2_ch_syslog should inherit socket logic to directly implement syslog communication, allowing for remote syslog operations. - l2_ch_syslog should be trimmed to necessary functionality, not necessarily everything that the system syslog(3) implements. - l2_ch_irc has a problem pinging its host to stay alive. - l2_ch_prefix produces implicitly two log messages if a following buffer channel would not accumulate them. - l2_ch_prefix should have printf style without strftime. - l2_ch_prefix should apply __FILE__, __LINE__, (__FUNCTION__) as options in case we are building in custom striptease mode. - l2_ch_file should optionally support file locking via fcntl(3). (RSE: why? Unix guarranties that up to 512 bytes one can write atomically) - l2_ch_pipe needs an overhaul once l2_ch_async is implemented. Also needs a review for dangling descriptors. During configure only checks existance of command in non-shell mode. Remove hard coded 256 in exec arguments. - l2_ch_pipe should use a standardized url like prg:/path/to/program. - l2_ch_pipe might consider taking a fd at configuration. - l2_ch_buffer buffer is duplicated when the process forks. This means the buffer has to be flushed in advance or the content is dumped twice. If the buffer would remember the pid of the last writer, it could discard the contents of the buffer when the pid changes. This is because the parent retains the pid and the buffer content while the child changes the pid and discards the content. Should be an optionial feature. -> Can be immediately implemented (rse) - l2_ch_buffer needs a L2_OK and L2_OK_PASS consistency check - l2_ch_buffer should not use different param types in alarm and setitimer(2). - l2_ch_buffer should incorporate multiplexer logic to properly flush when multiple buffers are in use. Can be implemented via a environment level timer and a lowest common denominator alarm function. For example, tmr1 9secs; tmr2 6secs; envtmr expires every lcd(9,6) = 3secs - l2_ch_buffer should not implement an autoflush, rather L2 should have a multiplexed timer object to serve timer requests from incoming callbacks. Library-wide changes -------------------------------------- - Add nonreentrant log method like mm. - Add striptease target to Makefile. - l2tool needs an [addr:]port option - API access through the LogManager logm; - Jump to a callback in case of error, throw/exit. - Offer the option to reopen the logfile on each write. - Optimize log operations from log(..) {} to log(....); - Implicit level decision should default to >=, then follow with <= = ; * - '\r\n' Uebersetzung Bei Input \r und \n wegstrippen, und von jedem Output Channel entweder \r\n (smtp) oder nur \n wieder anhaengen lassen. Oder als Kanaloption, wobei \r, \r\n, \n, gleich String, oder gar nix. - Spec-facility To debug an application, sometimes it's overkill to log everything at DEBUG level. I see an improvement when an additional facility can be specified. Example: DEBUG/LMTP but don't care about NNTP in the lmtp2nntp program. Possibly could be implemented as a second mask. -> Needs more consideration before implementation should start (rse) - Prelogging L2 should log even before it is initialized. Maybe the log function should buffer everything as long as a NULL l2-context is passed and if ever a non-NULL context is passed every remembered message should be logged afterwards or if destroy/flush is executed with a NULL- context it should print the buffered stuff to stderr. -> Can be immediately implemented, but one has to be carefully here. I want to think about this a little bit more in-depth. (rse) - Config file Might be used with the previous prelogging principle in mind. Namely, a file that describes a set of channel specs for one or many configurations. Path is searched in the following order: 1. /etc/liblog.conf 2. (in ., .., ../..) 3. $HOME/.liblog.conf - hook_write methods should receive a null-termined string instead of buf+size, because some channels like syslog otherwise have to rebuffer the message and append the null terminator character. Spec-parsing ---------------------------------------------- - implement location tracking - target=remote has to be currently after host=xxx because of single-configuration procedure - bugfix l2tool and after it works 100% remove l2_test in favor of l2tool and add instead a test shell script which calls l2tool with various input specifications. Channel-Only Revamping ------------------------------------ - l2_objects update - syscall override ala OSSP SA in l2_env_t - perhaps rename l2_env to l2_ctx and l2_channel_ to just l2_ - API cleanup for open semantics Documentation --------------------------------------------- - l2_ch_buffer How the buffer object behaves in relation to up/downstream channels. When does it pass its data to the next channel, when does it erase, what happens to its data when it is over written or flushed. - l2_ch_syslog Describe the options, and mention that more info is found in the man page for syslog(3). Identify features that change from one system to the next. - l2 errors Describe how an l2 channel reports errors when it fails during an operation, and how a user should interpret the error message? Describe whether a channel continues passing data on downstream if it somehow fails, and if the data will be corrupt. - l2_util_fmt_string This can be used like %s, but instead of fetching only a "char *" from the var-args stack, it fetches a "char *" plus a "size_t" and this way allows one to log only a sub-string of a larger string without the need for any temporary buffers, etc. - l2_util_fmt_dump: This can be used as "%{type}X" for dumping arbitrary octets. The parameter "type" can be either "text" (the default if only "%X" is used) for dumping the octets as text but with non-printable characters replaced by "\xXX" constructs; "hex" for dumping the octets in hexadecimal as "XX:XX:XX:XX" or "base64" for dumping the octets Base64 encoded. All three are intended for making it easier to produce reasonable L2_LEVEL_DEBUG messages without having to fiddle around with temporary buffers and having to care with non-printable characters. - formatter example l2_stream_formatter(st, 'D', l2_util_fmt_dump, NULL); : l2_stream_vlog(st, L2_LEVEL_DEBUG, "%{text}D %{hex}D %{base64}D\n", "foo", 12345, "foo\1bar", 7, "foo\1bar", 7, "foo\1bar", 7); : ...produces "foo\x01bar 66:6f:6f:01:62:61:72 Zm9vAWJhcg==" High-level configuration interface ================================== Config-File (OSSP): logging { channel 0 null ALL; channel 1 prefix CUSTOM1 { prefix="%Y-%m-%d/%H:%M:%S" }; channel 2 buffer { size=8k }; channel 3 file { path=/path/to/ossp.stat.log; mode=0644 }; channel 4 prefix wmask=DEBUG { prefix="%Y-%m-%d/%H:%M:%S %L" }; channel 5 buffer { size=4k }; channel 6 file { path=/path/to/ossp.error.log; mode=0644 }; channel 7 syslog wmask=ERROR { peer=remote; host=loghost }; channel 8 filter { regex="(memory|foo)" }; channel 9 smtp { host=loghost; rcpt="rse@@engelschall.com" }; stream { 0->(4,7); 4->(5,7); 5->6; 8->9; 1->2->3 }; log access 0; log stat 1; }; Alternative: log access { error: syslog; prefix -> { buffer(size=4k) -> file(path=/path/to/log, mode=0644); panic: smtp(host=en1, rcpt=rse@@engelschall.com); } } Command-Line (lmtp2nntp): $ lmtp2nntp -l 'INFO:' { prefix(prefix="%Y-%m-%d/%H:%M:%S %L") ->{ buffer(size=4k) ->file(path=/path/to/log,mode=0644); smtp(host=en1,rcpt=rse@@engelschall.com) }; error:syslog } Brainstorming --------------------------------------------- - Debugging can be implemented as a special case of logging - Tracing can be implemented as a special case of debugging *** *** *** *** *** *** Out of date *** *** *** *** *** *** *** *** *** *** *** *** Out of date *** *** *** *** *** *** *** *** *** *** *** *** Out of date *** *** *** *** *** *** Backend channels ------------------------------------------ - Level -> N Channels - file (append) - program (stdin) - syslog - stderr/stdout - null (discard, nicht nur /dev/null) - filedescriptor (escape/ext) - callback function Aufbau ---------------------------------------------------- Layer C++ API log.hpp, log.cpp Layer C API log.h log.c Layer C Backend backend.h backend.c - Ein Wort noch zu variablen Argumentlisten in cpp-Makros: gcc unterstützt dies in in der GNU- und der C99-Ausführung. Das heißt, der "..." Parameter kann im Makro respektive über "args" und über "__VA_ARGS__" angesprochen werden. Wichtig ist dabei, daß "..." nicht leer sein -- also kein Argument enthalten -- darf, da sonst der Präprocessor an einem eventuell vorhandenen Komma scheitert. Dies kann beim gcc durch Voranstellen von "##" vor dem "__VA_ARGS__" umgangen werden. Ouch. Beide Erweiterungen sind derzeit nicht aktiv, wenn mit -ansi compiliert wird. Explizit anschalten läßt sich die standard-konforme Erweiterung über "-std=c9x", bzw. "-std=c99" bei neueren gccs. Log Messages: - raw - optional prefixes (inclusive of order) string facility level timestamp pid (tid) - errno (like syslog %m) - eigene %{foo}x mit callback function mit context - automatisch number -> string mapping (fuer error strings) - !debug -> !code - add support for prefixes of all non-static symbols to better support embedding @ 1.61 log @update bug and feature request list @ text @a12 7 Fix Problem where l2 does not terminate a message when it contains a \n so if previous message was longer the part between \n and the end of the previous message is repeated in the current message. http://cvs.ossp.org/tktview?tn=23 20030611 thl @ 1.60 log @sanity checking in the l2_ch_syslog.c:hook_configure() @ text @d12 12 d140 2 a141 1 - l2_ch_nntp for administration through news. d144 4 @ 1.59 log @fix iteration through non malloc(3)ed memory in l2_channel_destroy(). Bug caught on FreeBSD5 @ text @d12 9 @ 1.58 log @- remove trailing whitespaces - adjust copyright messages - consistently use "OSSP l2" - consistently talk about "Flexible Logging" - use standard OSSP ASCII-art @ text @d12 12 @ 1.57 log @embedding @ text @d1 10 a10 2 OSSP L2 TODO ============ d248 1 a248 1 d272 1 a272 1 channel 0 null ALL; d294 1 a294 1 Alternative: d297 2 a298 2 prefix -> { buffer(size=4k) -> @ 1.56 log @proposal to cleanup code and improve documentation @ text @d357 2 @ 1.55 log @disable debug code blocks; request to enhance l2; FIXME/TODO statement @ text @d4 40 a43 1 New Idea as thl discussed with rse a few minutes ago @ 1.54 log @update entries @ text @d4 15 @ 1.53 log @updates @ text @d5 11 a146 4 - An optional syslog(3) compatible API for converting syslog-only based applications (like sendmail) to (restricted) liblog-based applications. -> Des ist schon implimentiert in fakesyslog oder? (ms) a161 10 Related Projects ------------------------------------------ - Custom libsyslog OpenPKG fakesyslog is nasty, because it doesn't provide logging to multiple files or filter out some messages. Additionally the app has to be restarted in order to reopen the logfile which is nasty for MTAs like Postfix in case of very high loads (because they start again processign the queue from scratch). What we need is a new L2-based libsyslog.a which sends the stuff via Unix Domain socket to an L2 daemon which in turn logs to targets via L2. @ 1.52 log @Remember to document the formatter functions. PR: Submitted by: Reviewed by: Approved by: Obtained from: @ text @a86 1 - Replace l2_sockmon with netcat or peep. a88 1 - Provide a C++ implementation also. a89 1 - Adjust license to ISC/MIT/BSD. @ 1.51 log @remember issue @ text @d187 25 @ 1.50 log @add no case support @ text @d59 1 @ 1.49 log @remember issue @ text @a41 2 - l2_ch_filter should have a PCRE_CASELESS option. @ 1.48 log @done... puhhh! @ text @d39 1 a39 1 - l2_ch_socket bind and udp parameters need work. @ 1.47 log @remember next steps @ text @a4 1 - spec parser location tracking (l2_spec*) @ 1.46 log @L2 really needs cronolog functionality in order to fulfill user requirements @ text @d4 6 @ 1.45 log @Deleted items marked 'to be deleted.' @ text @d5 3 @ 1.44 log @Reorganization and clarification. @ text @a276 19 To be deleted --------------------------------------------- - C: log lh; lh = log_init(LOG_CFGFILE|LOG_CFGPARENT|LOG_XXX|..., "foo" (=facility)); log_configure(lh, "foo", LOG_WARN|LOG_LESSER, null); log_cb(lh, "x", func, ctx); int func(void *ctx, char *str, ...); log_msg(lh, LOG_WARN, "..%{foo}x %s...%E..", cp); log_dbg(lh, "..%{foo}x %s...%E..", cp); log_kill(lh); - Buffered I/O: fuer manche channels non-buffered (debug, errors) fuer manche andere aber buffered (access log, performance) loesung: I/O ueber callbacks (3x: open, write, close) z.B. RRDTool - Varargs: log ist nur wrapper fuer vlog @ 1.43 log @Mostly finished phase in of timer code to l2_env. Buffer needs work. @ text @d1 2 d4 134 a137 2 OSSP L2 ======= a138 2 Spec-Parsing: - location tracking d140 1 a140 1 single-configuration procedure d142 3 a144 2 - prefix channel produces implicitly two log messages if a following buffer channel would not accumulate them. Hmmm.. d146 2 a147 5 - bugfix l2tool and after it works 100% remove l2_test in favor of l2tool and add instead a test shell script which calls l2tool with various input specifications. Channel-Only Revamping: a148 1 - l2_objects.fig update d150 1 d152 27 a178 327 - Perhaps we should later also write an l2_ch_bofh.c which something very cool but is undocumented and has its source mangled/scrambled ;) - Problem: OpenPKG fakesyslog ist nasty, because it doesn't provide logging to multiple files or filter out some messages. Additionally the app has to be restarted in order to reopen the logfile which is nasty for MTAs like Postfix in case of very high loads (because they start again processign the queue from scratch). What we need is a new L2-based libsyslog.a which sends the stuff via Unix Domain socket to an L2 daemon which in turn logs to targets via L2. - "smart debug buffer channel": extra debug buffer, flush only if error occurs - "repeat channel": "last message repeated 100 times" - "\r\n" Problematik: Vorschlag: Bei Input \r und \n wegstrippen, und von jedem Output Channel entweder \r\n (smtp) oder nur \n wieder anhaengen lassen. - thl: log facility To debug an application, sometimes it's overkill to log everything at DEBUG level. I see an improvement when an additional facility can be specified. Example: DEBUG/LMTP but don't care about NNTP in the lmtp2nntp program. Possibly could be implemented as a second mask. -> needs more consideration before implementation should start (rse) - thl: buffer fork() awareness When the process forks, the buffer is duplicated. Currently this means the buffer has to be flushed in advance or the content is dumped twice. If the buffer would remember the pid of the last writer, it could discard the contents of the buffer when the pid changes. This is because the parent retains the pid and the buffer content while the child changes the pid and discards the content. Should be an optionial feature. -> can be immediately implemented (rse) - thl: prelog I want to log everything even things that happen before L2 is initialized. Complicated, i know. Maybe the log function should buffer everything as long as a NULL l2-context is passed and if ever a non-NULL context is passed every remembered message should be logged afterwards or if destroy/ flush is executed with a NULL-context it should print the buffered stuff to stderr. -> can be immediately implemented, but one has to be carefully here. Seems like I want to think about this a little bit more in-depth. (rse) o Another great rse idea, make a proxy server channel that sniffs a tcp connection and dumps its traffic to standard out. Optionally, it could dump this to the next channel, allowing for parsing through an unknown protocol. RSE: - channel API cleanup: open semantics - bind and udp parameters in socket channel - prefix channels: - printf style without strftime; - l2tool [addr:]port - autoflush via shared memory and sub-process? - NNTP channel - SNMP trap channel - perhaps replace too large PCRE stuff with smaller pattern matching stuff - file channel should optionally support file locking via fcntl(3). - idea of an asynch channel, that spawns a new process and communicates with the stemming channels via shared memory Lunchtime: - Correct DNS resolve blocking problem by using a funky asynchronous DNS lib. This leads to easy reimplementation of the prefix channel (asynchronous) MS: - netcat can probably replace l2_sockmon, remove from build and add netcat - pipe channel may need a big overhaul if we redesign it around the asynch channel principle - review pipe handler for dangling descriptors - configure only checks existance of command in non-shell mode - find alternative to exec arguments which is hard coded to 256 - signal handler chaining, save old signal handler and call it after our own - consider adding options such as PCRE_CASELESS to filter channel - implement "action" channel, can be based on pipe channel - correct problem with multiple buffer channels using the timer can be solved by running a signal handler at the environment level, then approximating the (multiple) user-chosen timer intervals by setting the environment timer to expire at the time of the least common denominator of the configured timer intervals ie: tmr1 9secs; tmr2 6secs; envtmr will expire every lcd(9,6) = 3secs alarm and setitimer methods are using different param types! Fixit. review newly introduced timer code for L2_OK and L2_OK_PASS inconsistency - solve problem with buffer (timer), irc (ping), and buffer (autoflush), by creating sleep/ping threads. The disadvantage is that we must depend on a pth installation, and force the parent app to be multithreaded. Alternatively, we can spawn a management process in l2_stream_create(), who owns management resources globally available to all channels. Or write the l2 mini-protocol :-( - split all timer server code out of L2 and into new multiplexed timer object probably must use callbacks, unless user signal space is occupied ISSUES ------ o hook_write's should perhaps receive a nul-termined string instead of buf+size, because syslog else has to re-buffer it in order to append the nul terminator character. o Stream Members: channels static array - consider dynamic o buffer user needs to know how a buffer object behaves in relation to up/downstream channels. When does it pass its data to the next channel, when does it erase, what happens to its data when it is over written or flushed... o buffer timer How on earth to use a C-style exception handler to flush our buffer? We can't directly call ANY function from an exception state, so this might not work at all. Depends on SIGALARM, only one handler of which may exist. A more robust implementation would not use such a precious resource, and guard against signal collision with other channels. o syslog many options need docu, and we should mention to the user that more info is found in the man page for syslog(), because after all that is what is doing all the work in our implementation. Also, can we really properly document these features if they change from one system's syslog to the next? o errors when a channel fails during an operation, how does it report this? How should a user interpret the error message or other data? Do we need more accurate or detailed error messages in the channel code? When a channel fails, does it continue passing data on to downstream channels? Is it corrupt data? o Syslog Kanal - Trim down to what will be used, right now the channel supports ALL functionality through syslog(3) BRAINSTORMING ------------- Braindump: - debugging is special case of logging - tracing is special case of debugging Channel Handler Configuration: o l2_handler_null - no configuration at all o l2_handler_fd - mode="unix|stdio" - fd=int|FILE* o l2_handler_file - mode="unix|stdio" - path=char* - append="yes|no" o l2_handler_pipe - url="prg:/path/to/program" - fd=int o l2_handler_socket - url="tcp://hostname:port" - fd=int o l2_handler_syslog - ident=char* - should have its own logic and not use unix lib syslog() thus able to write to a remote syslog daemon o l2_handler_filter - pattern=char* o l2_handler_prefix - prefix=char* o l2_handler_buffer - size=size_t o all output channels - should they have downstream, or be true endpoints? o l2_ch_socket - write should handle partial send() thus check the return of send o l2_ch_buffer - write() must implicitly flush() when incoming data is larger than remaining buffer capacity o l2_ch_action - new action channnel could accept a regexp or L2_LEVEL as configuration, and then run a callback function or exec a binary depending on incoming l2 stream messages. For example, lmtp2nntp could conditionally run code via the new action channel to monitor spam or alert an admin about a flawed OR CURIOUs LMTP header. However, the action channel is also appealling to other applications that don't run as daemons or at the system level. - l2_channel_config(l2Act, "regexp,run", "(nobody|cz)", "callback"); - l2_channel_config(l2Act, "funcptr", pfnAddtcpwrap); License: - ISC/MIT/BSD Sprache: - C++ - C Aufbau: 1. Layer C++ API log.hpp, log.cpp 2. Layer C API log.h log.c 3. Layer C Backend backend.h backend.c - "make striptease" - optimierung: log(..) { } : log(....); : API Levels: - PANIC (-> LOG_EMERG) - CRITICAL (-> LOG_CRIT) - ERROR (-> LOG_ERR) - WARNING - NOTICE - INFO - TRACE (-> LOG_DEBUG) - DEBUG (-> LOG_DEBUG) - ALERT Level Entscheidungen: >= (default) <= = ; * Backend Channels: 1 Level -> N Channels - file (append) - program (stdin) - syslog - stderr/stdout - null (discard, nicht nur /dev/null) - filedescriptor (escape/ext) - callback function Log Messages: - raw - optional prefixes (inclusive order): string facility level timestamp pid (tid) - errno (like syslog %m) - eigene %{foo}x mit callback function mit context - automatisch: number -> string mapping (fuer error strings) - __FILE__, __LINE__, (__FUNCTION__) Configuration: - ueber C/C++ API - zusaetzlich Config-File 1. /etc/liblog.conf 2. (in ., .., ../..) 3. $HOME/.liblog.conf - !debug -> !code API C (ala MM): - reentrant: log_xxx - non-reentrant: Log_xxx Message Filtering/Masking: - facility und/oder levels und/oder wildcard pattern API Using: - C++: LogManager logm; logm.debug1("test"); logm.configure(" - C: log lh; lh = log_init(LOG_CFGFILE|LOG_CFGPARENT|LOG_XXX|..., "foo" (=facility)); log_configure(lh, "foo", LOG_WARN|LOG_LESSER, null); log_cb(lh, "x", func, ctx); int func(void *ctx, char *str, ...); log_msg(lh, LOG_WARN, "..%{foo}x %s...%E..", cp); log_dbg(lh, "..%{foo}x %s...%E..", cp); log_kill(lh); - Buffered I/O: fuer manche channels non-buffered (debug, errors) fuer manche andere aber buffered (access log, performance) loesung: I/O ueber callbacks (3x: open, write, close) z.B. RRDTool - Varargs: log ist nur wrapper fuer vlog - Error Handling: o log kein Return Code o aber error callback function (dadrin in C++: throw, in C: exit) - Newline Handling: option fuer channel: \r, \r\n, \n oder gleich string und moeglichkeit gar nix (string="") - Perhaps: optionally reopen logfile on each write - An optional syslog(3) compatible API for converting syslog-only based applications (like sendmail) to (restricted) liblog-based applications. - Ein Wort noch zu variablen Argumentlisten in cpp-Makros: gcc unterstützt dies in in der GNU- und der C99-Ausführung. Das heißt, der "..." Parameter kann im Makro respektive über "args" und über "__VA_ARGS__" angesprochen werden. Wichtig ist dabei, daß "..." nicht leer sein -- also kein Argument enthalten -- darf, da sonst der Präprocessor an einem eventuell vorhandenen Komma scheitert. Dies kann beim gcc durch Voranstellen von "##" vor dem "__VA_ARGS__" umgangen werden. Ouch. Beide Erweiterungen sind derzeit nicht aktiv, wenn mit -ansi compiliert wird. Explizit anschalten läßt sich die standard-konforme Erweiterung über "-std=c9x", bzw. "-std=c99" bei neueren gccs. d180 1 a180 1 High-Level Configuration Interface a181 1 a182 1 a203 1 a206 1 a216 1 d225 70 @ 1.42 log @remember idea @ text @d94 1 d104 7 d117 2 @ 1.41 log @fix old prefix problem, but find new one :-( @ text @d13 4 @ 1.40 log @remember issue @ text @d9 3 a11 1 - prefix channels seems not to do anything?! @ 1.39 log @HEADS UP, guys: Here comes the first cut for the long-awaited channel tree specification parser and channel tree builder. This certainly makes L2 finally _THE_ killer library in the logging field. Woohooo... and the crowd goes wild! It allows one to build an arbitrary complex logging channel tree out of a single textual specification. An example follows: noop -> { prefix(prefix="[%d-%m-%Y/%H:%M:%S] %L test[%P]: ", timezone="local") -> filter(regex="hecking", negate="0") -> buffer(size="800") -> file(path="l2_test.log",append="1",perm="420"); error: syslog(ident="L2-Test", facility="user", remotehost="en1", logpid="1", target="remote"); panic: smtp(rcpt="rse@@engelschall.com", host="en1", port="25" } This allows one to log nice timestamp-prefixed messages containing the word "hecking" to a buffered file. Additionally if the message has a level higher or equal to "error" it also logs it remotely via UDP to the syslogd on en1. And additionally if the message is a panic message, it is also sent out as an Email via SMTP to the MTA on en1. Ever thought a C library has to be dull and simple? ;) PS: This stuff certainly needs more polishing and cleanup and also a few things I'll enhance in the future (for instance to remove the restriction that parameter values have to be in quotation marks, etc.). @ text @d9 1 @ 1.38 log @Good group lunchtime idea. @ text @d5 5 a383 38 Grammar ======= hierarchy : stream | '{' stream_list '}' ; stream_list : hierarchy | hierarchy ';' stream_list ; stream : channel | channel "->" hierarchy ; channel : channel_mask ':' channel_spec | channel_spec ; channel_mask : T_ID_LEVEL | '(' channel_levels ')' ; channel_levels : T_ID_LEVEL | T_ID_LEVEL '|' channel_mask_list ; channel_spec : T_ID_CHANNEL | T_ID_CHANNEL '(' channel_params ')' ; channel_params : channel_param | channel_param ',' channel_params ; channel_param : T_ID_PARAM '=' T_STRING ; @ 1.37 log @remove now solved issues @ text @d77 4 @ 1.36 log @Channel-Only Revamping Step 2: - moved code of l2_stream.c into (new) l2_env.c and l2_channel.c - created new l2_env_t and l2_env_xxx() - changed l2_xx_create() functions to also return l2_result_t - moved error handling into l2_env_t - replaced l2_channel_stack() with two new and more flexible l2_channel_link() and l2_channel_unlink() functions - rewritten test stuff in l2_test.c to use new structure - added new l2_channel_env() function for retriving l2_env_t Puhhh.... @ text @a63 1 - socket channel convert to use SA library a67 1 - IRC channel a69 3 - perhaps generalize channels and get rid of stream by putting formatters into external object referenced by channels and by providing a T-channel at the top. a73 2 - syslog channel should support direct remote logging - implement "filter" channel @ 1.35 log @remember ideas @ text @d5 5 @ 1.34 log @New ideas and dreams about asynch, pipe, shared memory, and multithreading. @ text @d5 4 @ 1.33 log @Implemented half-solution to occupied timer problem. @ text @d72 2 d76 2 a77 1 - implement pipe channel d83 1 a83 1 - implement "action" channel d85 6 @ 1.32 log @remember a view ideas @ text @d81 1 @ 1.31 log @Added Ralf's idea of l2_ch_proxy. @ text @d5 17 @ 1.30 log @Idea of action channel to conditionally run user code depending on L2 message stream content. @ text @d31 4 @ 1.29 log @remember Thomas' issues @ text @d59 1 d151 10 @ 1.28 log @Added untested code to implement alarm exceptions in the buffer channel. The buffer will auto-flush after a user specified timeout if it is even possible to do this with a c-style exception handler. @ text @d5 26 @ 1.27 log @Remember adding options to the filter channel. @ text @d51 9 @ 1.26 log @brain dump to remember my L2 thoughts of today @ text @d32 1 @ 1.25 log @Remember chaining the signals if we get that far. @ text @d244 89 @ 1.24 log @Debugging of channel's behavior logic when facing problematic children. @ text @d31 1 @ 1.23 log @Pipe channel handler redesign, bugfixes, and cleanup. @ text @d29 2 @ 1.22 log @SMTP channel is now complete @ text @d28 1 @ 1.21 log @update TODO @ text @a8 1 - smtp channel: MIME quoted printable? @ 1.20 log @update TODO @ text @d8 1 a11 1 - levels logged with l2_util_l2s() d22 7 @ 1.19 log @remember a few things @ text @a7 1 - "subdir config then quiet" feature a19 1 - l2_stream_flush() @ 1.18 log @do not forget interesting channel ideas @ text @d18 6 @ 1.17 log @update TODO list @ text @d15 3 @ 1.16 log @Added idea of rewriting the syslog write hook. @ text @a4 9 Idea: l2_handler_t: - function pointers - sizeof(handler struct) - table of config callsbacks which use offsetof into handler struct Auf diese Weise ist l2_channel_setparams aus API draussen und.. d16 6 a21 2 Timeplan -------- d23 2 a24 13 o M1 - fixed C API - established build environment o M2 - implemented C API - fixed C++ API o M3 - implemented C++ API - documented C API - documented C++ API - release version 0.9.0 o M4 - release version 1.0.0 d26 6 a31 2 ATTENTION --------- d33 7 a39 2 o hook_write's should perhaps receive a nul-termined string instead of buf+size, because syslog else has to re-buffer it. d41 8 a48 2 QUESTIONS --------- d50 3 a52 2 o Should the following line copy the string or just use the reference? L2_PARAM_SET(pa[0], ident, CHARPTR, &cfg->pszIdent); d57 4 a96 38 Stream Members: o channels static array - consider dynamic Documentation: o buffer user needs to know how a buffer object behaves in relation to up/downstream channels. When does it pass its data to the next channel, when does it erase, what happens to its data when it is over written or flushed... o syslog many options need docu, and we should mention to the user that more info is found in the man page for syslog(), because after all that is what is doing all the work in our implementation. Also, can we really properly document these features if they change from one system's syslog to the next? o errors when a channel fails during an operation, how does it report this? How should a user interpret the error message or other data? Do we need more accurate or detailed error messages in the channel code? When a channel fails, does it continue passing data on to downstream channels? Is it corrupt data? Braindump: - debugging is special case of logging - tracing is special case of debugging Kanalen: o Syslog Kanal - Trim down to what will be used, right now the channel supports ALL functionality through syslog(3) @ 1.15 log @remember my personal points @ text @d75 4 @ 1.14 log @Replace generic L2_ERROR with more granular L2_ERR_XXX and make sure that we always check with "!= L2_OK". @ text @d14 11 @ 1.13 log @revamped L2 stream API @ text @a4 2 Improvement: return L2_ERR_XXX instead of just a single L2_ERROR @ 1.12 log @Added remarks about buffer write, socket send, and output endpoint. @ text @d5 2 @ 1.11 log @Added syslog channel remarks and question about using a pointer reference. @ text @d70 8 @ 1.10 log @Added one more loggin level definition. @ text @d37 6 d83 8 d103 5 @ 1.9 log @Added two important things to remember - Error msgs and Doku. @ text @d120 1 @ 1.8 log @A stream could grow its member channels dynamically. @ text @d69 17 @ 1.7 log @add one idea which we could follow later... @ text @d65 4 @ 1.6 log @remember things @ text @d5 8 @ 1.5 log @forget timestamps now that I've to do it myself and we are now in holiday for two weeks @ text @d23 2 a24 2 TODO ---- d26 2 a27 6 - channel backends - message formatting - rc-file autoconfiguration - C++ API - documentation C API - documentation C++ API @ 1.4 log @test commit @ text @d9 1 a9 1 o M1 (14.05.2001) d12 1 a12 1 o M2 (21.05.2001) d15 1 a15 1 o M3 (11.06.2001) d20 1 a20 1 o M4 (25.06.2001) @ 1.3 log @Remember timeplan @ text @d5 1 @ 1.2 log @update todo @ text @d5 17 @ 1.1 log @Initial revision @ text @d2 40 a41 2 OSSP liblog =========== a148 20 - Source Tree: README .......... information about package INSTALL ......... instructions for installation LICENSE ......... license information TODO ............ to-do list ChangeLog ....... history of all changes Makefile.in ..... build specification configure.in .... GNU Autoconf script source aclocal.m4 ...... GNU Autoconf internal macros libtool ......... GNU Libtool utility shtool .......... GNU Shtool utility l2.m4 ........... GNU Autoconf public macro AC_L2() l2-config.in .... configuration utility script l2.h ............ C API definition l2.c ............ C API+internals implementation l2.hh ........... C++ API definition l2.cc ........... C++ API implementation l2.pod .......... Unix manual page source (POD format) l2_test.c ....... Test application d151 1 a151 1 o aber error callback function (dadrin in C++: throw, in C: exit) a170 1 @ 1.1.1.1 log @L2 initial source tree @ text @@