syslog-ng v2.0 reference manual

One of the new features of syslog-ng 2.0 is the support for messages originating from different timezones. This is a difficult problem, as the original syslog protocol does not include timezone information. syslog-ng provides a solution by extending the syslog protocol to include this information and also by giving finegrained control to the administrator to supply timezone information for legacy devices which do not support the protocol extension.

Timezone information is associated with messages entering syslog-ng using the following algorithm:

There is another case when message timestamps are formatted:

A source is a set of source drivers collecting messages using a given method. For instance, there is a source driver for AF_UNIX, SOCK_STREAM style sockets, used by the Linux syslog() call.

To declare a source, the source statement has to be used in the configuration file with the following syntax:

					source <identifier> { source-driver(params); source-driver(params); ... };
				

The identifier has to uniquely identify the given source, and may not clash with any of the reserved words (in case of a name clash, simply enclose the identifier in quotation marks).

You can control exactly which drivers are used to gather log messages, thus you have to know how your system and its native syslogd communicate. Below is an introduction to the inner workings of syslogd on some of the tested platforms:

Each possible communication mechanism has the corresponding source driver in syslog-ng. For instance, to open a unix socket with SOCK_DGRAM style communication use the driver unix-dgram, the same with SOCK_STREAM style - as used under Linux - is called unix-stream.

					source src { unix-stream("/dev/log"); internal(); udp(ip(0.0.0.0) port(514)); };
				

Each driver may take parameters; some of them are required, others are optional. The required parameters are positional, meaning that they must be specified in a defined order. A unix-stream() driver has a single required argument, the name of the socket to listen to, and several optional parameters, which follow the socket name. Optional arguments can be specified in any order using the option(value) syntax.

For a complete descriptions on the above drivers, see Section 1, “Source drivers”.

Filters perform log routing within syslog-ng. You can write a boolean expression using internal functions: a message passes if the expression is true.

Filters also have a unique identifying name that can be referenced in log statements.

Syntax for the filter statement:

				filter <identifier> { expression; };
			

An expression may contain parentheses, the boolean operators "and", "or" and "not", and any of the functions listed in Table 3.10, “Available filter functions in syslog-ng”.

					filter f_blurp_deny { host("foo") and match("deny"); };
				

For a complete description on the above functions, see Section 3, “Filter functions”.

In earlier revisions of syslog-ng there was a special filter identifier, "DEFAULT", which matched all not-yet-matched messages. This could make your configuration much simpler and easier to manage. This feature was removed in syslog-ng 1.5.x, and a more powerful idea was introduced. For more details consult Section 5, “Log paths”.

A destination is where a log is sent if the filtering rules match. Similarly to sources, destinations are comprised of one or more drivers, each defining how messages are handled. Destinations can be declared in the configuration file via a destination statement using the syntax below:

					destination <identifier> { destination-driver(params); destination-driver(params); ... };
				

For detailed description of the supported drivers, see Section 2, “Destination drivers”.

syslog-ng 2.0 allows you to define common templates, and refer to them from every object requiring a template. For example:

          template t_filetmpl { template("$ISODATE $HOST $MSG\n"); template_escape(no)); };
          
          destination d_file { file("/var/log/messages" template(t_filetmpl); };
        

Templates can reference one or more macros as described in Section 4, “Macros”.

The previous chapters described how to define sources, filters and destinations. These components have to be connected together using log statements. The syntax of log statements is described below:

				log { source(s1); source(s2); ... 
				filter(f1); filter(f2); ... 
				destination(d1); destination(d2); ... 
				flags(flag1[, flag2...]); };
			

Any message coming from any of the listed sources, matching all the filters is sent to all listed destinations. Log statements are processed in the order they appear in the configuration file.

By default, all matching log statements are processed, therefore a single log message might be sent to the same destination several times, provided the destination is listed in several log statements.

This default behavior can be changed using the flags() parameter.

There are several options that can modify the behavior of syslog-ng. For an exact list of possible options see Section 5, “Options”. Each option may have parameters, similarly to driver specifications. The general syntax is:

				options { option1(params); option2(params); ... };
			

The following drivers may be used in source statements, as described in Section 1, “Sources”. The option log_msg_size() is available in each source: it specifies the maximum length of incoming log messages in bytes. If not specified, the value of the global option is used (see Section 5, “Options”).

Some parameters affecting message parsing are common for all sources:

					Declaration: internal()
				

						source s_local { internal(); };
					

						Declaration: 
						unix-stream(filename [options]);
						unix-dgram(filename [options]); 
					

						# source declaration on Linux
						source s_stream { unix-stream("/dev/log" max-connections(10)); };
						
						# source declaration on BSD
						source s_dgram { unix-dgram("/var/run/log"); };						
					

					Declaration:
					tcp([options]);
					udp([options]);
				

						source s_tcp { tcp(ip(127.0.0.1) port(1999) max-connections(10)); };
						source s_udp { udp(); };
					

						#!/bin/sh
						tail -f logfile | logger -p local4.info
					

					Declaration:
					file(filename);
				

						source s_file { file("/proc/kmsg" log_prefix("kernel: ")); };
					

					Declaration:
					pipe(filename);
				

						source s_pipe { pipe("/dev/log" pad_size(2048)); };
					

						source s_stream { sun-streams("/dev/log" door("/etc/.syslog_door"); };
					

Destination drivers output log messages to somewhere outside syslog-ng: a file or a network socket.

Some of the parameters affecting message formatting and sending are common for all destinations.

						destination d_file { file("/var/log/messages" ); };
					

						destination d_file {
						file("/var/log/$YEAR.$MONTH.$DAY/messages"
						template("$HOUR:$MIN:$SEC $TZ $HOST [$LEVEL] $MSG $MSG\n")
						template_escape(no)
						);
						};
					

					Declaration:
					pipe(filename);
				

						destination d_pipe { pipe("/dev/xconsole"); };
					

					Declaration: 
					unix-stream(filename [options]);
					unix-dgram(filename [options]); 
				

						destination d_unix_stream { unix-stream("/var/run/logs"); };
					

					Declaration:
					tcp(host [options]);
					udp(host [options]);
					tcp6(host [options]);
					udp6(host [options]);					
				

						destination d_tcp { tcp("10.1.2.3" port(1999); localport(999)); };
					

					Declaration: 
					usertty(username);
				

						destination d_usertty { usertty("root"); };
					

					Declaration: 
					program(commandtorun);
				

						destination d_prg { program("/bin/cat >/dev/null"); };
					

The following functions may be used in the filter statement, as described in Section 2, “Filters”.

Certain parts of syslog-ng (e.g.: destination filenames and message content templates) can refer to one or more macros, which get expanded as a message is processed. The table below summarizes the macros available in syslog-ng.

Macros can be included by prefixing the macro name with a $ sign, just like in Bourne compatible shells. syslog-ng 2.0 uses a new syntax for braces around macro names: "$MSG" and "${MSG}" are equivalent.

The macros related to the time of the message (e.g.: ISODATE, HOUR, etc.) have two further versions each: one with the S_ and one with the R_ prefix (e.g.: S_DATE and R_DATE ). The S_DATE macro represents the date found in the log message, i.e. when the message was sent by the original application. R_DATE is the date when syslog has received the message. DATE equals either S_DATE or R_DATE, depending on the global option set in the now deprecated use_time_recvd() parameter (see Section 5, “Options”).

The following options can be specified in the options statement, as described in Section 6, “Options”.

Syslog-ng 2.0 is a complete reimplementation of syslog-ng 1.6, and does not use the mark and sweep garbage collector at all. The garbage collector parameters (gc_idle_threshold, gc_busy_threshold) are still accepted but are ignored.

When there are a lot of parallel connections to syslog-ng, the amount of time required to prepare for a single poll loop iteration is significant. Since the arrival of every single log message triggers a new iteration, the CPU usage of syslog-ng might increase significantly even if the number of messages received is otherwise low. The solution is to add a fixed latency to message processing to wait some messages to arrive and to process them in blocks. This can be accomplished by setting the time_sleep() to a non-zero value. It is not recommended to increase it above 100ms, as that might skew timestamps, slow syslog-ng, and cause messages to be dropped. An alternative solution is to avoid tcp() and unix-stream() sources and use the DGRAM versions, udp() and unix-dgram(), respectively.

The value of fetch_limit() and log_fifo_size() must also be sized accordingly when time_sleep() is modified.

syslog-ng always reads its incoming log channels to prevent the running daemons from blocking. This may result in lost messages if the output queue is full, it is therefore important to set the output queue size (termed in number of messages). The size of the output queue can be set either globally, or on a per destination basis.

			options { log_fifo_size(1000); };
			

or

				destination d_messages { file("/var/log/messages" log_fifo_size(1000)); };
			

You should set your fifo size to the estimated number of messages in a message burst. If bursts extend the bandwidth of your destination pipe, syslog-ng can feed messages into the destination pipe after the burst has collapsed.

Of course syslog-ng cannot widen the network bandwidth, so if the destination host lives on a noisy network and the logtraffic extends the bandwidth of this network, syslog-ng cannot do anything. However, it will do its best.

The sync parameter does not exactly do what its name implies. As you have seen messages to be sent are buffered in an output queue. The sync parameter specifies the number of messages held in this buffer before anything is written.

Note that it does not write all buffered messages into a single chunk; each distinct message is written with a single write() system call.