Logging with syslog-ng

Managing Linux system and application logging is important and a bit tricky. You want to capture important information, not bales of noise. You need to be able to find what you want in your logs without making it your life's work. The venerable old syslogd has served nobly for many years, but it's not quite up to meeting more complex needs. For this we have the next-generation Linux logger, syslog-ng.

Syslog-ng has a number of advantages over syslogd: better networking support, highly-configurable filters, centralized network logging, and lots more flexibility. For example, with syslogd all iptables messages get dumped in kern.log along with all the other kernel messages. Syslog-ng lets you direct iptables messages to a separate logfile. Syslogd uses only UDP; syslog-ng runs over UDP and TCP, so you can run it over encrypted network tunnels to a central logging server.

Installation
Syslog-ng is a standard package on all Linuxes. The current stable release is 1.6. If you are using apt-get or yum sysklogd should be removed automatically. But if it isn't, make sure to remove it and to stop the syslogd daemon. (sysklogd is the package name, syslogd is the daemon.)

Configuration File Structure
With all of this flexibility comes a bit of a learning curve. Syslog-ng's configuration file is /etc/syslog-ng/syslog-ng.conf, or on some systems /etc/syslog-ng.conf. You'll need man 5 syslog-ng.conf to understand all the different options and parameters. syslog-ng.conf has five possible sections:

options{}
Global options. These can be overridden in any of the next four sections

source{}
Message sources, such as files, local sockets, or remote hosts

destination{}
Message destinations, such as files, local sockets, or remote hosts

filter{}
Filters are powerful and flexible; you can filter on any aspect of a log message, such as standard syslogd facility names (man 5 syslog.conf), log level, hostname, and arbitrary contents like words or number strings

log{}
Log statements connect the source, destination, and filter statements, and tell syslog-ng what to do with them

Here are some typical global options:

options {

sync (0);
log_fifo_size (2048);
create_dirs (yes);
group (logs);
dir_group (logs);
perm (0640);
dir_perm (0750);
};

Options statements must use options as defined in /etc/syslog-ng.conf. This what the examples mean:

sync
How many lines of messages to keep in the output queue before writing to disk. (Logging output is called "messages.") Zero is the preferred option, since you want to be sure to capture everything, and not risk losing data in the event of a power failure

log_fifo_size
The maximum number of lines of messages in the output queue. The default is 100 lines of messages. You can do some calculations to figure out a suitable value, as this syslog-ng mail list post shows. To quote the relevant bit:

Each message source receives maximum 1024 bytes of data. A single log message is about 20 to 30 bytes. So on a single run each log source may emit 1024/20 = 50 messages. Your log fifo must be able to hold this number of messages for each source. So for two sources, you'll need at least 100 slots in your fifo.

It is easy to get confused by these two options. Incoming messages from remote hosts arrive in bursts, so what you need to do is make sure your log_fifo_size is large enough to handle these bursts. Otherwise you risk losing messages. Ultimately you will be limited by i/o and network speeds

create_dirs
Enable or disable automatic directory creation for destination files. In this article this value is "yes", so that remote host directories can be created on demand

group
dir_group
Set the group owner of logfiles and directories, so you don't have to read and analyze logs as the superuser

perm
dir_perm
Default permissions on new files and directories

Source, Destination, and Filter Statements
Source, destination, and filter names are arbitrary, like these examples show:

source s_local { internal(); unix-stream("/dev/log"); file("/proc/kmsg" log_prefix("kernel: ")); };
destination d_auth { file("/var/log/auth.log"); };
filter f_auth { facility(auth, authpriv); };

So "source s_local" could be "source frederick_remington_depew" if you so desired, and "destination d_auth" could be "destination moon." The convention is to prefix source names with "s", destination names with "d", and filter names with "f", but this is not required. All other elements of source, destination, and filter statements must use parameters as defined in man 5 syslog-ng.conf.

The "source s_local" example collects all locally-generated log messages into a single source statement. The "destination d_auth" statement directs authorization messages to /var/log/auth.log, which are defined by the "filter f_auth" statement. auth and authpriv are standard syslog facility names.

Log statements pull it all together:

log {source(s_local); filter(f_auth); destination(d_auth); };

So these four lines filter out all the auth and authpriv messages from all local messages, and write them to /var/log/auth.log.

Enabling Remote Logging
While it's possible to send log messages from remote clients with good old syslogd, it's really not adequate because it only transmits UDP packets. So you need syslog-ng installed on all client hosts as well. Adding these lines to syslog-ng.conf on the server accepts remote messages from clients and dumps them into a single file per host:

source s_remote { tcp(); };
destination d_clients { file("/var/log/HOSTS/$HOST"); };
log { source(s_remote); destination(d_clients); };

This is a very simple, but functional example for your client hosts that collects all local messages and sends them to the remote server:

#sample syslog-ng.conf for a remote client
source s_local { internal(); unix-stream("/dev/log"); file("/proc/kmsg" log_prefix("kernel: ")); };
destination d_loghost {tcp("192.168.1.10" port(514));};
log { source(s_local); destination(loghost); };

Sample syslog-ng.conf
A complete sample configuration file is a bit long to include here, so you should take a look at the one that came with your syslog-ng installation. Debian users get a customized file that replicates a syslogd setup. Let's put together our examples here in a single file for the server, set up a remote client, and run some tests to see how it works:

#sample syslog-ng.conf for a central logging server
options {

sync (0);
log_fifo_size (2048);
create_dirs (yes);
group (logs);
dir_group (logs);
perm (0640);
dir_perm (0750);
};

source s_local { internal(); unix-stream("/dev/log"); file("/proc/kmsg" log_prefix("kernel: ")); };
destination d_auth { file("/var/log/auth.log"); };
filter f_auth { facility(auth, authpriv); };

source s_remote { tcp(); };
destination d_clients { file("/var/log/HOSTS/$HOST"); };

log { source(s_remote); destination(d_clients); };
log { source(s_local); filter(f_auth); destination(d_auth); };

Whenever you make changes to syslog-ng.conf you must restart it:

# /etc/init.d/syslog-ng restart

Testing Everything
Now you can runs some simple tests on both the server and the client. Since the only local server messages that are going to be logged are authorization messages, you can test these by opening a new login session on the server, or running su or sudo. Then check /var/log/auth.log. Test the client by doing anything, then see if a new directory was created for the remote client in /var/log/HOSTS.

Another way is to use the useful and excellent logger command:

# logger "this is a test"
# logger -p auth.debug "this is a test"

This will create a line like this in your logfiles:

Apr 1 16:08:42 localhost.localdomain logger: this is a test

Now that we have a grasp of syslog-ng basics, come back next week to learn how to fine-tune and organize syslog-ng just the way you like, for both local and remote logging, and how to securely encrypt all syslog-ng network transmissions.