Snmpd filling up /var/log/messages

Update May 2009: This post has generated lots of alternative ideas in the comments so make sure you read through them to see what might work for your server.

At work we have a central monitoring system for servers called Solarwinds Orion Network Manager, this uses standard snmp connections to servers to get their status, disk usage, CPU performance.

On my RHEL5 linux servers the standard snmpd daemon works well with Solarwinds but the monitoring server seems to make a lot of connections to the system and each one gets logged via the syslog daemon to /var/log/messages giving rise to lots of lines saying things like

snmpd[345435]: Connection from UDP: [10.225.46.136]:135

last message repeated 8 times

last message repeated 13 times

These are only information messages saying a connection has been established. This is rather annoying when you are trying to read other things in /var/log/messages. The way to turn off these messages is to change the logging options of the snmpd daemons.

On Redhat ( and Ubuntu) the default logging ( the -L options ) show:–

-Ls d

Meaning log to syslog using the facility of daemon ( see syslogd and syslog.conf for more information on what that means in detail, for now suffice it to say it means all messages are written to /var/log/messages ).

The man pages for snmpcmd ( common to all net-snmp programmes ) explain you can set this to only log messages above a certain priority.

Using priorities 0-4 means warning messages, errors, alerts and critical etc messages are logged but notice info and debug level messages are ignored.

The manual pages are not that clear, to me at least at first, hence this blog.

So if we change the -Ls d to the following this will stop those messages but still allow important messages to get through:–

LS 0-4 d

The capital S is crucial to the syntax.

So where and how do we set these options? Well the snmpd daemon is started by a standard init script /etc/init.d/snmpd

In both RHEL5 and Ubuntu the scripts have some default options but also read in settings from a config file. In Ubuntu the relevant portion of the script is:-

SNMPDOPTS=’-Lsd -Lf /dev/null -p /var/run/snmpd.pid’
TRAPDRUN=no
TRAPDOPTS=’-Lsd -p /var/run/snmptrapd.pid’
#Reads config file (will override defaults above)
[ -r /etc/default/snmpd] && . /etc/default/snmpd

So this sets the variable SNMPDOPTS to the default value and then if the file /etc/default/snmpd is readable it “sources” the content of that file.

Thus if /etc/default/snmpd contains the line

SNMPDOPTS='-LS 0-4 d -Lf /dev/null -p /var/run/snmpd.pid'

Then stopping and starting the snmpd daemon will make it run with the new logging options we want.

sudo /etc/init.d/snmpd restart

In RHEL5 the equivalent file is /etc/snmp/snmpd.options and the equivalent variable is OPTIONS rather than SNMPDOPTS

Now there could be security implications to not recording the IP address of every SNMP request on your server in case some other system is connecting that shouldn’t be, but there are ways with community strings and other authentication options for SNMP to reduce the risk of that.

All in all the I think the risk of missing an important message in /var/log/messages outweighs the risks from not logging the snmpd messages.

Hey look a whole post and I never mentioned FTP once )