Operation

Monitoring

Fuglu has various places where you can see what it is is currently doing (eg. for debugging purposes) or what it has done (for statistical analysis)

Command Line Interface

The fuglu_control command offers various options to get real-time information on fuglu internals

statistics:

#$ fuglu_control stats
Fuglu statistics
---------------
Uptime:         19:33:56.119661
Avg scan time:  3.9657
Total msgs:     86667
Ham:            547
Spam:           86124
Virus:          8

Status of currently running scanner threads:

#fuglu_control workerlist
Total 10 Threads

[98]: Suspect 8790fa16a3733350baf02d5b4e7d98e1: from=<redacted> to=<redacted> size=1043 , spam=no, virus=no tags={'virus': {'F-Prot': False, 'ClamAV': False}, 'FprotPlugin.time': '0.0080', 'spam': {}, 'incomingport': 10025, 'decisions': [('F-Prot Plugin', 0), ('ClamavPlugin', 0)], 'ClamavPlugin.time': '0.0055'} : Running Plugin SAPlugin
*******
[99]: Suspect b6f3019df4da4170888e144f98eaa3ad: from=<redacted> to=<redacted> size=5187 , spam=no, virus=no tags={'virus': {'F-Prot': False, 'ClamAV': False}, 'FprotPlugin.time': '0.0088', 'spam': {}, 'incomingport': 10025, 'decisions': [('F-Prot Plugin', 0), ('ClamavPlugin', 0)], 'ClamavPlugin.time': '0.0161'} : Running Plugin SAPlugin
*******
[100]: waiting for task
*******
[101]: Suspect eefa38e4ae4a9c2938670a46d6b2b922: from=<redacted> to=<redacted> size=5203 , spam=no, virus=no tags={'virus': {'F-Prot': False, 'ClamAV': False}, 'FprotPlugin.time': '0.0086', 'spam': {}, 'incomingport': 10025, 'decisions': [('F-Prot Plugin', 0), ('ClamavPlugin', 0)], 'ClamavPlugin.time': '0.0092'} : Running Plugin SAPlugin
*******
[102]: waiting for task
*******
[103]: waiting for task
*******
[104]: waiting for task
*******
[105]: waiting for task
*******
[106]: Suspect b47e2ebf00e3f0d83dc1c7f64dad69e8: from=<redacted> to=<redacted> size=29442 , spam=no, virus=no tags={'virus': {'F-Prot': False, 'ClamAV': False}, 'FprotPlugin.time': '0.0240', 'spam': {}, 'incomingport': 10025, 'decisions': [('F-Prot Plugin', 0), ('ClamavPlugin', 0)], 'ClamavPlugin.time': '0.0240'} : Running Plugin SAPlugin
*******
[107]: waiting for task

Run fuglu_control without arguments to get a list of all supported commands.

Logs

Fuglu writes to /var/log/fuglu/fuglu.log by default. A summary of each analyzed message is stored in that file immediately after it has been analyzed..

Fuglu uses the standard python logging framework. This allows you to configure logging for each plugin individually (verbosity, own logfile, ...)

Full documentation on the python logging configuration: http://docs.python.org/library/logging.html#configuring-logging

Logging is configured in /etc/fuglu/logging.conf

In standard configuration, fuglu logs only INFO messages. Let’s say you wanted to debug the spamassassin plugin. So you’d check if there is a section dealing with spamassassin and you’d find this:

[logger_saplugin]
level=INFO
handlers=logfile
propagate=0
qualname=fuglu.plugin.SAPlugin

You would change level=INFO to level=DEBUG. Also you need to activate this special configuration for spamassassin by adding it to the loggers section at the top of the file:

[loggers]
keys=root,saplugin

After reloading fuglu, the spamassassin plugin should be very verbose:

2009-04-08 17:11:07,693 fuglu.plugin.SAPlugin: DEBUG Contacting spamd localhost (Try 1 of 5)
2009-04-08 17:11:07,701 fuglu.plugin.SAPlugin: DEBUG Sent 4065 bytes to spamd
2009-04-08 17:11:10,033 fuglu.plugin.SAPlugin: DEBUG SPAMD/1.1 0 EX_OK

2009-04-08 17:11:10,034 fuglu.plugin.SAPlugin: DEBUG Got 4778 message bytes from back from spamd
2009-04-08 17:11:10,047 fuglu.plugin.SAPlugin: DEBUG Spamscore: 2.2
2009-04-08 17:11:10,047 fuglu.plugin.SAPlugin: DEBUG Message is not spam

There should be logger sections configured for most tasks / plugins, if not, do not hesitate to open a bug on this.

Qualnames for new loggers :

Plugins usually use fuglu.plugin.<classname>

fuglu internal qualnames:

  • fuglu.SessionHandler - What plugins are being run, what are the results, ...
  • fuglu.smtp.incoming - Incoming smtp connections
  • fuglu.smtp.incoming.<portnumber> - Incoming smtp sessions on port <portnumber>
  • fuglu.smtpsession - information on the incoming smtp transaction
  • fuglu.threadpool - information when threads are created / destroyed
  • fuglu.suspectfilter - debug info on parsing/applying the special header filter config files

Additional handlers

In addition to python’s builtin handlers , fuglu supports:

  • custom_handlers.GroupReadableRotatingFileHandler - Like python’s RotatingFileHandler , but the file is readable by fuglu’s group
  • custom_handlers.GroupWritableRotatingFileHandler - Like python’s RotatingFileHandler , but the file is readable/writable by fuglu’s group
  • custom_handlers.GroupReadableTimedRotatingFileHandler - Like python’s TimedRotatingFileHandler , but the file is readable by fuglu’s group
  • custom_handlers.GroupWritableTimedRotatingFileHandler - Like python’s TimedRotatingFileHandler, but the file is readable/writable by fuglu’s group

MRTG

Fuglu can be configured to peridically write status files readable by mrtg. To do this, you only have to create a new directory (eg /usr/local/fuglu/mrtg) and set this in fuglu.conf:

#Statistics
mrtgdir=/usr/local/fuglu/mrtg

You’ll need mrtg and a webserver for this to work... create a file called /etc/fuglu/fuglu_mrtg.cfg like this:

Interval: 5
WorkDir: /usr/local/fuglu/mrtg/output
WriteExpires: yes
Options[_]: growright,nopercent
LogFormat: rrdtool

#---------------------------------------------------------------
Target[inout]: `cat /usr/local/fuglu/mrtg/inout`
PageTop[inout]: <H1>Messages in/out</H1>
Options[inout]: growright,nopercent
Title[inout]: Messages in/out
ShortLegend[inout]: msgs/s
YLegend[inout]: messages
LegendI[inout]: in
Legend1[inout]: Incoming Messages
Legend2[inout]: Messages re-injected
LegendO[inout]: re-inject
MaxBytes[inout]: 60000000

Target[hamspam]: `cat /usr/local/fuglu/mrtg/hamspam`
Options[hamspam]: growright,nopercent
Title[hamspam]: Ham / Spam
PageTop[hamspam]: <H1>Ham / Spam</H1>
YLegend[hamspam]: messages/second
ShortLegend[hamspam]: msgs/s
LegendI[hamspam]: ham
Legend1[hamspam]: Clean Messages (ham)
LegendO[hamspam]: spam
Legend2[hamspam]: Messages detected as spam
MaxBytes[hamspam]: 600000
AbsMax[hamspam]: 600000

Target[scantime]: `cat /usr/local/fuglu/mrtg/scantime`
Options[scantime]: growright,nopercent,gauge
Title[scantime]: Scan Time
PageTop[scantime]: <H1>Scan Time</H1>
WithPeak[scantime]: ymwd
YLegend[scantime]: sec
ShortLegend[scantime]: sec
Legend1[scantime]: Time needed to analyze message
LegendI[scantime]: scantime
LegendO[scantime]:
Legend2[scantime]:
Legend3[scantime]: Peak Scantime
Legend4[scantime]:
MaxBytes[scantime]: 600000
AbsMax[scantime]: 600000


Target[threads]: `cat /usr/local/fuglu/mrtg/threads`
Options[threads]: growright,nopercent,gauge
Title[threads]: Threads
PageTop[threads]: <H1>Running Threads</H1>
YLegend[threads]: threads
ShortLegend[threads]: threads
LegendO[threads]:
Legend1[threads]: Number of running threads
Legend2[threads]:
LegendI[threads]: threads
MaxBytes[threads]: 600000
AbsMax[threads]: 600000


Target[virus]: `cat /usr/local/fuglu/mrtg/virus`
Options[virus]: growright,nopercent
Title[virus]: virus
PageTop[virus]: <H1>Virus</H1>
YLegend[virus]: virus
ShortLegend[virus]: virus
#LegendO[virus]: virus
LegendI[virus]: virus
Legend1[virus]: Number of viri detected
MaxBytes[virus]: 600000
AbsMax[virus]: 600000

Make sure, the path in WorkDir exists.

Then create a script /usr/local/bin/fuglu_mrtg.sh

#!/bin/sh
/usr/bin/env LANG=C /usr/bin/mrtg /etc/fuglu/fuglu_mrtg.cfg

run fuglu_mrtg.sh to check for errors.

If all went well, add it to your crontab:

*/5 * * * * /usr/local/bin/fuglu_mrtg.sh >/dev/null 2>&1

install mrtg-rrd.cgi (probably rename it to fuglu.cgi ), and configure the path to fuglu_mrtg.cfg correctly:

# EDIT THIS to reflect all your MRTG config files
BEGIN { @config_files = qw(/etc/fuglu/fuglu_mrtg.cfg); }

after pointing your browser to http://yourfuglubox.example.com/cgi-bin/fuglu.cgi you should see nice graphs.

Troubleshooting & Debugging

Fuglu doesn’t even start

  • run fuglu --lint as root. are there any errors?
  • did you create /etc/fuglu/logging.conf (by renaming the logging.conf.dist file) ?
  • check if the log directory (/var/log/fuglu/ by default) exists an is writable by the user running fuglu
  • check for errors in /var/log/fuglu.log
  • run fuglu fuglu in foreground using fuglu --foreground or by setting daemonize=0 in fuglu.conf and check for errors

Debug Trace a single message

Fuglu runs but you want to know what exactly would happen to a single email message?

Make sure message debugging is enabled (it is by default, but if you screwed up your config...) :

  • prependers= should start with debug to make sure the debugger plugin is loaded
  • incomingport option should contain port 10888 (eg. incomingport=10025,10099,10888 ) , so fuglu listens on the debugging port
  • in the [debug] section, set debugport=10888 and debugfile=/tmp/fuglu_debug.log , so the debugger plugin knows what port to treat as debug port and where to store debug information
  • nobounce should be set to True, so debugged messages can not be bounced
  • noreinject shoud be set to True, so debugged messages are not re-injected to postfix for further processing
  • noappender should be set to True, so appenders are not run for debugged messages

Save the message on the server (rfc822 format), and run

fuglu_debug /path/to/message.eml

you may add envelope sender and recipient if these are relevant for your test, eg:

fuglu_debug /path/to/message.eml sender@example.org recipient@example.net

then check /tmp/fuglu_debug.log

Debug a suspect filter

So you have this filter in a plugin with hundreds of rules and you just don’t know which one matched a specific message?

fuglu_suspectfilter to the rescue! just pass a rule file and a message eml file as arguments and it will list all matches.

fuglu_suspectfilter /etc/fuglu/myfilterfile.regex /tmp/testmessage.eml
INFO:fuglu.suspectfilter:Reloading Rulefile /home/gryphius/Documents/skipreviewfilter.regex
INFO:root:194 valid rules found
INFO:root:1 matches found

Match #1:
Matched header/field: From
Matched value in message: "Edgar" <edgarmail@example.com>
Regex : edgarmail@example\.com
Rule argument : Messages from Edgar

Add a -D argument for a full debug trace of the suspect filter.

Debug the fuglu core

If you are a python developer and would like to inspect fuglu at run-time, add the argument --console when starting fuglu. You will then get an interactive python console where all plugins etc are available from the ‘mc’ (MainController) object.

Starting with fuglu 0.6.1 it is also possible to start a network-enabled python shell using the command fuglu_control netconsole [<port> [<bindaddress>]]

Example: on the host running fuglu:

# fuglu_control netconsole 9999 0.0.0.0
Python interactive console starting on 0.0.0.0 port 9999

on your host:

$ nc 192.168.1.40 9999
Fuglu Python Shell - MainController available as 'mc'
>>>