Monitor Logs in Real-time with Swatch

Though limited in its abilities, Swatch is a very powerful tool to implement alongside other security products to proactively monitor system logs. Swatch gives systems administrators great log-monitoring options. It’s a perfect tool for monitoring SSH or denial-of-service attacks on Linux servers, and alerts admins about trouble before it’s too late.

Swatch stands for ‘simple watcher’ or ‘Syslog watcher’, depending on whom you ask. Either way, it is a helpful program that does your log-watching, and alerts you only when things that you are specifically looking for get logged. Note that Swatch is a Perl program that regularly sweeps the main log files and looks for certain keywords that you can define. It can be run in two ways — in the background as a daemon or as a cron job. You can configure Swatch to notify you of any events in the messages or syslog files that might indicate a security problem. However, Swatch can also be used to flag just about any kind of activity: a certain program being used, a certain user logging in, or anything that might appear in a log file. Swatch can even be configured to watch application-specific log files instead of the general log files that it does by default. It is a Linux tool and it helps in monitoring the log files as they are being written. It then takes the necessary action if it finds something that it is configured to look for. This tool can be used to proactively scan log files in real-time for various suspicious activities, error messages or specific keywords.
Swatch basically started out as a simple watchdog for actively monitoring the log files produced by UNIX’s syslog facility. Since then, it has evolved into a utility that can monitor just about any type of log. One can consider Swatch as a command line utility and it can be started by issuing a swatch command with various settings after it.
Please note that certain events that are logged have a great significance from a security standpoint. The default items that Swatch looks for are a good start.

• Bad logins criteria: The words Invalid, Repeated, or Incomplete appear in the message file.
• System crashes criteria: The words panic or halt appear in the log files.
• System reboots criteria: The banner of your OS should appear in the log files only when you reboot.

How to install Swatch
Swatch requires Perl 5 or higher. If you have a fairly new installation of Linux or BSD, then you should have a sufficiently current version.
1. Swatch requires multiple Perl modules to be installed in order to function correctly. You must first install CPAN and download each module via the CPAN console. In order to install these modules, you may be prompted to install additional modules as well. The configuration process will tell you if the required modules are missing.
2. We will need to use the following command to ensure that the required Perl modules are installed:

cpan –i module-name

Here, module-name needs to be replaced with Date::Calc, Date::HiRes, and Date::Format.
Download the tar file from the Sourceforge website and unzip it.
Use the following command to extract the files:

   $tar –zxvf swatch-3.2.3.tar.gz

3. Since Swatch is a Perl program, the installation process is slightly different. Here is the sequence of commands that one needs to follow:

perl Makefile.PL
make
make test
make install
make realclean

Once these processes are done, Swatch is installed and we are ready to go.
After installing Swatch, one might be interested in creating a configuration file. If you look at the contents of a Swatch configuration file, you will notice that the syntax is very simple – all it requires is a definition of what to search for, followed by an action or remedy if a specific match is located. It is important to note here that Swatch uses Perl regular expressions as a way to define the search parameters and can perform a variety of actions.
First, you need to create an empty file to be used as a configuration file. The normal practice is to create this file under /etc. One can modify the configuration file by using any basic text editor. The Swatch configuration file is where all the important settings are located. In this file, called swatchrc by default, one can instruct the program what to look for in the log files and what to do if what has been specified does show up. Since the whole point of Swatch is to simplify our lives, configuring Swatch is pretty simple. As stated earlier, Swatch is controlled by a single file, $HOME/.swatchrc (by default). This file contains text patterns, in the form of regular expressions that you want Swatch to watch for. Each regular expression is followed by the action(s) you wish Swatch to take whenever it encounters that text.

The configuration file syntax begins with a ‘watchfor’ keyword. Basically, this line instructs Swatch to search for specific patterns in the form of regular expressions. Each ‘watchfor’ line is then preceded by an action.
There are two options available in swatchrc to specify the patterns to watch out for. These are:
1. Watch for regex – Here, an appropriate action will be taken when the regular expression specified in regex is found within the file or command being monitored.
2. Ignore regex – Take an action when there is any activity within the file or command being monitored, except for events that match the expression specified in regex.
In the following example, we will simply output the log entry to the console if an ssh connection has been established by issuing the echo action.

watchfor /ssh/
            echo bold

Let us take a quick look at some of the options that are available with Swatch.

For example, the following command…

./swatch --config-file /home/swayam/my-swatch-config -- daemon

…will run Swatch using the configuration file found at /home/swayam/my-swatch-config instead of the default config file. It will also run it as a background process or daemon. Please note that the above options may be issued alone or together.
Swatch expects the .swatchrc file to reside in the home directory of the user who invokes Swatch. Swatch also keeps its temporary files there by default. Each time it’s invoked, it creates and runs a script called a ‘watcher process’, the name of which ends with a dot, followed by the PID of the Swatch process that created it. Swatch generally doesn’t clean up after itself very well; it tends to leave watcher-process scripts behind. Thus, users are expected to keep an eye out and periodically delete these in their home directory.
The command –tail-file=[path to log file] directs Swatch to watch a specific log file for potential matches.
An interesting feature of Swatch is that one can run multiple instances of it, each configured to use a customised configuration file and watch a different log file. Some of the common log files that Swatch uses are listed below.

• /var/log/maillog – logs all e-mail messages
• /var/log/cron – logs messages regarding cron job schedules
• /var/log/messages – logs system messages

Let us look at all the actions Swatch can take when a specific search is completed. Swatch provides a variety of actions that you can perform in response to a matched event. One can output alerts to the console, pipe output to another log file, send email alerts, or even execute a remedial script. As more than one action can be applied to a single event, you can combine these to suit your requirements. The following table illustrates various Swatch action statements.

As can be seen from the above table, Swatch can notify you of flagged log events in several different ways. The easiest is to just have it beep or echo on the screen. If you are not around the server all the time, then you can have it email you. If your pager or cell phone supports text messaging via email, you could have it send the message directly to you. You can also write a script to have the server dial a pager number using the UNIX tip command.
Echoing the log output is considered as one of the most basic functions of Swatch. This can be utilised as a way to gain the attention of users (if they are currently using the console) by outputting the log contents to the console.
To utilise the echo action, simply issue the ‘echo’ keyword underneath the ‘watchfor’ line as follows:

watchfor /su|sudo/
                         echo [formatting keywords]

Although it is very basic, echo offers a variety of formatting methods in which to display alerts. One can change the text colour, underline it, make it bold, strike-through, flash text, as well as combine multiple formatting keywords.
Swatch can alert a user when there is a positive match by issuing a bell sound. The following example illustrates this:

 watchfor /su|sudo/
                     bell [total # of rings]

In what is considered as a more efficient way of alerting, one can configure Swatch to send an email whenever there is a match of a specific event. This is a convenient way to alert systems administrators of real-time events without requiring them to be at the console. Let us take a look at an example of this. We will be sending an email to the administrator of a new application that has been installed on the system. Note that the address contains an escape character for the email address \@ — this is required for Perl to format the address correctly. Also, if you would like to send spaces within your subject line, you must place an escape character prior to each space.

watchfor /[iI]nstalled/
           mail addresses=admin_person\@mycompany.com,
           subject=Unauthorized\ Application\ Installation

It is important to note here that the throttle action helps to prevent denial-of-service attacks via Swatch (e.g., deliberately triggering huge numbers of Swatch events in a short period). In other words, throttle gives Swatch the intelligence to ignore repeated occurrences of a given event, potentially preventing minor events from becoming major annoyances.
As part of the remedy, note that Swatch has the ability to execute a secondary script if a specific event has been detected. This functionality could trigger some further action(s) in response to an event. The syntax required uses the exec keyword and accepts some bash, Perl, and other useful commands.
The following example illustrates this, where we direct Swatch to execute a remediation Perl script if it detects a port scan against the system.

watchfor /[cC]onnection closed by/
    exec “perl /usr/bin/custom_remediation.pl”

Another important scenario where Swatch can be used is to avoid a storm of alert messages. We can configure Swatch to take a specific action only if it detects a certain number of similar events within a certain time frame. Let us consider a scenario where there are multiple SSH failed login attempts within a specific time period. We can configure Swatch to search for this case and take action — sending a message to the systems administrator as well as outputting a log message to the console. The following commands will handle such a case:

watchfor /ssh.*failed/
                                    echo bold
mail addresses=admin_person\@mycompany.com,
subject=Possible\ SSH\ Brute\ Force\ Attempts
threshold track_by=$1, type=threshold, count=5, seconds=10

In the above example, we have set the threshold as ‘5 failed login attempts within 10 seconds’.
Another powerful feature that Swatch provides is its ability to specify a given time of day that an action will be performed. This is very beneficial to perform certain sets of actions over the weekend or after business hours. To apply a time constriction to an action, we just need to append the keyword ‘when=’, followed by the time duration after an action. The syntax used to represent the timeframe is indicated in numerical format, whereas each weekday is represented by a number 1-7 (7 = Saturday, 1 = Sunday, etc), and hours are represented as a 24-hour timeframe (1 – 24). The following example illustrates this concept:

watchfor /system full/
                    mail addresses=admin_person\@company.com,
                    subject=File\ System\ Full,
                    when=7-1:1-24

In the above example, a user will be alerted by Swatch only over the weekend if a storage drive becomes full.
It is always advisable to follow some best practices when you are using Swatch. If Swatch’s actions don’t fire very often, it could be that your system isn’t getting probed or is misused very much, but it’s at least as likely that Swatch isn’t casting its net wide enough. In such cases, you may need to periodically scan through your logs manually to see if you’re missing anything, and continue to tweak the Swatch configuration file, .swatchrc. Also, as another best practice, you should never forget to periodically reconsider the auditing/logging configurations of the daemons that generate log messages in the first place. It is critical to note here that Swatch won’t catch events that aren’t logged at all.