Introduction
As you may already know, that I love Virtual Computing. My program of choice is VMWare, but regardless of what you use, I just love the concept. Virtual computing allows a small company like the Assay Depot to create many virtual computers, each handling a different aspect of the business. However, nothing is free in this world, and Virtual Computing comes with its own set of challenges. In this post, I hope to show how you can use Splunk, sysnlog-ng and named pipes to keep you informed as to what is happening on each of those virtual computers (it works just as well for physical computers). I use Ubuntu, so this post assumes you do too. You may have to do some interpreting if this is not the case.
During this project I found this post informative: http://mysfitt.net/tutorials/splunk_fifo.php
Design
I have many Ubuntu machines, database servers, development servers, mail relays, application servers, etc. I want to monitor them using Splunk. Splunk is a program that reads log files, makes them searchable and generally more manageable. Splunk can read log files from many different sources, for this post, I chose to to tail files from the local file system.
Since I want to manage all my machines with one instance of Splunk, my next challenge was to continuously feed log files from those machines onto one central logging server. This is where syslog-ng comes in. On one end, syslog-ng can be configured to send all system log statements to a network port. On the other end, it can be configured to receive those statements and write them to a file. In this way, our central logging server can handle all the system log files of all the Ubuntu machines on our network and Splunk can index them all.
Finally, I want to monitor services that don’t write directly to the system log. I would like to have syslog-ng simply tail those log files that I’m interested, however, either syslog-ng can’t do that, or I couldn’t figure it out (probably the latter). Instead it wrote a Ruby daemon that will tail a file and pipe the results to a named pipe. Syslog-ng then reads from that named pipe and sends the information over a network port to the central logging server that writes the statement to a file for indexing.
Simple enough, lets get to it.
Install Splunk
For this post, I’ve chosen the Splunk 3.0 beta. If you choose to use version 2.2.6, you can download the deb file. For the beta, I downloaded the tgz version.
The following code should be executed on the machine you plan to use as your central logging server.
That’s it, we downloaded the package, expanded it to /opt, copied the init script to /etc/init.d and made it executable, told the server to start it when the machine starts, and finally started the server. Since we used the defaults (installed /opt/splunk) we didn’t have to change anything. If you want to install to a different location, their README is very informative.
Install syslog-ng (Server)
Next we need to syslog-ng on the central logging server and configure it to accept logging statements from the clients.
First, install syslog-ng using apt-get:
by default, syslog-ng doesn’t use dns names for performance reasons, but we want to turn it on so our logs are stored in directories named by server name, rather than by IP address. To turn it on, edit /etc/syslog-ng/syslog-ng.conf and change the line that says: to
Next, you have to set up some sources and destinations. Sources are where syslog-ng will read from and are typically named with a leading “s_”. Destinations are where syslog-ng will write to and are typically named with a leading “d_”. Log statements connect sources and destinations. I added the following to my /etc/syslog-ng/syslog-ng.conf file.
The source reads logging statements from port 5140. The destination writes those logging statements to files in /u01/log/hosts based on the host name, date and which service published the logging statement. Lastly, the log statement connects the two.
Once you restart the syslog-ng service, your central logging server is ready to go:
Install syslog-ng (Client)
Now that you have completed your central logging server, its time to setup some clients. First (just like with the server) you need to install syslog-ng:
Now setup some sources and destinations in /etc/syslog-ng/syslog-ng.conf, except this time, you will be reading log statements from the system and writing them to a network port:
The destination statement sets up syslog-ng to write to port 5140 on the server splunk01 (that’s the name of my central logging server). The log statement connects s_all, which is the default source for syslog-ng, to your logging server.
Your logging server which should now be collecting all kernel log messages from all the servers you’ve configured.
Log Custom Services
Ok, so far we are logging all kernel log statements to your logging server. That’s great, but we really want to log our custom services which write to standard log files. Services such as postgres or Rails. Now, I’m certainly not a syslog-ng expert, but I couldn’t figure out how to get syslog-ng to tail a file, I could get it to read a file once, but that didn’t help much. I did however get syslog-ng to read from a named pipe. So I took it upon myself to write a program that would tail a file to a named pipe. (BTW, a named pipe, aka a FIFO, is tool for interprocess communication, one process can write to it like a file, while another reads from it like a file).
First, we create our named pipe:
I called mine, /var/syslog-ng/syslog_fifo. Now lets make syslog-ng read from that fifo and write to our network port. First we add a new source to /etc/syslog-ng/syslog-ng.conf
Next we update the log statement from the previous section, it should now look like:
Now, it is reading from the s_all source as well as our new s_pipe source and writing everything to d_splunk, our logging server.
You must restart syslog-ng for the changes to take effect:
Finally, we need to tail the files we are interested in and pipe them to our new fifo. To accomplish this I wrote the following daemon using Ruby:
The daemon reads from /etc/rlogd.conf. The config file consists of the filenames of any file that you want to be tailed separated by newlines. For instance, if you wanted to monitor your postgres server, your rlogd.conf file would look something like:
Now start the rlogd server:
At this point, your client is setup. It should be sending all kernel log messages, as well as messages from log files you’ve configured to your logging server where they are stored and indexed.
Since I will be performing this on all of my servers, I went ahead and packaged it up as a deb file, so setting up my clients is as easy as:
Configure Splunk
Your Splunk server should already be running and your clients should be logging to it. The last step is to configure Splunk to read the log files. First, visit http://splunk01:8000/admin, where splunk01 is the name of your logging server. Next, click on “Data Inputs”, then on “Files and Directories”. From there you can “Add Input”; Set the path to /u01/log (or whatever directory you are using) and change the host to “Regex on path”. I use the following regex:
