Mihai Anca

RSS

Processing logs in real-time

I thought about how to process logs faster than using a cron job to poll for updates and I found mkfifo which you can use to get what’s written in real time. It makes a FIFO pipe which acts like a regular unix file except it has to be opened for reading (using fopen) before writing to it. I tested it on Snow Leopard, it is the same in linux as well.

After getting the updates in real time you can pushing them to a queue server like RabbitMQ for processing. Here’s how I done it using Ruby, Eventmachine and amqp gem:

mihai$ mkfifo log
mihai$ ruby1.9 publisher.rb log &
mihai$ echo "Test" > log