Watch a log file and send new lines to an HTTP endpoint – with log2http

Recently, I wanted to watch a couple of log files for new entries and have them sent to an http endpoint for collection and later analysis. I did a quick research on what tools exist, but eventually decided to create a small Python app myself which doesn’t require a complicated setup.

I thought of something along the lines of:

1. pip install <the module>
2. Define which log files to watch and where to send the contents to
3. Run it from the terminal.

And so I built it.

log2http

You can find the Python module log2http on GitHub, and here’s how you can use it:

Install

Install the module in a Python 3.6+ environment:

pip install log2http

Create a config file

Create a new YAML config file (anywhere you wish) and insert the following key:values for each file you want to watch:

-
  logfile: /Users/demo/example.log
  endpoint: http://endpoint1.example.com/tag/sample
  min_lines: 5 # only send to endpoint when at least 5 new lines were collected
-
  logfile: /var/log/system.log
  endpoint: http://endpoint2.example.com/tag/syslog
  min_lines: 150

The endpoint is the address to which new lines are being POSTed to. You could use a logging service like Loggly (endpoint would be something like this: http://logs-01.loggly.com/bulk/token/tag/example/) or build a small service yourself.

Run it

Now you can run the module from your terminal with:

log2http --config=/absolute/path/to/the/config.yml

To see how it looks like when events are being collected, generate a few sample entries for the file you are watching, e.g. in bash:

for i in {1..6}; do echo "hello world" >> /your/sample.log; done

You should see an output like this:

$ log2http --config=/your/path/to/config.yml 
collected 6 new events from /your/sample.log
Sent to http endpoint http://your-endpoint.com/

That’s it

I hope this little tool comes in handy when the need arises. Having issues or questions? Just open an issue over at GitHub.