Export Data From Splunk To CSV

Posted on Posted in Data & Business Intelligence

I needed to export some data out of Splunk, based on a searched query, I needed to export it on a daily/hourly basis.
I used Splunk-SDK and Python.

How to install Splunk-SDK (Python 2.7, Python3):

The attached Python script is connecting to Splunk and executing a query (which is part of config).
Since Splunk limits the number of exported rows to 50K (unless you changed it on searchresults.conf – which isn’t recommended), I used pagination in order to export all rows (into one csv file).

With this script I’ve exported ~10M rows into one csv, you can export more if needed.

config_file:
The config contains details about how to connect to Splunk (Splunk host, Splunk port – default is 8089, username and password) and about the search:
ENG_METRIC is the name of the search (it can be any name) and the value is the actual search in Splunk. Note that the query is using a parameter ({0}), which is last execution time (calculated and stored using the Python script).

LAST_RUN is the location of a file which will hold last_execution time. This will allow us to export data and then export more data from the last execution time until now. The interval between execution doesn’t have to be same, it can be 2 hours, 4 hours, 8 hours or days.

config_file:

This is the full script. There’s an input parameter, which is the name of the search (as defined in config_file). The script saves last_execution time in a file (if the file doesn’t exist – it will create it and will export last hour)

Example: # python export_splunk.py ENG_METRICS

Leave a Reply

Your email address will not be published. Required fields are marked *