zeek2es.py
This Python application translates Zeek's ASCII TSV logs into ElasticSearch's bulk load JSON format. For JSON logs, see Elastic's File Beats application.
This application will recognize gzip or uncompressed logs.
This application assumes you have Elasticsearch set up on your localhost at the default port.
Run this program on a system with the same timezone that was logged by Zeek originally, as zeek-cut -d -u translates the timestamps into UTC for Elasticsearch.
Command Line:
python zeek2es.py your_zeek_log.gz -i your_es_index_name
This script can be run in parallel on all connection logs, 10 at a time, with the following command:
find /some/dir -name “conn*.log.gz” | parallel -j 10 python zeek2es.py {1} :::: -
Command Line Options:
$ python zeek2es.py -h
usage: zeek2es.py [-h] [-i ESINDEX] [-u ESURL] [-l LINES] [-n NAME] [-c] [-s] [-b] filename
Process Zeek ASCII logs into Elasticsearch.
positional arguments:
filename The Zeek log in *.log or *.gz format. Include the full path.
optional arguments:
-h, --help show this help message and exit
-i ESINDEX, --esindex ESINDEX
The Elasticsearch index name.
-u ESURL, --esurl ESURL
The Elasticsearch URL. (default: http://localhost:9200/)
-l LINES, --lines LINES
Lines to buffer for RESTful operations. (default: 50,000)
-n NAME, --name NAME The name of the system to add to the index for uniqueness. (default: empty string)
-c, --checkindex Check for the ES index first, and if it exists exit this program.
-s, --stdout Print JSON to stdout instead of sending to Elasticsearch directly.
-b, --nobulk Remove the ES bulk JSON header. Requires --stdout.
Requirements:
- A Unix-like environment (MacOs works!)
- Python
- zeek-cut in your path
6 Dec 7, 2021
45 Nov 25, 2022
1 Dec 7, 2021
4 Jul 20, 2022
14 Dec 22, 2022
62 Sep 20, 2022
8 Dec 18, 2022
79 Nov 11, 2022
9 Dec 20, 2022
51 Dec 01, 2021
4 Jan 01, 2022
3 Aug 08, 2022
72 Dec 14, 2022
56 Dec 13, 2022
5 Jul 12, 2019
68 Dec 07, 2022
44 Dec 29, 2022
150 Nov 23, 2022
11 Aug 28, 2022
4 Mar 07, 2022
168 Dec 25, 2022
196 Dec 15, 2022
12.6k Dec 30, 2022
95 Nov 28, 2022
16.3k Dec 29, 2022
3.4k Dec 29, 2022