osqueryIR is an artifact collection tool for Linux systems.

Overview

osqueryIR

osqueryIR is an artifact collection tool for Linux systems. It provides the following capabilities:

  • Execute osquery SQL queries
  • Collect files and folder
  • Execute system commands
  • Parse log files (ex. nginx, auth, syslog, etc) using regex

Try it

  1. Clone this repo

    git clone https://github.com/abdulrhmanalfaifi/osqueryIR
  2. Download python dependencies

    python3 -m pip install -r requirments.txt
  3. Try it using this command

    python3 osqueryIR.py -h

Usage

The following is the help message for osqueryIR:

usage: osqueryIR.py [-h] [--osquery-binary OSQUERY_BINARY] [-c CONFIG]
                    [-o OUTPUT] [-q] [--log-file-name LOG_FILE_NAME]
                    [--log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG}]
                    [--output-format {jsonl,kjson}] [--disable-collect]

A Linux artifact collection tool

optional arguments:
  -h, --help            show this help message and exit
  --osquery-binary OSQUERY_BINARY
                        osqueryd binary path (Default=./osqueryd)
  -c CONFIG, --config CONFIG
                        Path to the configuration file (Default=./config.yaml)
  -o OUTPUT, --output OUTPUT
                        Change the output folder name (Defaults to the machine
                        hostname)
  -q, --quiet           Do not print log messages
  --log-file-name LOG_FILE_NAME
                        Name of the log file (Default=osqueryIR_log)
  --log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG}
                        Set logging level (Default=INFO)
  --output-format {jsonl,kjson}
                        Change the output format (Default=jsonl)
  --disable-collect     Disable collection artifacts
  • --osquery-binary: osqueryd binary path. By default it will uses the binary in this repo.

  • -c or --config: path to osqueryIR configuration. By default it will be config.yaml in this repo.

  • -o or --output: the output file (zip file) name, By default it will be the machine hostname.

  • -q or --quit: Do not print logging to the stdout. osqueryIR will always write the log to the output file.

  • --log-file-name: change the default name for the log file (osqueryIR_log).

  • --log-level: the log level, default is INFO.

  • --output-format: osqueryIR support writing the results in two different formats:

    • jsonl: a newline separated JSON object. Each object represent a record.
    • kjson: the format understood by Kuiper. If you are planing to use Kuiper for analysis then you should use this format.
  • --disable-collect: disable artifact collecting. Only parsing and osquery artifacts will be acquired.

Configuration

osqueryIR accepts a configuration file that contains artifact specification. The following is an example configuration along with comments:

artifacts:  
    # Name of artifact. the results will be saved to a file with this name
  - logged_in_users:
      # artifact type. queries run osquery SQL queries and return the results as json
      queries:
        - 'select * from logged_in_users'
      # Optional: map the field called `name` to `@timestamp` and run the modfier `epoch_to_iso` on the value. `modifier` field is not required
      maps:
        - name: time
          map_to: '@timestamp'
          modifier: epoch_to_iso
      # Optional: description of the artifact
      description: 'Collect and parse the currently loggedin users'
  - logs:
      # artifact type. collect the specified files and directories without parsing
      collect:
        - '/var/log/**'
        - '/home/*/.vnc/*.log'
      description: 'Collect logs wellknow paths'
  - auth_log:
      # artifact type. parse the specified files using regex and return the results as json.
      parse:
        # files to parse
        path: '/var/log/auth.log*'
        # regex used for parsing
        regex: '([A-Z][a-z]{2}[ ]{1,}[0-9]{1,2}[ ]{1,2}[0-9]{1,2}:[0-9]{2}:[0-9]{2}) ([a-zA-Z0-9_\-]+) ([a-zA-Z0-9_\-\]\(\)=\./]+)\[?([0-9]+)?\]?: (.*)'
        # the name of the extracted fields
        fields:
          - 'time'
          - 'hostname'
          - 'service'
          - 'pid'
          - 'msg'
      maps:
        - name: time
          map_to: '@timestamp'
          modifier: time_without_year_to_iso
      description: 'Parse auth logs from the path /var/log/, and return the results as jsonl/kjson'
  - bad_logins:
  	  # artifact type. Execute system command and return stdout & stderr
      command:
        - 'lastb'

Example

To collect the artifacts from the provided configurations, execute the following command:

python3 osqueryIR.py

A file will be created named {HOSTNAME}.zip that contains all artifacts.

Useing osqueryIR with Kuiper

osqueryIR can generate the result in kjson format which could be ingested by Kuiper. To collect artifacts in kjson format execute the following command:

python3 osqueryIR.py --output-format kjson --disable-collect

upload the file to Kuiper and execute the kjson parser

osqueryIR_Kuiper

Owner
AbdulRhman Alfaifi
AbdulRhman Alfaifi
Dice Rolling Simulator using Python-random

Dice Rolling Simulator As the name of the program suggests, we will be imitating a rolling dice. This is one of the interesting python projects and wi

PyLaboratory 1 Feb 02, 2022
✨ Un générateur d'adresse IP aléatoire totalement fait en Python par moi, et en français.

IP Generateur ❗ Un générateur d'adresse IP aléatoire totalement fait en Python par moi, et en français. 🔮 Avec l'utilisation du module "random", j'ai

MrGabin 3 Jun 06, 2021
This is a python table of data implementation with styles, colors

Table This is a python table of data implementation with styles, colors Example Table adapts to the lack of data Lambda color features Full power of l

Урядов Алексей 5 Nov 09, 2021
A multipurpose python module

pysherlock pysherlock is a Python library for dealing with web scraping using images, it's a Python application of the rendertron headless browser API

Sachit 2 Nov 11, 2021
Simple collection of GTPS Flood in Python.

GTPS Flood Simple collection of GTPS Flood in Python. NOTE Give me credit if you use this source, don't trade/sell this tool, And USE AT YOUR OWN RISK

PhynX 6 Dec 07, 2021
PyGMT - A Python interface for the Generic Mapping Tools

PyGMT A Python interface for the Generic Mapping Tools Documentation (development version) | Contact | Try Online Why PyGMT? A beautiful map is worth

The Generic Mapping Tools (GMT) 564 Dec 28, 2022
Prime Path Generator is a prime path generator used to generate prime paths.

Prime Path Generator is a prime path generator used to generate prime paths.

1 Nov 06, 2021
JeNot - A tool to notify you when Jenkins builds are done.

JeNot - Jenkins Notifications NOTE: under construction, buggy, and not production-ready What A tool to notify you when Jenkins builds are done. Why Je

1 Jun 24, 2022
glip is a module for retrieve ip address like local-ip, global-ip, external-ip as string.

gle_ip_info glip is a module for retrieve ip address like local-ip, global-ip, external-ip as string.

Fatin Shadab 3 Nov 21, 2021
Playing with python imports and inducing those pesky errors.

super-duper-python-imports In this repository we are playing with python imports and inducing those pesky ImportErrors. File Organization project │

James Kelsey 2 Oct 14, 2021
Group imports from Windows binaries

importsort This is a tool that I use to group imports from Windows binaries. Sometimes, you have a gigantic folder full of executables, and you want t

【☆ ゆう ☆ 】 15 Aug 27, 2022
Auto-generate /etc/hosts for HackTheBox machines

Auto-generate /etc/hosts for HackTheBox machines Save yourself some tedium on getting started on a new machine by having your /etc/hosts ready to go.

3 Feb 16, 2022
extract gene TSS/TES site form gencode/ensembl/gencode database GTF file and export bed format file.

GetTsite python Package extract gene TSS/TES site form gencode/ensembl/gencode database GTF file and export bed format file. Install $ pip install Get

laojunjun 7 Nov 21, 2022
✨ Un code pour voir les disponibilités des vaccins contre le covid totalement fait en Python par moi, et en français.

Vaccine Notifier ❗ Un chois aléatoire d'un article sur Wikipedia totalement fait en Python par moi, et en français. 🔮 Grâce a une requète API, on peu

MrGabin 3 Jun 06, 2021
Simple python module to get the information regarding battery in python.

Battery Stats A python3 module created for easily reading the current parameters of Battery in realtime. It reads battery stats from /sys/class/power_

Shreyas Ashtamkar 5 Oct 21, 2022
A python mathematics module

A python mathematics module

Fayas Noushad 4 Nov 28, 2021
Build capture utility for Linux

CX-BUILD Compilation Database alternative Build Prerequisite the CXBUILD uses linux system call trace utility called strace which was customized. So I

GLaDOS (G? L? Automatic Debug Operation System) 3 Nov 03, 2022
It is a tool that looks for a specific username in social networks

It is a tool that looks for a specific username in social networks

MasterBurnt 6 Oct 07, 2022
Display your calendar on the wallpaper.

wallCal Have your calendar appear as the wallpaper. disclaimer Use at your own risk. Don't blame me if you miss a meeting :-) Some parts of the script

7 Jun 14, 2022
The producer-consumer problem implemented with threads in Python

This was developed using a Python virtual environment, I would strongly recommend to do the same if you want to clone this repository. How to run this

Omar Beltran 1 Oct 30, 2021