osqueryIR is an artifact collection tool for Linux systems.

Overview

osqueryIR

osqueryIR is an artifact collection tool for Linux systems. It provides the following capabilities:

  • Execute osquery SQL queries
  • Collect files and folder
  • Execute system commands
  • Parse log files (ex. nginx, auth, syslog, etc) using regex

Try it

  1. Clone this repo

    git clone https://github.com/abdulrhmanalfaifi/osqueryIR
  2. Download python dependencies

    python3 -m pip install -r requirments.txt
  3. Try it using this command

    python3 osqueryIR.py -h

Usage

The following is the help message for osqueryIR:

usage: osqueryIR.py [-h] [--osquery-binary OSQUERY_BINARY] [-c CONFIG]
                    [-o OUTPUT] [-q] [--log-file-name LOG_FILE_NAME]
                    [--log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG}]
                    [--output-format {jsonl,kjson}] [--disable-collect]

A Linux artifact collection tool

optional arguments:
  -h, --help            show this help message and exit
  --osquery-binary OSQUERY_BINARY
                        osqueryd binary path (Default=./osqueryd)
  -c CONFIG, --config CONFIG
                        Path to the configuration file (Default=./config.yaml)
  -o OUTPUT, --output OUTPUT
                        Change the output folder name (Defaults to the machine
                        hostname)
  -q, --quiet           Do not print log messages
  --log-file-name LOG_FILE_NAME
                        Name of the log file (Default=osqueryIR_log)
  --log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG}
                        Set logging level (Default=INFO)
  --output-format {jsonl,kjson}
                        Change the output format (Default=jsonl)
  --disable-collect     Disable collection artifacts
  • --osquery-binary: osqueryd binary path. By default it will uses the binary in this repo.

  • -c or --config: path to osqueryIR configuration. By default it will be config.yaml in this repo.

  • -o or --output: the output file (zip file) name, By default it will be the machine hostname.

  • -q or --quit: Do not print logging to the stdout. osqueryIR will always write the log to the output file.

  • --log-file-name: change the default name for the log file (osqueryIR_log).

  • --log-level: the log level, default is INFO.

  • --output-format: osqueryIR support writing the results in two different formats:

    • jsonl: a newline separated JSON object. Each object represent a record.
    • kjson: the format understood by Kuiper. If you are planing to use Kuiper for analysis then you should use this format.
  • --disable-collect: disable artifact collecting. Only parsing and osquery artifacts will be acquired.

Configuration

osqueryIR accepts a configuration file that contains artifact specification. The following is an example configuration along with comments:

artifacts:  
    # Name of artifact. the results will be saved to a file with this name
  - logged_in_users:
      # artifact type. queries run osquery SQL queries and return the results as json
      queries:
        - 'select * from logged_in_users'
      # Optional: map the field called `name` to `@timestamp` and run the modfier `epoch_to_iso` on the value. `modifier` field is not required
      maps:
        - name: time
          map_to: '@timestamp'
          modifier: epoch_to_iso
      # Optional: description of the artifact
      description: 'Collect and parse the currently loggedin users'
  - logs:
      # artifact type. collect the specified files and directories without parsing
      collect:
        - '/var/log/**'
        - '/home/*/.vnc/*.log'
      description: 'Collect logs wellknow paths'
  - auth_log:
      # artifact type. parse the specified files using regex and return the results as json.
      parse:
        # files to parse
        path: '/var/log/auth.log*'
        # regex used for parsing
        regex: '([A-Z][a-z]{2}[ ]{1,}[0-9]{1,2}[ ]{1,2}[0-9]{1,2}:[0-9]{2}:[0-9]{2}) ([a-zA-Z0-9_\-]+) ([a-zA-Z0-9_\-\]\(\)=\./]+)\[?([0-9]+)?\]?: (.*)'
        # the name of the extracted fields
        fields:
          - 'time'
          - 'hostname'
          - 'service'
          - 'pid'
          - 'msg'
      maps:
        - name: time
          map_to: '@timestamp'
          modifier: time_without_year_to_iso
      description: 'Parse auth logs from the path /var/log/, and return the results as jsonl/kjson'
  - bad_logins:
  	  # artifact type. Execute system command and return stdout & stderr
      command:
        - 'lastb'

Example

To collect the artifacts from the provided configurations, execute the following command:

python3 osqueryIR.py

A file will be created named {HOSTNAME}.zip that contains all artifacts.

Useing osqueryIR with Kuiper

osqueryIR can generate the result in kjson format which could be ingested by Kuiper. To collect artifacts in kjson format execute the following command:

python3 osqueryIR.py --output-format kjson --disable-collect

upload the file to Kuiper and execute the kjson parser

osqueryIR_Kuiper

Owner
AbdulRhman Alfaifi
AbdulRhman Alfaifi
Python program for analyzing the output files of phonopy.

PhononTools Description Python program to analyze the results generated by phonopy. Using the .yaml and .dat files that phonopy generates one can plot

Harry LaBollita 8 Nov 27, 2022
🚧Useful shortcuts for simple task on windows

Windows Manager A tool containg useful utilities for performing simple shortcut tasks on Windows 10 OS. Features Lit Up - Turns up screen brightness t

Olawale Oyeyipo 0 Mar 24, 2022
Simple collection of GTPS Flood in Python.

GTPS Flood Simple collection of GTPS Flood in Python. NOTE Give me credit if you use this source, don't trade/sell this tool, And USE AT YOUR OWN RISK

PhynX 6 Dec 07, 2021
Patch the pclntable from Go binaries

Pretrain and Fine-tune a T5 model with Flax on GCP This tutorial details how pretrain and fine-tune a FlaxT5 model from HuggingFace using a TPU VM ava

6 Oct 05, 2022
cssOrganizer - organize a css file by grouping them into categories

This python project was created to scan through a CSS file and produce a more organized CSS file by grouping related CSS Properties within selectors. Created in my spare time for fun and my own utili

Andrew Espindola 0 Aug 31, 2022
Just some scripts to export vector tiles to geojson.

Vector tiles to GeoJSON Nowadays modern web maps are usually based on vector tiles. The great thing about vector tiles is, that they are not just imag

Lilith Wittmann 77 Jul 26, 2022
Aggregating gridded data (xarray) to polygons

A package to aggregate gridded data in xarray to polygons in geopandas using area-weighting from the relative area overlaps between pixels and polygons.

Kevin Schwarzwald 42 Nov 09, 2022
Backup a folder to an another folder by using mirror update method.

Mirror Update Backup Backup a folder to an another folder by using mirror update method. How to use Install requirement pip install -r requirements.tx

1 Nov 21, 2022
A python package containing all the basic functions and classes for python. From simple addition to advanced file encryption.

A python package containing all the basic functions and classes for python. From simple addition to advanced file encryption.

PyBash 11 May 22, 2022
Daiho Tool is a Script Gathering for Windows/Linux systems written in Python.

Daiho is a Script Developed with Python3. It gathers a total of 22 Discord tools (including a RAT, a Raid Tool, a Nuker Tool, a Token Grabberr, etc). It has a pleasant and intuitive interface to faci

AstraaDev 32 Jan 05, 2023
A Randomizer Oracle

Tezos Randomizer Tezod Randomizer "Oracle". It's a smart contract that you can call to get a random number between X and Y (for now). It uses entropy

Asbjorn Enge 19 Sep 13, 2022
Make your functions return something meaningful, typed, and safe!

Make your functions return something meaningful, typed, and safe! Features Brings functional programming to Python land Provides a bunch of primitives

dry-python 2.6k Jan 09, 2023
A pythonic dependency injection library.

Pinject Pinject is a dependency injection library for python. The primary goal of Pinject is to help you assemble objects into graphs in an easy, main

Google 1.3k Dec 30, 2022
Early version for manipulate Geo localization data trough API REST.

Backend para obtener los datos (beta) Descripción El servidor está diseñado para recibir y almacenar datos enviados en forma de JSON por una aplicació

Víctor Omar Vento Hernández 1 Nov 14, 2021
Manage your exceptions in Python like a PRO

A linter to manage all your python exceptions and try/except blocks (limited only for those who like dinosaurs).

Guilherme Latrova 353 Dec 31, 2022
UUID version 7, which are time-sortable (following the Peabody RFC4122 draft)

uuid7 - time-sortable UUIDs This module implements the version 7 UUIDs, proposed by Peabody and Davis in https://www.ietf.org/id/draft-peabody-dispatc

Steve Simmons 22 Dec 20, 2022
A collection of utility functions to prototype geometry processing research in python

gpytoolbox This repo is a work in progress and contains general utility functions I have needed to code while trying to work on geometry process resea

Silvia Sellán 73 Jan 06, 2023
This is Cool Utility tools that you can use in python.

This is Cool Utility tools that you can use in python. There are a few tools that you might find very useful, you can use this on pretty much any project and some utils might help you a lot and save

Senarc Studios 6 Apr 18, 2022
✨ Un générateur d'adresse IP aléatoire totalement fait en Python par moi, et en français.

IP Generateur ❗ Un générateur d'adresse IP aléatoire totalement fait en Python par moi, et en français. 🔮 Avec l'utilisation du module "random", j'ai

MrGabin 3 Jun 06, 2021
Extract the download URL from OneDrive or SharePoint share link and push it to aria2

OneDriveShareLinkPushAria2 Extract the download URL from OneDrive or SharePoint share link and push it to aria2 从OneDrive或SharePoint共享链接提取下载URL并将其推送到a

高玩梁 262 Jan 08, 2023