osqueryIR is an artifact collection tool for Linux systems.

Overview

osqueryIR

osqueryIR is an artifact collection tool for Linux systems. It provides the following capabilities:

  • Execute osquery SQL queries
  • Collect files and folder
  • Execute system commands
  • Parse log files (ex. nginx, auth, syslog, etc) using regex

Try it

  1. Clone this repo

    git clone https://github.com/abdulrhmanalfaifi/osqueryIR
  2. Download python dependencies

    python3 -m pip install -r requirments.txt
  3. Try it using this command

    python3 osqueryIR.py -h

Usage

The following is the help message for osqueryIR:

usage: osqueryIR.py [-h] [--osquery-binary OSQUERY_BINARY] [-c CONFIG]
                    [-o OUTPUT] [-q] [--log-file-name LOG_FILE_NAME]
                    [--log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG}]
                    [--output-format {jsonl,kjson}] [--disable-collect]

A Linux artifact collection tool

optional arguments:
  -h, --help            show this help message and exit
  --osquery-binary OSQUERY_BINARY
                        osqueryd binary path (Default=./osqueryd)
  -c CONFIG, --config CONFIG
                        Path to the configuration file (Default=./config.yaml)
  -o OUTPUT, --output OUTPUT
                        Change the output folder name (Defaults to the machine
                        hostname)
  -q, --quiet           Do not print log messages
  --log-file-name LOG_FILE_NAME
                        Name of the log file (Default=osqueryIR_log)
  --log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG}
                        Set logging level (Default=INFO)
  --output-format {jsonl,kjson}
                        Change the output format (Default=jsonl)
  --disable-collect     Disable collection artifacts
  • --osquery-binary: osqueryd binary path. By default it will uses the binary in this repo.

  • -c or --config: path to osqueryIR configuration. By default it will be config.yaml in this repo.

  • -o or --output: the output file (zip file) name, By default it will be the machine hostname.

  • -q or --quit: Do not print logging to the stdout. osqueryIR will always write the log to the output file.

  • --log-file-name: change the default name for the log file (osqueryIR_log).

  • --log-level: the log level, default is INFO.

  • --output-format: osqueryIR support writing the results in two different formats:

    • jsonl: a newline separated JSON object. Each object represent a record.
    • kjson: the format understood by Kuiper. If you are planing to use Kuiper for analysis then you should use this format.
  • --disable-collect: disable artifact collecting. Only parsing and osquery artifacts will be acquired.

Configuration

osqueryIR accepts a configuration file that contains artifact specification. The following is an example configuration along with comments:

artifacts:  
    # Name of artifact. the results will be saved to a file with this name
  - logged_in_users:
      # artifact type. queries run osquery SQL queries and return the results as json
      queries:
        - 'select * from logged_in_users'
      # Optional: map the field called `name` to `@timestamp` and run the modfier `epoch_to_iso` on the value. `modifier` field is not required
      maps:
        - name: time
          map_to: '@timestamp'
          modifier: epoch_to_iso
      # Optional: description of the artifact
      description: 'Collect and parse the currently loggedin users'
  - logs:
      # artifact type. collect the specified files and directories without parsing
      collect:
        - '/var/log/**'
        - '/home/*/.vnc/*.log'
      description: 'Collect logs wellknow paths'
  - auth_log:
      # artifact type. parse the specified files using regex and return the results as json.
      parse:
        # files to parse
        path: '/var/log/auth.log*'
        # regex used for parsing
        regex: '([A-Z][a-z]{2}[ ]{1,}[0-9]{1,2}[ ]{1,2}[0-9]{1,2}:[0-9]{2}:[0-9]{2}) ([a-zA-Z0-9_\-]+) ([a-zA-Z0-9_\-\]\(\)=\./]+)\[?([0-9]+)?\]?: (.*)'
        # the name of the extracted fields
        fields:
          - 'time'
          - 'hostname'
          - 'service'
          - 'pid'
          - 'msg'
      maps:
        - name: time
          map_to: '@timestamp'
          modifier: time_without_year_to_iso
      description: 'Parse auth logs from the path /var/log/, and return the results as jsonl/kjson'
  - bad_logins:
  	  # artifact type. Execute system command and return stdout & stderr
      command:
        - 'lastb'

Example

To collect the artifacts from the provided configurations, execute the following command:

python3 osqueryIR.py

A file will be created named {HOSTNAME}.zip that contains all artifacts.

Useing osqueryIR with Kuiper

osqueryIR can generate the result in kjson format which could be ingested by Kuiper. To collect artifacts in kjson format execute the following command:

python3 osqueryIR.py --output-format kjson --disable-collect

upload the file to Kuiper and execute the kjson parser

osqueryIR_Kuiper

Owner
AbdulRhman Alfaifi
AbdulRhman Alfaifi
A collection of utility functions to prototype geometry processing research in python

gpytoolbox This repo is a work in progress and contains general utility functions I have needed to code while trying to work on geometry process resea

Silvia Sellán 73 Jan 06, 2023
A python tool give n number of inputs and parallelly you will get a output by separetely

http-status-finder Hello Everyone!! This is kavisurya, In this tool you can give n number of inputs and parallelly you will get a output by separetely

KAVISURYA V 3 Dec 05, 2021
Compute the fair market value (FMV) of staking rewards at time of receipt.

tendermint-tax A tool to help calculate the tax liability of staking rewards on Tendermint chains. Specifically, this tool calculates the fair market

5 Jan 07, 2022
An URL checking python module

An URL checking python module

Fayas Noushad 6 Aug 10, 2022
A collection of tools for biomedical research assay analysis in Python.

waltlabtools A collection of tools for biomedical research assay analysis in Python. Key Features Analysis for assays such as digital ELISA, including

Tyler Dougan 1 Apr 18, 2022
pydsinternals - A Python native library containing necessary classes, functions and structures to interact with Windows Active Directory.

pydsinternals - Directory Services Internals Library A Python native library containing necessary classes, functions and structures to interact with W

Podalirius 36 Dec 14, 2022
tade is a discussion/forum/link aggregator application. It provides three interfaces: a regular web page, a mailing list bridge and an NNTP server

tade is a discussion/forum/link aggregator application. It provides three interfaces: a regular web page, a mailing list bridge and an NNTP server

Manos Pitsidianakis 23 Nov 04, 2022
Homebase Name Changer for Fortnite: Save the World.

Homebase Name Changer This program allows you to change the Homebase name in Fortnite: Save the World. How to use it? After starting the HomebaseNameC

PRO100KatYT 7 May 21, 2022
A Python library for reading, writing and visualizing the OMEGA Format

A Python library for reading, writing and visualizing the OMEGA Format, targeted towards storing reference and perception data in the automotive context on an object list basis with a focus on an urb

Institut für Kraftfahrzeuge, RWTH Aachen, ika 12 Sep 01, 2022
Color box that provides various colors‘ rgb decimal code.

colorbox Color box that provides various colors‘ rgb decimal code

1 Dec 07, 2021
Make some improvements in the Pizza class and pizzashop file by refactoring.

Make some improvements in the Pizza class and pizzashop file by refactoring.

James Brucker 1 Oct 18, 2021
Tool for generating Memory.scan() compatible instruction search patterns

scanpat Tool for generating Frida Memory.scan() compatible instruction search patterns. Powered by r2. Examples $ ./scanpat.py arm.ks:64 'sub sp, sp,

Ole André Vadla Ravnås 13 Sep 19, 2022
Tools for binary data on cassette

Micro Manchester Tape Storage Tools for storing binary data on cassette Includes: Python script for encoding Arduino sketch for decoding Eagle CAD fil

Zack Nelson 28 Dec 25, 2022
UUID version 7, which are time-sortable (following the Peabody RFC4122 draft)

uuid7 - time-sortable UUIDs This module implements the version 7 UUIDs, proposed by Peabody and Davis in https://www.ietf.org/id/draft-peabody-dispatc

Steve Simmons 22 Dec 20, 2022
ecowater-softner is a Python library for collecting information from Ecowater water softeners.

Ecowater Softner ecowater-softner is a Python library for collecting information from Ecowater water softeners. Installation Use the package manager p

6 Dec 08, 2022
A simple python implementation of Decision Tree.

DecisionTree A simple python implementation of Decision Tree, using Gini index. Usage: import DecisionTree node = DecisionTree.trainDecisionTree(lab

1 Nov 12, 2021
A simple Python app that generates semi-random chord progressions.

chords-generator A simple Python app that generates semi-random chord progressions.

53 Sep 07, 2022
A Tool that provides automatic kerning for ligature based OpenType fonts in Microsoft Volt

Kerning A Tool that provides automatic kerning for ligature based OpenType fonts in Microsoft Volt There are three stages of the algorithm. The first

Sayed Zeeshan Asghar 6 Aug 01, 2022
Cleaning-utils - a collection of small Python functions and classes which make cleaning pipelines shorter and easier

cleaning-utils [] [] [] cleaning-utils is a collection of small Python functions

4 Aug 31, 2022
🍰 ConnectMP - An easy and efficient way to share data between Processes in Python.

ConnectMP - Taking Multi-Process Data Sharing to the moon 🚀 Contribute · Community · Documentation 🎫 Introduction : 🍤 ConnectMP is the easiest and

Aiden Ellis 1 Dec 24, 2021