Python Service for MISP Feed Management

Overview

Python Service for MISP Feed Management

This set of scripts is designed to offer better reliability and more control over the fetching of feeds into MISP. For the moment, the schedule is broken up into multiple components, at the top of each plugin and in config.py:

  • MISP_TIMES: An array of times (24hr format) when enabled MISP feeds will be fetched and cached.
  • TEXT_TIMES: An array of times (24hr format) when enabled plaintext and CSV feeds will be fetched and cached.
  • HOURLY_FEEDS An array of the ID's of enabled feeds that you wish to run at the beginning of every hour.
  • FULL_EXPORT_TIME The time (24hr format) that you want to run a full text export of attributes.

In addition to this are "ENABLE" options for all external services. By default, Abuse.ch is configured to run every hour.

Am still working out the best way of going about granular scheduling.

Variable Notes:

  • MISP_ADMIN_KEY: MISP feeds must be fetched by a Site Admin user.
  • MISP_USER_KEY: This can be the key of an Org Admin, Sync User or your own custom role. They must be able to both manage and publish events, and hold the Tag Editor permission.

Installation:

  • Recommended: Ensure that the fetch_feeds and cache_feeds Scheduled Tasks are not enabled. Also, disable the default Abuse.ch feeds as this project includes a module that loads the data with more context and into a separate event each day.
  • SCP this folder to your MISP server.
  • Alter the paths in misp-feeds.service and start_worker.sh to point to where you've dropped the folder.
  • Correct the user in misp-feeds.service if it is not ubuntu.
  • Complete the variables at the top of the feed_manager.py, misp_export.py, otx_misp.py, twitter_misp.py and xforce_misp.py scripts.
  • Run the following (in the misp-feeds folder):
chmod +x start_worker.sh
apt install nodejs
pip3 install -r requirements.txt
sudo mv misp-feeds.service /etc/systemd/system
sudo chown root:root /etc/systemd/system/misp-feeds.service
sudo systemctl daemon-reload
sudo systemctl start misp-feeds.service
  • nodejs is required for cfscrape (used by the Twitter module to get Ghostbin pastes).
  • Check misp_feeds.log for errors. You can also run both of the Python scripts from the command line for standalone, ad-hoc operation.

Module Notes:

Export:

  • This is a rough script that I use for exporting a plaintext list of attributes for ingestion into external facilities. They're output to a subfolder of the MISP webroot, so ensure the script user has permission to write here and there's adequate access control in place.
  • A full export is run once a day for the number of days defined by EXPORT_DAYS. Incremental updates are made daily.
  • The sample values for EXPORT_TAGS and EXPORT_TYPES should give you an idea of how to configure this. 'domain' and 'hostname' can be output separately or together. Use EXPORT_MERGE_HOSTNAME to configure this.

Plugins:

At the top of each plugin are three variables which determine its operation:

  • PLUGIN_NAME: The friendly name of the Plugin. Only used for logging and ad-hoc operation.

  • PLUGIN_ENABLED: Boolean setting to enable/disable the plugin.

  • PLUGIN_TIMES: The times throughout the day to run the plugin. Also accepts 'hourly', which will run it on the hour every hour.

Default plugins are as follows:

  • Abuse.ch: Pulls URLhaus, Feodo Tracker, MalwareBazaar and ThreatFox into a single event per day. Attributes are tagged according to the feed tags and/or classification.
  • CleanMX: Virus and Phishing feeds are pulled into a single event per day. No tagging yet.
  • OTX: Individual pulses form a separate events in MISP. OTX tags can be spammy so are ignored, but Adversary, Malware and ATT&CK techniques are used. Galaxy tags are attempted, and if no appropriate tag can be found, the feed supplied tag is used.
  • RiskIQ: Individual articles form a separate events in MISP. The same method of tagging is employed as OTX.
  • Twitter: Pulls IOC's found on Twitter into a single event per day. GitHub, PasteBin and GhostBin links are followed and also scraped. Attributes are tagged with the hashtags included in the Tweet and the same method as OTX.
  • X-Force: Individual articles form a separate events in MISP. X-Force articles are not tagged, so the Title of the article is parsed to identify Galaxy tags that match Title keywords.
Owner
Chris
Security Architect / Malware Wrangler
Chris
Tools Elit Adalah Sebuah Script Crack Yang Wajib Tap Yes...

Tools Elit Adalah Sebuah Script Crack Yang Wajib Tap Yes...

Risky [ Zero Tow ] 10 Apr 07, 2022
BlackMamba is a multi client C2/post exploitation framework

BlackMamba is a multi client C2/post exploitation framework with some spyware features. Powered by Python 3.8.6 and QT Framework.

Gustavo 873 Dec 29, 2022
Xkcd.py - Script to generate wallpapers based on XKCD comics

xkcd.py Script to generate wallpapers based on XKCD comics Usage python3 xkcd.py

Gideon Wolfe 11 Sep 06, 2022
decorator

Decorators for Humans The goal of the decorator module is to make it easy to define signature-preserving function decorators and decorator factories.

Michele Simionato 734 Dec 30, 2022
Python Repository for Bachelor Ski Sign.

BachelorSkiSign Python Repository for Bachelor Ski Sign. This application reads data from https://bachelorapi.azurewebsites.net/ It is written in Ciru

Winston 1 Jan 04, 2022
MatroSka Mod Compiler for ts4scripts

MMC Current Version: 0.2 MatroSka Mod Compiler for .ts4script files Requirements Have Python 3.7 installed and set as default. Running from Source pip

MatroSka 1 Dec 13, 2021
LanguageCreator - Simple library for easy creation transpilator.

LanguageCreator - Simple library for easy creation transpilator. Create transpilators in one hour! Install. Download code, rename folder to "LanguageC

Ivan Perzhinsky. 2 Dec 31, 2021
Your missing PO formatter and linter

pofmt Your missing PO formatter and linter Features Wrap msgid and msgstr with a constant max width. Can act as a pre-commit hook. Display lint errors

Frost Ming 5 Mar 22, 2022
A Python application that simulates the rolling of a dice, randomly picking one of the 6 faces and then displaying it.

dice-roller-app This is an application developed in Python that shuffles between the 6 faces of a dice, using buttons to shuffle and close the applica

Paddy Costelloe 0 Jul 20, 2021
NFT generator for Solana!

Solseum NFT Generator for Solana! Check this guide here! Creating your randomized uniques NFTs, getting rarity information and displaying it on a webp

Solseum™ VR NFTs 145 Dec 30, 2022
A MCPI hack with many features.

Morpheus 2.0 A MCPI hack with many features To Use: You will need to install the keyboard, pysimplegui, and MCPI python modules and you will need to e

11 Oct 11, 2022
BMI-Calculator: Program to Calculate Body Mass Index (BMI)

The Body Mass Index (BMI) or Quetelet index is a value derived from the mass (weight) and height of an individual, male or female.

PyLaboratory 0 Feb 07, 2022
NCAR/UCAR virtual Python Tutorial Seminar Series lesson on MetPy.

The Project Pythia Python Tutorial Seminar Series continues with a lesson on MetPy on Wednesday, 2 February 2022 at 1 PM Mountain Standard Time.

Project Pythia Tutorials 6 Oct 09, 2022
Artificial intelligence based on 5-dimensional quantum selection

Deep Thought An artificial intelligence based on 5-dimensional quantum selection. Algorithm The payload Make an random bit array (e.g. 1101...) Conver

Larry Holst 3 Dec 14, 2022
Lagrange Interpolation Method-Python

Lagrange Interpolation Method-Python The Lagrange interpolation formula is a way to find a polynomial, called Lagrange polynomial, that takes on certa

Motahare Soltani 2 Jul 05, 2022
🤖️ Plugin for Sentry which allows sending notification via DingTalk robot.

Sentry DingTalk Sentry 集成钉钉机器人通知 Requirments sentry = 21.5.1 特性 发送异常通知到钉钉 支持钉钉机器人webhook设置关键字 配置环境变量 DINGTALK_WEBHOOK: Optional(string) DINGTALK_CUST

1 Nov 04, 2021
Exercicios de Python do Curso Em Video, apresentado por Gustavo Guanabara.

Exercicios Curso Em Video de Python Exercicios de Python do Curso Em Video, apresentado por Gustavo Guanabara. OBS.: Na data de postagem deste repo já

Lorenzo Ribeiro Varalo 0 Oct 21, 2021
objectfactory is a python package to easily implement the factory design pattern for object creation, serialization, and polymorphism

py-object-factory objectfactory is a python package to easily implement the factory design pattern for object creation, serialization, and polymorphis

Devin A. Conley 6 Dec 14, 2022
Usos Semester average helper

Usos Semester average helper Dzieki temu skryptowi mozesz sprawdzic srednia ocen na kazdy odbyty przez ciebie semestr PARAMETERS required: '--username

2 Jan 17, 2022
A basic ticketing software.

Ticketer A basic ticketing software. Screenshots Program Launched Issuing Ticket Show your Ticket Entry Done Program Exited Code Features to implement

Samyak Jain 2 Feb 10, 2022