A lightweight python module for building event driven distributed systems

Overview

Build Status Coverage Status Documentation Status

Eventify

A lightweight python module for building event driven distributed systems.

Installation

pip install eventify

Problem

Developers need a easy and fast way to implement business logic in a modern asynchronous way. Developers should not have to worry about service communication.

Contributing

TBD

Architecture

ArchDiagram

Tutorial

Below is everything you need to get started using the eventify project.

Minimum Viable Service

Create an Event handler function to receive events. To publish events, use session.emit_event() within the Event handler function. To start your service (and begin listening for events), pass the Event handler function as a callback to run a Service.

#!/usr/bin/env python
"""
Minimal viable microservice
"""
import asyncio
import logging
import sys

from eventify.base_handler import BaseHandler
from eventify.event import Event
from eventify.service import Service


class GoogleCollector:
    """
    Google Specific Collector
    """

    async def collect_vm_data(self):
        print('...collecting data from gce api...')
        await asyncio.sleep(1)

class Collector(BaseHandler, GoogleCollector):
    """
    Generic collector
    """

    async def init(self):
        """
        Service initialization
        """
        print('...service initialized...')


def run():
    """
    Run an eventify service
    """
    Service(
        config_file='config.json',
        handlers=[Collector]
    ).start()

if __name__ == '__main__':
    run()

Minimum Configuration - config.json

Each application (service, microservice, etc.) can publish to 0 or 1 topic and subscribe to 0 or more topics. Thus each application requires its own configuration file which defines its publications & subscriptions. Each configuration file also specifies the location of the transport host.

{
    "_comment": "service configuration",
    "name": "example",
    "image": "example/service",
    "driver": "crossbar",
    "transport_host": "ws://events-server:8080/ws",
    "pub_options": {
        "acknowledge": true,
        "retain": true
    },
    "publish_topic": {
        "topic": "example",
        "timeout": 20,
        "reply_in": 0
    },
    "subscribed_topics": [
        "ui-events"
    ],
    "replay_events": true,
    "replay_type": "event_store"
}

Event History Support

To add event history support add the @event_tracker decorator to any event handler your define.

Environment variables

  • EVENT_DB_HOST

Hostname of database server

  • EVENT_DB_USER

Username for database server

  • EVENT_DB_PASS

Password for username

  • EVENT_DB_TYPE

Database driver type - defaults to postgres

  • EVENT_DB_POOL_SIZE

Database connection pool size - defaults to 5

  • EVENT_TRACKING_HOST

For use with replay functionality - host of in memory cache

Supported Drivers

  • Crossbar
  • Kafka

Crossbar

Quickest way to get started with crossbar is to use docker; both redis and postgresql are required for a minimal working service.

Running Crossbar

docker run -d -p 8080:8080 -it crossbario/crossbar:latest

Running Kafka

  • TBD

Running Redis

docker run -d -p 6379:6379 -t redis:latest

Running Postgresql

docker run --name postgres-container -p 5432:5432 -e POSTGRES_PASSWORD=mysecretpassword -d postgres

Note: This does not have a volume or persistant storage so in a production environment you would want to configure a volume

See More

http://crossbar.io

Pypi

https://pypi.python.org/pypi/eventify

Read The Docs

http://eventify.readthedocs.io/en/latest/

Owner
Eventify
Event driven service design
Eventify
Framework and Library for Distributed Online Machine Learning

Jubatus The Jubatus library is an online machine learning framework which runs in distributed environment. See http://jubat.us/ for details. Quick Sta

Jubatus 701 Nov 29, 2022
PowerGym is a Gym-like environment for Volt-Var control in power distribution systems.

Overview PowerGym is a Gym-like environment for Volt-Var control in power distribution systems. The Volt-Var control targets minimizing voltage violat

Siemens 44 Jan 01, 2023
Deluge BitTorrent client - Git mirror, PRs only

Deluge is a BitTorrent client that utilizes a daemon/client model. It has various user interfaces available such as the GTK-UI, Web-UI and a Console-UI. It uses libtorrent at it's core to handle the

Deluge team 1.3k Jan 07, 2023
An distributed automation framework.

Automation Kit Repository Welcome to the Automation Kit repository! Note: This package is progressing quickly but is not yet ready for full production

Automation Mojo 3 Nov 03, 2022
ZeroNet - Decentralized websites using Bitcoin crypto and BitTorrent network

ZeroNet Decentralized websites using Bitcoin crypto and the BitTorrent network - https://zeronet.io / onion Why? We believe in open, free, and uncenso

ZeroNet 17.8k Jan 03, 2023
Distributed-systems-algos - Distributed Systems Algorithms For Python

Distributed Systems Algorithms ISIS algorithm In an asynchronous system that kee

Tony Joo 2 Nov 30, 2022
Distributed machine learning platform

Veles Distributed platform for rapid Deep learning application development Consists of: Platform - https://github.com/Samsung/veles Znicz Plugin - Neu

Samsung 897 Dec 05, 2022
Distributed Synchronization for Python

Distributed Synchronization for Python Tutti is a nearly drop-in replacement for python's built-in synchronization primitives that lets you fearlessly

Hamilton Kibbe 4 Jul 07, 2022
Privacy enhanced BitTorrent client with P2P content discovery

Tribler Towards making Bittorrent anonymous and impossible to shut down. We use our own dedicated Tor-like network for anonymous torrent downloading.

4.2k Dec 31, 2022
Ray provides a simple, universal API for building distributed applications.

An open source framework that provides a simple, universal API for building distributed applications. Ray is packaged with RLlib, a scalable reinforcement learning library, and Tune, a scalable hyper

23.5k Jan 05, 2023
Python Stream Processing

Python Stream Processing Version: 1.10.4 Web: http://faust.readthedocs.io/ Download: http://pypi.org/project/faust Source: http://github.com/robinhood

Robinhood 6.4k Jan 07, 2023
Run Python in Apache Storm topologies. Pythonic API, CLI tooling, and a topology DSL.

Streamparse lets you run Python code against real-time streams of data via Apache Storm. With streamparse you can create Storm bolts and spouts in Pyt

Parsely, Inc. 1.5k Dec 22, 2022
蓝鲸基础计算平台(BK-BASE)是一个专注于运维领域的的基础平台,打造一站式、低门槛的基础服务

蓝鲸基础计算平台(BK-BASE)是一个专注于运维领域的的基础平台,打造一站式、低门槛的基础服务。通过简化运维数据的收集、获取,提升数据开发效率,辅助运维人员实时运维决策,助力企业运营体系数字化、智能化转型。

Tencent 80 Dec 16, 2022
An open source framework that provides a simple, universal API for building distributed applications. Ray is packaged with RLlib, a scalable reinforcement learning library, and Tune, a scalable hyperparameter tuning library.

Ray provides a simple, universal API for building distributed applications. Ray is packaged with the following libraries for accelerating machine lear

23.2k Dec 30, 2022
Run MapReduce jobs on Hadoop or Amazon Web Services

mrjob: the Python MapReduce library mrjob is a Python 2.7/3.4+ package that helps you write and run Hadoop Streaming jobs. Stable version (v0.7.4) doc

Yelp.com 2.6k Dec 22, 2022
Microsoft Distributed Machine Learning Toolkit

DMTK Distributed Machine Learning Toolkit https://www.dmtk.io Please open issues in the project below. For any technical support email to

Microsoft 2.8k Nov 19, 2022
PArallel Distributed Deep LEarning: Machine Learning Framework from Industrial Practice (『飞桨』核心框架,深度学习&机器学习高性能单机、分布式训练和跨平台部署)

English | 简体中文 Welcome to the PaddlePaddle GitHub. PaddlePaddle, as the only independent R&D deep learning platform in China, has been officially open

19.4k Dec 30, 2022
A lightweight python module for building event driven distributed systems

Eventify A lightweight python module for building event driven distributed systems. Installation pip install eventify Problem Developers need a easy a

Eventify 16 Aug 18, 2022
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

Luigi is a Python (3.6, 3.7 tested) package that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow managemen

Spotify 16.2k Jan 01, 2023
Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.

Horovod Horovod is a distributed deep learning training framework for TensorFlow, Keras, PyTorch, and Apache MXNet. The goal of Horovod is to make dis

Horovod 12.9k Dec 29, 2022