PipeLayer is a lightweight Python pipeline framework

Overview

PipeLayer

PipeLayer is a lightweight Python pipeline framework. Define a series of steps, and chain them together to create modular applications.

Table of Contents

Installation

From the command line:

pip install pipelayer

Getting Started

Step 1: Create Pipeline Filters

hello_world_filters.py

from pipelayer import Filter


class HelloFilter(Filter):
    def run(self, data, context):
        return "Hello"


class WorldFilter(Filter):
    def run(self, data, context):
        return f"{data},  World!"

functions.py

def create_message_dict(data, context):
    return {"message": data}

Step 2: Create a Pipeline

Create a module to run the pipeline:

app.py

from pipelayer import Pipeline

from functions import create_message
from hello_world_filters import HelloFilter, WorldFilter


if __name__ = "__main__":
    hello_world_pipeline = Pipeline([
        HelloFilter,                           # pipeline.Filter type
        WorldFilter,                           # pipeline.Filter instance
        create_message_dict                    # function type
        lambda data, context: json.dumps(data) # anonymous function
    ])

    output = hello_world_pipeline.run()

    # output = '{"message": "Hello, World!"}'

    print(f"Pipeline Output: {output}")
    print(hello_world_pipeline.manifest.__dict__)

Step 3: Run the Pipeline

from the command line:

run app.py

The Framework

pipelayer.Pipeline

__init__(steps, name)

args:

  • steps: List[Union[Step, Callable[[Any, Context], Any]]]
    A list of:

    • Classes and Instances that derive from pipelayer.Filter and implement the run method

    • Classes that implement the pipelayer.Step protocol

    • Functions (instance/class/static/module) that have the following signature

      def func(data: Any, context: Any)
    • Anonymous functions (lambda) with two arguments that follow this pattern:

      my_func = lambda data, context: data
    • Instances of pipelayer.Pipeline

  • name: Optional[str]
    If not specified, the class name will be used.

Properties:

name: str

state: Pipeline.State

steps: List[Union[Step, Callable[[Any, Context], Any]]]

manifest: Manifest
An instance of pipelayer.Manifest that is created when the run method is called.

Methods:

run(data, context) -> Any
The pipeline runner that iterates through the steps and pipes filter output to the next step.

args:

  • data: Any
  • context: pipelayer.Context

pipelayer.Switch

__init__(expression, cases, name)
An implementation of a Switch statement as a pipeline filter

args:

  • expression: Union[Step, Callable[[Any, Context], Any]]
  • cases: Dict[Union[Step, Callable[[Any, Context], Any]]]
  • name: Optional[str]
    If not specified, the class name will be used.

Properties:

expression: Union[Step, Callable[[Any, Context], Any]]
cases: Dict[Union[Step, Callable[[Any, Context], Any]]]
name: Optional[str]
manifest: Manifest

Methods:

run(data, context) -> Any
The switch runner that evaluates the specified expresssion executes the matching case.

pipelayer.Filter

__init__(name, pre_process, post_process)

args:

  • name: Optional[str]
    If not specified, the class name will be used.
  • pre_process: Optional[Callable[[Any, Context], Any]
  • post_process: Optional[Callable[[Any, Context], Any]

Properties:

pre_process: Optional[Callable[[Any, Context], Any]
post_process: Optional[Callable[[Any, Context], Any]

Events:
Events are lists of callables assignable after instantiation and are raised if the pipelayer.filter.raise_events decorator is applied to the implementation of the run method.

start: List[Callable[[Filter, Any], Any]]
Raised before the run method is invoked.

exit: List[Callable[[Filter, Any], Any]]
Raised if action is set to Action.SKIP or Action.EXIT in either a start or stop event handler.

end: List[Callable[[Filter, Any], Any]]
Raised after the run method is invoked.

pipelayer.FilterEventArgs

__init__(data, context, state)

args:

  • data: Any
  • context: Context
  • state: State

pipelayer.Context

A abstract base class for runtime app data.

pipelayer.Manifest

The Manifest keeps a record of Pipeline and Filter activity.

Utilities

pipelayer.util.render_manifest(manifest, indent) -> str
Static function that renders formatted JSON data

args:

  • manifest: Manifest
  • indent: Optional[int]
    Default value is 2.
You might also like...
The Python micro framework for building web applications.

Flask Flask is a lightweight WSGI web application framework. It is designed to make getting started quick and easy, with the ability to scale up to co

Tornado is a Python web framework and asynchronous networking library, originally developed at FriendFeed.

Tornado Web Server Tornado is a Python web framework and asynchronous networking library, originally developed at FriendFeed. By using non-blocking ne

Async Python 3.6+ web server/framework | Build fast. Run fast.
Async Python 3.6+ web server/framework | Build fast. Run fast.

Sanic | Build fast. Run fast. Build Docs Package Support Stats Sanic is a Python 3.6+ web server and web framework that's written to go fast. It allow

bottle.py is a fast and simple micro-framework for python web-applications.

Bottle: Python Web Framework Bottle is a fast, simple and lightweight WSGI micro web-framework for Python. It is distributed as a single file module a

Pyramid - A Python web framework

Pyramid Pyramid is a small, fast, down-to-earth, open source Python web framework. It makes real-world web application development and deployment more

The no-nonsense, minimalist REST and app backend framework for Python developers, with a focus on reliability, correctness, and performance at scale.
The no-nonsense, minimalist REST and app backend framework for Python developers, with a focus on reliability, correctness, and performance at scale.

The Falcon Web Framework Falcon is a reliable, high-performance Python web framework for building large-scale app backends and microservices. It encou

web.py is a web framework for python that is as simple as it is powerful.

web.py is a web framework for Python that is as simple as it is powerful. Visit http://webpy.org/ for more information. The latest stable release 0.62

A familiar HTTP Service Framework for Python.
A familiar HTTP Service Framework for Python.

Responder: a familiar HTTP Service Framework for Python Powered by Starlette. That async declaration is optional. View documentation. This gets you a

The Modern And Developer Centric Python Web Framework. Be sure to read the documentation and join the Slack channel questions: http://slack.masoniteproject.com
The Modern And Developer Centric Python Web Framework. Be sure to read the documentation and join the Slack channel questions: http://slack.masoniteproject.com

NOTE: Masonite 2.3 is no longer compatible with the masonite-cli tool. Please uninstall that by running pip uninstall masonite-cli. If you do not unin

Comments
  • Python 3.7 - unable to run sample code

    Python 3.7 - unable to run sample code

    Hi,

    With Python 3.7 I cannot run sample code. I have below error:

    Traceback (most recent call last): File "C:/Github/Python/pipelayers/app.py", line 1, in from pipelayer import Pipeline File "C:\Github\Python\venv\lib\site-packages\pipelayer_init_.py", line 6, in from pipelayer.pipeline import Pipeline # NOQA F401 File "C:\Github\Python\venv\lib\site-packages\pipelayer\pipeline.py", line 10, in from pipelayer.protocol import IFilter, IStep File "C:\Github\Python\venv\lib\site-packages\pipelayer\protocol.py", line 5, in from typing import (Any, Callable, List, Optional, Protocol, Tuple, ImportError: cannot import name 'Protocol' from 'typing' (C:\Program Files\Python\lib\typing.py)

    opened by zizu1985 3
  • Version 0.2.0

    Version 0.2.0

    • Adds support for module/static/lambda funcs as pipeline filters and pre/post process functions
    • Adds support type of Filter as a pipeline filter
    • Example project "simple_pipelayer" can run as a standalone app
    • Initializes/validates all filters/processors before running the pipeline
    opened by greater-than 0
Releases(v0.7.0)
Owner
greaterthan
greaterthan
cirrina is an opinionated asynchronous web framework based on aiohttp

cirrina cirrina is an opinionated asynchronous web framework based on aiohttp. Features: HTTP Server Websocket Server JSON RPC Server Shared sessions

André Roth 32 Mar 05, 2022
Web APIs for Django. 🎸

Django REST framework Awesome web-browsable Web APIs. Full documentation for the project is available at https://www.django-rest-framework.org/. Fundi

Encode 24.7k Jan 03, 2023
A library that makes consuming a RESTful API easier and more convenient

Slumber is a Python library that provides a convenient yet powerful object-oriented interface to ReSTful APIs. It acts as a wrapper around the excellent requests library and abstracts away the handli

Sam Giles 597 Dec 13, 2022
Persistent remote applications for X11; screen sharing for X11, MacOS and MSWindows.

Table of Contents About Installation Usage Help About Xpra is known as "screen for X" : its seamless mode allows you to run X11 programs, usually on a

xpra.org 785 Dec 30, 2022
Dazzler is a Python async UI/Web framework built with aiohttp and react.

Dazzler is a Python async UI/Web framework built with aiohttp and react. Create dazzling fast pages with a layout of Python components and bindings to update from the backend.

Philippe Duval 17 Oct 18, 2022
A Flask API REST to access words' definition

A Flask API to access words' definitions

Pablo Emídio S.S 9 Jul 22, 2022
Fast⚡, simple and light💡weight ASGI micro🔬 web🌏-framework for Python🐍.

NanoASGI Asynchronous Python Web Framework NanoASGI is a fast ⚡ , simple and light 💡 weight ASGI micro 🔬 web 🌏 -framework for Python 🐍 . It is dis

Kavindu Santhusa 8 Jun 16, 2022
A Simple Kivy Greeting App

SimpleGreetingApp A Simple Kivy Greeting App This is a very simple GUI App that receives a name text input from the user and returns a "Hello" greetin

Mariya 40 Dec 02, 2022
Bromelia-hss implements an HSS by using the Python micro framework Bromélia.

Bromélia HSS bromelia-hss is the second official implementation of a Diameter-based protocol application by using the Python micro framework Bromélia.

henriquemr 7 Nov 02, 2022
A microservice written in Python detecting nudity in images/videos

py-nudec py-nudec (python nude detector) is a microservice, which scans all the images and videos from the multipart/form-data request payload and sen

Michael Grigoryan 8 Jul 09, 2022
aiohttp-ratelimiter is a rate limiter for the aiohttp.web framework.

aiohttp-ratelimiter aiohttp-ratelimiter is a rate limiter for the aiohttp.web fr

JGL Technologies 4 Dec 11, 2022
Loan qualifier app - Loan Qualifier Application Built With Python

Loan Qualifier Application This program is designed to automate the discovery pr

Phil Hills 1 Jan 04, 2022
Daniel Vaz Gaspar 4k Jan 08, 2023
The little ASGI framework that shines. ?

✨ The little ASGI framework that shines. ✨ Documentation: https://www.starlette.io/ Community: https://discuss.encode.io/c/starlette Starlette Starlet

Encode 7.7k Jan 01, 2023
A simple todo app using flask and sqlachemy

TODO app This is a simple TODO app made using Flask. Packages used: DoodleCSS Special thanks to Chris McCormick (@mccrmx) :) Flask Flask-SQLAlchemy Fl

Lenin 1 Dec 26, 2021
A public API written in Python using the Flask web framework to determine the direction of a road sign using AI

python-public-API This repository is a public API for solving the problem of the final of the AIIJC competition. The task is to create an AI for the c

Lev 1 Nov 08, 2021
Web3.py plugin for using Flashbots' bundle APIs

This library works by injecting a new module in the Web3.py instance, which allows submitting "bundles" of transactions directly to miners. This is done by also creating a middleware which captures c

Georgios Konstantopoulos 294 Jan 04, 2023
Free and open source full-stack enterprise framework for agile development of secure database-driven web-based applications, written and programmable in Python.

Readme web2py is a free open source full-stack framework for rapid development of fast, scalable, secure and portable database-driven web-based applic

2k Dec 31, 2022
Library for building WebSocket servers and clients in Python

What is websockets? websockets is a library for building WebSocket servers and clients in Python with a focus on correctness and simplicity. Built on

Aymeric Augustin 4.3k Dec 31, 2022
Async Python 3.6+ web server/framework | Build fast. Run fast.

Sanic | Build fast. Run fast. Build Docs Package Support Stats Sanic is a Python 3.6+ web server and web framework that's written to go fast. It allow

Sanic Community Organization 16.7k Dec 28, 2022