PyWorkflow(PyWF) - A Python Binding of C++ Workflow

Overview

PyWorkflow(PyWF) - A Python Binding of C++ Workflow

License Language PyPI - Python Version PyPI

概览

C++ Workflow是一个高性能的异步引擎,本项目着力于实现一个Python版的Workflow,让Python用户也能享受Workflow带来的绝佳体验。

快速上手

在用户深入了解Workflow相关概念之前,先来看几个简单的示例,可以对使用方法有一个初步印象。pywf是本项目Python包的名称,在文档中有时会直接使用wf作为其简称。

发起一个Http请求

import pywf as wf

def http_callback(http_task):
    resp = http_task.get_resp()
    print("Http status:{}\n{}".format(
        resp.get_status_code(), resp.get_body())) # body is bytes

http_task = wf.create_http_task("http://www.sogou.com/", redirect_max=4, retry_max=2, callback=http_callback)
http_task.start()
wf.wait_finish()

依次发起多个Http请求

import pywf as wf

def series_callback(s):
    print("All task in this series is done")

def http_callback(http_task):
    req = http_task.get_req()
    resp = http_task.get_resp()
    print("uri:{} status:{}".format(
        req.get_request_uri(),
        resp.get_status_code()))

def create_http_task(url):
    return wf.create_http_task(url, 4, 2, http_callback)

first_task = create_http_task("http://www.sogou.com")
series = wf.create_series_work(first_task, series_callback)
series.push_back(create_http_task("https://www.zhihu.com/people/kedixa"))
series.push_back(create_http_task("https://fanyi.sogou.com/document"))
series.start()
wf.wait_finish()

同时发起多个Http请求

import pywf as wf

def parallel_callback(p):
    print("All series in this parallel is done")

def http_callback(http_task):
    req = http_task.get_req()
    resp = http_task.get_resp()
    print("uri:{} status:{}".format(
        req.get_request_uri(),
        resp.get_status_code()))

url = [
    "http://www.sogou.com",
    "https://www.zhihu.com/people/kedixa",
    "https://fanyi.sogou.com/document"
]
parallel = wf.create_parallel_work(parallel_callback)
for u in url:
    task = wf.create_http_task(u, 4, 2, http_callback)
    series = wf.create_series_work(task, None) # without callback
    parallel.add_series(series)
parallel.start()

wf.wait_finish()

基本概念

任务

通过create_xxx_task等工厂函数创建的对象称作任务(task),例如create_http_task。一个任务被创建后,必须被启动或取消,通过执行http_task.start(),会自动以http_taskfirst_task创建一个串行并立即启动任务。如果用户指定了回调函数,当任务完成时回调函数会被调用,但在任务启动后且回调函数前,用户不能再操作该任务。当回调函数结束后,该任务被立即释放,用户也不能再操作该任务。

串行

通过create_series_work创建的对象称作串行(series),用户在创建时需要指定一个first_task来作为启动该series启动时应当执行的任务,用户可选地指定一个回调函数,当所有任务执行完成后,回调函数会被调用。

series的回调函数用于通知用户该串行中的任务均已完成,不能再继续添加新的任务,且回调函数结束后,该串行会立即被销毁。

并行

通过create_parallel_work创建的对象称作并行(parallel),用户可以创建一个空的并行,然后通过add_series接口向并行中添加串行,也可以在创建时指定一组串行。并行本身也是一种任务,所以并行也可以放到串行中。parallel.start()就会自动创建一个串行,并将parallel作为first_task立即开始执行。

parallel的回调函数用于通知用户该并行中的串行均已完成,不能再继续添加新的串行,且回调函数结束后,该并行会立即被销毁。

有了上述三个概念,就可以构建出各种复杂的任务结构,并在Workflow的管理下高效执行。

设计理念

Workflow认为,一个典型的后端程序由三个部分组成,并且完全独立开发。即:程序=协议+算法+任务流。

  • 协议
    • 大多数情况下,用户使用的是内置的通用网络协议,例如http,redis或各种rpc。
    • PyWF未支持用户自定义协议。
  • 算法
    • 算法是与协议对称的概念。
    • 如果说协议的调用是rpc,算法的调用就是一次apc(Async Procedure Call)。
    • 任何一次边界清晰的复杂计算,都应该包装成算法。
  • 任务流
    • 任务流就是实际的业务逻辑,就是把开发好的协议与算法放在流程图里使用起来。
    • 典型的任务流是一个闭合的串并联图。复杂的业务逻辑,可能是一个非闭合的DAG。
    • 任务流图可以直接构建,也可以根据每一步的结果动态生成。所有任务都是异步执行的。

Python Workflow将会逐步支持Workflow的六种基础任务:通讯,文件IO,CPU,GPU,定时器,计数器。

注意事项

  • 框架本身不抛出异常,也未处理任何异常,所以用户需要保证回调函数不会抛出异常,context的构造和析构不抛出异常。
  • 所有通过工厂函数创建出的task,必须start、dismiss或添加至一个series中。
  • 所有创建出的series必须start、dismiss或添加至一个parallel中。
  • 所有创建出的parallel必须start、dismiss或添加至一个series中。
  • 由PyWF工厂函数创建的对象的生命周期均由内部管理,在Python层面仅是一个引用,用户不能使用超出生命周期的对象。
  • 用户使用大部分get接口获取的对象可以自由使用,例如Http中的get_body

构建和安装

通过pip安装

本项目仅支持Python3.6以上,正在准备发布一组manylinux2014版本,用户即将可以通过较高版本的pip直接安装。

# We are working on it
pip3 install pywf

编译安装

用户可以下载本项目源码进行编译安装。

# CentOS 7
yum install cmake3 ninja-build python36 python36-devel python36-pip
yum install gcc-c++ # if needed
git clone https://github.com/sogou/pyworkflow --recursive
cd pyworkflow
pip3 install wheel
python3 setup.py bdist_wheel
pip3 install dist/*.whl --user
# CentOS 8
yum install cmake ninja-build python36 python36-devel python3-pip
git clone https://github.com/sogou/pyworkflow --recursive
cd pyworkflow
pip3 install wheel
python3 setup.py bdist_wheel
pip3 install dist/*.whl --user
Owner
Sogou-inc
Sogou-inc
Plock : A stack based programming language

Plock : A stack based programming language

1 Oct 25, 2021
BDD base project: Python + Behave

BDD base project: Python + Behave Basic example of using Python with Behave (BDD). This Gherkin example includes: Basic Scenario Scenario Outline Tagg

eccanto 1 Dec 08, 2021
CarolinaCon CTF Online

CarolinaCon Online CTF CTF challenges from CarolinaCon Online April 23 through April 25, 2021. All challenges from the CTF will eventually be here. Co

49th Security Division 6 May 04, 2022
Python tools for working with Orbit Ephemeris Messages (OEMs).

Python Orbit Ephemeris Message tools Python tools for working with Orbit Ephemeris Messages (OEMs). Development Status Installation The oem package is

Brad Sease 4 Apr 06, 2022
Print 'text color' and 'text format' on Term with Python

term-printer Print 'text color' and 'text format' on Term with Python ※ It may not work depending on the OS and shell used. PIP $ pip install term-pri

ななといつ 10 Nov 12, 2022
In this project, we are going to display the battery notification and the time left for the battery to drain out using the battery capacity value.

In this project, we are going to display the battery notification and the time left for the battery to drain out using the battery capacity value.

Ritoban Biswas 1 Dec 20, 2021
Easily Generate Revolut Business Cards

RevBusinessCardGen Easily Generate Revolut Business Cards Prerequisites Before you begin, ensure you have met the following requirements: You have ins

Younes™ 35 Dec 14, 2022
LOL英雄联盟云顶之弈挂机刷代币脚本,全自动操作,智能逻辑,功能齐全。

LOL云顶之弈挂机刷代币脚本 这是2019年全球总决赛写的一个云顶挂机脚本,python完成的。 功能: 自动拿牌卖牌 策略是高星策略,非固定阵容 自动登陆账号、打码、异常重启 战利品截图上传百度云 web中控发号,改密码,查看信息等 代码是三天赶出来的,所以有点混乱,WEB中控代码也不知道扔哪去了

77 Oct 10, 2022
Interactive class notebooks for ECE4076 Computer Vision, weeks 1 - 6

ECE4076 Interactive class notebooks for ECE4076 Computer Vision, weeks 1 - 6. ECE4076 is a computer vision unit at Monash University, covering both cl

Michael Burke 9 Jun 16, 2022
VCM EE1.2 P-layer feature map anchor generation 137th MPEG-VCM

VCM EE1.2 P-layer feature map anchor generation 137th MPEG-VCM

IPSL 6 Oct 18, 2022
Build your own Etherscan with web3.py

Build your own Etherscan with web3.py Video Tutorial: Run it pip3 install -r requirements.txt export FLASK_APP=app export FLASK_ENV=development flask

35 Jan 02, 2023
Freeze your objects in python

gelidum Freeze your objects in python. Latin English Caelum est hieme frigidum et gelidum; myrtos oleas quaeque alia assiduo tepore laetantur, asperna

Diego J. 51 Dec 22, 2022
Plux - A dynamic code loading framework for building plugable Python distributions

Plux plux is the dynamic code loading framework used in LocalStack. Overview The

LocalStack 65 Dec 20, 2022
Application launcher and environment management

Application launcher and environment management for 21st century games and digital post-production, built with bleeding-rez and Qt.py News Date Releas

10 Nov 03, 2022
Qt-creator-boost-debugging-helper - Qt Creator Debugging Helper for Boost Library

Go to Tools Options Debugger Locals & Expressions. Paste the script path t

Dmitry Bravikov 2 Apr 22, 2022
Materials for the Introduction in Python , Linux , Git and Github

This repository contains all the materials of the presentation on the introduction of python, linux, git and Github.

AMMI 3 Aug 28, 2022
Grade 8 Version of Space Invaders

Space-Invaders Grade 8 Version of Space Invaders Compatability This program is Python 3 Compatable, and not Python 2 Compatable because i haven't test

Space64 0 Feb 16, 2022
Leveraging pythonic forces to defeat different coding challenges 🐍

Pyforces Leveraging pythonic forces to defeat different coding challenges! Table of Contents Pyforces Tests Pyforces Pyforces is a study repo with a c

Igor Grillo Peternella 8 Dec 14, 2022
Python-Kite: Simple python code to make kite pattern

Python-Kite Simple python code to make kite pattern. Getting Started These instr

Anoint 0 Mar 22, 2022
A free micro-blog written in Python and powered by Heroku. *Merge requests are appreciated!*

Background Hobo is an ultra-lightweight blog engine written in Python. It has two dependencies, fully integrated into the codebase with no additional

Andrew Nelder 48 Jan 28, 2021