This is a survey of python's async concurrency features by example.

Overview

Survey of Python's Async Features

This is a survey of python's async concurrency features by example.

The purpose of this survey is to demonstrate that using Python's async features isn't difficult, and they are simple to use in all IO bound contexts. They can be used simply and to great effect in scripts, existing sync backends, ETLs, wherever.

Most examples necessarily demonstrate how to interop with synchronous code, because all python programs begin in a blocking main thread. The event loop must be explicitly evoked and given control of the main thread. The interop examples become more useful in later parts.

Sync/Async interop is particularly necessary in a language that introduced async features after the fact. Python supplies several ways for async code to easily await on blocking functions (from, for instance, db drivers that block on io) without halting the event loop. Some of these techniques are covered in later sections.

When to use these features?

Whenever the program's performance is constrained by the performance of the hosts I/O subsystem. In other words, I/O bound contexts. This is usually contrasted with CPU bound applications, where the applications performance is dependent on the speed of the CPU. If the CPU in question has multple cores, the event loop, having a single execution thread, cannot execute multiple coroutines in parallel.

However, Python's async features could still be useful in dispatching and coordinating work between coroutines and different process workers, which can take advantage of multi core CPU's. Python provides some useful abstractions (e.g. Executors) which are designed to interop between the event loop and python threads/processes.

Why not use python multithreading for an IO Bound application?

NOTE: Generally, the GIL ensures that only one thread can execute at once.

  1. There is effectively an event loop anyway, but the operating system decides which thread should execute and for how long.
  2. The operating system doesn't understand how the threads should cooperate. Therefore opens the possiblity of race conditions and unsafe operations on memory unless the program accounts for the unpredictable nature of threaded execution (via locks, queues, etc.). In otherwords, the program has to be written to cooperate after the fact, versus an environment which is inherently cooperative.
  3. Threads have overhead.

Versus async concurrency:

  1. The programmer controls context switching between execution units.
  2. The event loop executes in a single thread, and the coroutines must explicitly yield back control making race conditions impossible. Also, since only one coroutine can execute at once, memory access and mutation is incidentally atomic.
  3. One call stack, versus one per thread (in additon to other overhead from OS thread). Also, exceptions are easier to trace, because there is one stack.

Setup

make build
make server
# new tab or terminal window
make part_<01-16>

Index

Event Loop

  1. Obtaining or creating the event loop

  2. Scheduling and running a coroutine to completion on the event loop

Awaitables

  1. Coroutines

  2. Futures

  3. Tasks

  4. Awaitable Objects

  5. Async Context Managers

  6. Async Iterators

  7. Async Generators

Concurrency

  1. Via loop.create_task

  2. Via asyncio.gather (asyncio.sleep example)

  3. Via asyncio.gather (aiohttp example)

  4. Via asyncio.as_completed

Sync / Async Interop

  1. loop.run_in_executor (thread pool, io bound)

  2. loop.run_in_executor (process pool, cpu bound)

Extras

  1. async queue consumer / producer

  2. asyncio Stream API - mini redis

Owner
Tyler Lovely
Tyler Lovely
The purpose of this code base is to add a specified signal-to-noise ratio noise from MUSAN dataset to a pure speech signal and to generate far-field speech data using room impulse response data from BUT [email protected] Reverb Database.

Add_noise_and_rir_to_speech The purpose of this code base is to add a specified signal-to-noise ratio noise from MUSAN dataset to a pure speech signal

Yunqi Chen 7 Oct 30, 2022
Python Monopoly Simulator

Monopoly simulator Original creator: Games Computer Play YouTube: https://www.youtube.com/channel/UCTrp88f-QJ1SqKX8o5IDhWQ Config file (optional) conf

Games Computers Play 37 Jan 03, 2023
The most hackable keyboard in all the land

MiRage Modular Keyboard © 2021 Zack Freedman of Voidstar Lab Licensed Creative Commons 4.0 Attribution Noncommercial Share-Alike The MiRage is a 60% o

Zack Freedman 558 Dec 30, 2022
Covid-ml-predictors - COVID predictions using AI.

COVID Predictions This repo contains ML models to be trained on COVID-19 data from the UK, sourced off of Kaggle here. This uses many different ML mod

1 Jan 09, 2022
An Airdrop alternative for cross-platform users only for desktop with Python

PyDrop An Airdrop alternative for cross-platform users only for desktop with Python, -version 1.0 with less effort, just as a practice. ##############

Bernardo Olisan 6 Mar 25, 2022
【AI创造营】参赛作品

-AI-emmmm 【AI创造营】参赛作品 鬼畜小视频 AiStuido地址:https://aistudio.baidu.com/aistudio/projectdetail/1647685 BiliBili视频地址:https://www.bilibili.com/video/BV1Zv411b

107 Nov 09, 2022
Python Project For Beginner

Basic-Vitrual-AI-Assistant Python Project For Beginner Hey There, I had manipulated Selenium WebDriver to make this assistant. I hope, It will be help

Maruf Billah 13 Dec 12, 2022
Utility/Raiding selfbot made by Shell and Roover.

Utility/Raiding selfbot made by Shell and Roover. We are open to suggestions and ideas.

Shell 2 Dec 08, 2021
Block fingerprinting for the beacon chain, for client identification & client diversity metrics

blockprint This is a repository for discussion and development of tools for Ethereum block fingerprinting. The primary aim is to measure beacon chain

Sigma Prime 49 Dec 08, 2022
A common, beautiful interface to tabular data, no matter the format

rows No matter in which format your tabular data is: rows will import it, automatically detect types and give you high-level Python objects so you can

Álvaro Justen 834 Jan 03, 2023
Mini-calculadora escrita como exemplo para uma palestra relâmpago sobre `git bisect`

Calculadora Mini-calculadora criada para uma palestra relâmpado sobre git bisect. Tem até uma colinha! Exemplo de uso Modo interativo $ python -m calc

Eduardo Cuducos 3 Dec 14, 2021
Basic code and description for GoBigger challenge 2021.

GoBigger Challenge 2021 en / 中文 Challenge Description 2021.11.13 We are holding a competition —— Go-Bigger: Multi-Agent Decision Intelligence Challeng

OpenDILab 183 Dec 29, 2022
Geodesic Dome Math

dome Geodesic Dome Math Python dome tool dome.py calculates an icosahedron or 2v geodesic dome and creates 3d printable hubs as OpenSCAD sources. usag

Brian Olson 2 Feb 09, 2022
[arXiv 2020] Video Representation Learning with Visual Tempo Consistency

Video Representation Learning with Visual Tempo Consistency [Paper] [Project Page] News Full codebae is coming soon Pretained Models For now, we provi

DeciForce: Crossroads of Machine Perception and Autonomy 24 Nov 23, 2022
little proyect to organize myself, but maybe can help someone else

TaskXT 0.1 Little proyect to organize myself, but maybe can help someone else Idea The main idea is to ogranize you work and stuff to do, but with onl

Gabriel Carmona 4 Oct 03, 2021
CD for MachineLearnia

Codebase supporting my talk on CI/CD for MachineLearnia (Nov 12 2021) The dataset used is available here. The point of the talk is to demonstrate a si

0 Feb 23, 2022
A python library with various gambling and gaming classes

gamble is a simple library that implements a collection of some common gambling-related classes Features die, dice, d-notation cards, decks, hands pok

Jacobi Petrucciani 16 May 24, 2022
A normal phoneNumber tracker made with python.

A normal phoneNumber tracker made with python.

CLAYZANE 2 Dec 30, 2021
BestBuy Script Designed to purchase any item when it becomes available.

prerequisites: Selnium; undetected-chromedriver. This Script is designed to order an Item provided a link from BestBuy.com only.

Bransen Smith 0 Jan 12, 2022
Code and data for learning to search in local branching

Code and data for learning to search in local branching

Defeng Liu 7 Dec 06, 2022