This is a survey of python's async concurrency features by example.

Overview

Survey of Python's Async Features

This is a survey of python's async concurrency features by example.

The purpose of this survey is to demonstrate that using Python's async features isn't difficult, and they are simple to use in all IO bound contexts. They can be used simply and to great effect in scripts, existing sync backends, ETLs, wherever.

Most examples necessarily demonstrate how to interop with synchronous code, because all python programs begin in a blocking main thread. The event loop must be explicitly evoked and given control of the main thread. The interop examples become more useful in later parts.

Sync/Async interop is particularly necessary in a language that introduced async features after the fact. Python supplies several ways for async code to easily await on blocking functions (from, for instance, db drivers that block on io) without halting the event loop. Some of these techniques are covered in later sections.

When to use these features?

Whenever the program's performance is constrained by the performance of the hosts I/O subsystem. In other words, I/O bound contexts. This is usually contrasted with CPU bound applications, where the applications performance is dependent on the speed of the CPU. If the CPU in question has multple cores, the event loop, having a single execution thread, cannot execute multiple coroutines in parallel.

However, Python's async features could still be useful in dispatching and coordinating work between coroutines and different process workers, which can take advantage of multi core CPU's. Python provides some useful abstractions (e.g. Executors) which are designed to interop between the event loop and python threads/processes.

Why not use python multithreading for an IO Bound application?

NOTE: Generally, the GIL ensures that only one thread can execute at once.

  1. There is effectively an event loop anyway, but the operating system decides which thread should execute and for how long.
  2. The operating system doesn't understand how the threads should cooperate. Therefore opens the possiblity of race conditions and unsafe operations on memory unless the program accounts for the unpredictable nature of threaded execution (via locks, queues, etc.). In otherwords, the program has to be written to cooperate after the fact, versus an environment which is inherently cooperative.
  3. Threads have overhead.

Versus async concurrency:

  1. The programmer controls context switching between execution units.
  2. The event loop executes in a single thread, and the coroutines must explicitly yield back control making race conditions impossible. Also, since only one coroutine can execute at once, memory access and mutation is incidentally atomic.
  3. One call stack, versus one per thread (in additon to other overhead from OS thread). Also, exceptions are easier to trace, because there is one stack.

Setup

make build
make server
# new tab or terminal window
make part_<01-16>

Index

Event Loop

  1. Obtaining or creating the event loop

  2. Scheduling and running a coroutine to completion on the event loop

Awaitables

  1. Coroutines

  2. Futures

  3. Tasks

  4. Awaitable Objects

  5. Async Context Managers

  6. Async Iterators

  7. Async Generators

Concurrency

  1. Via loop.create_task

  2. Via asyncio.gather (asyncio.sleep example)

  3. Via asyncio.gather (aiohttp example)

  4. Via asyncio.as_completed

Sync / Async Interop

  1. loop.run_in_executor (thread pool, io bound)

  2. loop.run_in_executor (process pool, cpu bound)

Extras

  1. async queue consumer / producer

  2. asyncio Stream API - mini redis

Owner
Tyler Lovely
Tyler Lovely
Fixed waypoint(pose) navigation for turtlebot simulation.

Turtlebot-NavigationStack-Fixed-Waypoints fixed waypoint(pose) navigation for turtlebot simulation. Task Details Task Permformed using Navigation Stac

Shanmukha Vishnu 1 Apr 08, 2022
A streaming animation of all the edits to a given Wikipedia page.

WikiFilms! What is it? A streaming animation of all the edits to a given Wikipedia page. How it works. It works by creating a "virtual camera," which

Tal Zaken 2 Jan 18, 2022
Import some key/value data to Prometheus custom-built Node Exporter in Python

About the app In one particilar project, i had to import some key/value data to Prometheus. So i have decided to create my custom-built Node Exporter

Hamid Hosseinzadeh 1 May 19, 2022
Developed a website to analyze and generate report of students based on the curriculum that represents student’s academic performance.

Developed a website to analyze and generate report of students based on the curriculum that represents student’s academic performance. We have developed the system such that, it will automatically pa

VIJETA CHAVHAN 3 Nov 08, 2022
That is a example of a Book app on Python, made with support of all JS libraries on React framework

React+Python Books App You can use this repository whenever you want Used for a video Create the database: python -m dbutils Start the web server: pyt

Koma Human 1 Apr 20, 2022
Python library for the Unmand APIs.

Unmand Python SDK This is a simple package to aid in consuming the Unmand APIs. For more help, see our docs. Getting Started Create virtual environmen

Unmand 4 Jul 22, 2022
This is where I learn machine learning

This is where I learn machine learning🤷‍ This means that this repo covers no specific topic of machine learning or a project - I work in here when I want to learn/try something

Wilhelm Berghammer 47 Nov 16, 2022
Simply create JIRA releases based on your github releases

Simply create JIRA releases based on your github releases

8 Jun 17, 2022
Python script to combine the statistical results of a TOPAS simulation that was split up into multiple batches.

topas-merge-simulations Python script to combine the statistical results of a TOPAS simulation that was split up into multiple batches At the top of t

Sebastian Schäfer 1 Aug 16, 2022
ERPNext Easy Letterhead

ERPNext Easy Letterhead Intro Quality letterheads are a problem for non-technical users. So we've built (really hacked together) a slightly easier sol

Bantoo 3 Jan 02, 2023
We want to check several batch of web URLs (1~100 K) and find the phishing website/URL among them.

We want to check several batch of web URLs (1~100 K) and find the phishing website/URL among them. This module is designed to do the URL/web attestation by using the API from NUS-Phishperida-Project.

3 Dec 28, 2022
1000+ ready code templates to kickstart your next AI experiment

AI Seed Projects Start with ready code for your next AI experiment. Choose from 1000+ code templates, across a wide variety of use cases. All examples

BlobCity, Inc 98 Jan 03, 2023
Transform your boring distro into a hacking powerhouse.

Pentizer Transform your boring distro into a hacking powerhouse. Pentizer is a personal project that imports Kali and Parrot repositories in any Debia

Michail Tsimpliarakis 2 Nov 05, 2021
Simple yet flexible natural sorting in Python.

natsort Simple yet flexible natural sorting in Python. Source Code: https://github.com/SethMMorton/natsort Downloads: https://pypi.org/project/natsort

Seth Morton 712 Dec 23, 2022
Contains the code of my learning of Python OOP.

OOP Python This repository contains the code of my learning of Python OOP. All the code: is following PEP 8 ✅ has proper concept illustrations and com

Samyak Jain 2 Jan 15, 2022
Weakly-Divisable - Takes an interger and seee if it is weakly divisible by seven

Weakly Divisble Project by Diana Arce-Hernandez, Ryan McAlpine, and Rommel Ravan

Diana Arce-Hernandez 1 Jan 12, 2022
36 key ergo split keyboard, designed around the Seeeduino Xiao platform

Slice36 Minimalist Split Keyboard 36 key ergo split keyboard, designed around the Seeeduino Xiao platform. Inspired by the Corne, Ferris, Ben Vallack'

54 Dec 21, 2022
Academic planner application designed for students and counselors.

Academic planner application designed for students and counselors.

Ali bagheri 2 Dec 31, 2021
Replite - An embeddable REPL powered by JupyterLite

replite An embeddable REPL, powered by JupyterLite. Usage To embed the code cons

Jeremy Tuloup 47 Nov 09, 2022
Covid 19 status. Flask application. CovidAPI. Heroku.

Covid 19 In this project we see total count of people who got this virus and total death. How does it works Written in Python. Web app, Flask. package

AmirHossein Mohammadi 12 Jan 16, 2022