Asynchronous, fast, pythonic DynamoDB Client

Overview

AsyncIO DynamoDB

CircleCI Code style: black Documentation Status

Asynchronous pythonic DynamoDB client; 2x faster than aiobotocore/boto3/botocore.

Quick start

With httpx

Install this library

pip install "aiodynamo[httpx]" or, for poetry users poetry add aiodynamo -E httpx

Connect to DynamoDB

from aiodynamo.client import Client
from aiodynamo.credentials import Credentials
from aiodynamo.http.httpx import HTTPX
from httpx import AsyncClient

    async with AsyncClient() as h:
        client = Client(HTTPX(h), Credentials.auto(), "us-east-1")

With aiohttp

Install this library

pip install "aiodynamo[aiohttp]" or, for poetry users poetry add aiodynamo -E aiohttp

Connect to DynamoDB

from aiodynamo.client import Client
from aiodynamo.credentials import Credentials
from aiodynamo.http.aiohttp import AIOHTTP
from aiohttp import ClientSession

    async with ClientSession() as session:
        client = Client(AIOHTTP(session), Credentials.auto(), "us-east-1")

API use

        table = client.table("my-table")

        # Create table if it doesn't exist
        if not await table.exists():
            await table.create(
                Throughput(read=10, write=10),
                KeySchema(hash_key=KeySpec("key", KeyType.string)),
            )

        # Create or override an item
        await table.put_item({"key": "my-item", "value": 1})
        # Get an item
        item = await table.get_item({"key": "my-item"})
        print(item)
        # Update an item, if it exists.
        await table.update_item(
            {"key": "my-item"}, F("value").add(1), condition=F("key").exists()
        )

Why aiodynamo

  • boto3 and botocore are synchronous. aiodynamo is built for asynchronous apps.
  • aiodynamo is fast. Two times faster than aiobotocore, botocore or boto3 for operations such as query or scan.
  • aiobotocore is very low level. aiodynamo provides a pythonic API, using modern Python features. For example, paginated APIs are automatically depaginated using asynchronous iterators.
  • Legible source code. botocore and derived libraries generate their interface at runtime, so it cannot be inspected and isn't typed. aiodynamo is hand written code you can read, inspect and understand.
  • Pluggable HTTP client. If you're already using an asynchronous HTTP client in your project, you can use it with aiodynamo and don't need to add extra dependencies or run into dependency resolution issues.

Complete documentation is here

Comments
  • question: Any chance for transaction support?

    question: Any chance for transaction support?

    I'm wondering if there are any plans to support TransactWriteItems or just Transactions in general.

    I can probably work on this if/when I have more time 🤔

    opened by stupoid 16
  • add AWS_PROFILE env var to credentials chain

    add AWS_PROFILE env var to credentials chain

    Adds support for using AWS_PROFILE env var in the credentials chain.
    This behavior is more in line with the behavior expected by users coming from boto3 or aiobotocore. Added a unit test to cover this feature.

    Relates to: Issue #139

    opened by BTripp1986 8
  • Http wrapping issues

    Http wrapping issues

    When I was adding credentials handler to assume role with web identity which we need for our EKS setup I came across few issues why I am for now implementing it as custom code in our own project instead of providing it for the library.

    Http wrappers assume response is json, https://github.com/HENNGE/aiodynamo/blob/master/src/aiodynamo/http/base.py but as endpoint response is xml (and at least I can't find any mention of possibility to request it as json, but you know AWS docs 💩 ). This is blocker and thought that perhaps there needs to be some refactoring around http handling which takes too long for us.

    Some additional things I noticed

    1. in MetadataCredentials there is fetch_with_retry which couldn't use because it assumes GET but endpoint for assume role is POST. Was thinking should/could this retry option be in base http classes?

    2. Missing timeout in POST, would prefer to have timeout also for this credentials call. As mentioned in https://github.com/HENNGE/aiodynamo/issues/45

    3. http.post(..) requires body as bytes even when it can in aiohttp and httpx also be None. https://github.com/HENNGE/aiodynamo/blob/master/src/aiodynamo/http/base.py#L28-L30

    opened by jarikujansuu 6
  • Add support for pay_per_request billing mode during table creation

    Add support for pay_per_request billing mode during table creation

    Currently aiodynamo creates tables using the provisioned billing mode. DynamoDB also supports pay_per_request where no throughput config is provided. This PR is backwards compatible, but also adds the support to provide a billing_mode parameter to create_table. The throughput parameter has been changed to optional as the pay_per_request billing mode doesn't use it.

    PS. I'd be very grateful if you could also add the hacktoberfest-accepted label to this PR once it is accepted so that it can be counted as a hacktoberfest PR. Thanks :)

    hacktoberfest-accepted 
    opened by TheEdgeOfRage 6
  • Handle ClientConnectorError in connector

    Handle ClientConnectorError in connector

    Exception text: ClientConnectorError: Cannot connect to host dynamodb.<snip>.amazonaws.com:443 ssl:default [None]

    Exception type: aiohttp.client_exceptions.ClientConnectorError

    Stack:

    aiohttp/connector.py in _wrap_create_connection at line 943
    aiohttp/connector.py in _create_direct_connection at line 980
    aiohttp/connector.py in _create_direct_connection at line 1004
    aiohttp/connector.py in _create_connection at line 858
    aiohttp/connector.py in connect at line 523
    aiohttp/client.py in _request at line 480
    aiohttp/client.py in __aenter__ at line 1012
    aiodynamo/http/aiohttp.py in post at line 31
    aiodynamo/client.py in send_request at line 651
    aiodynamo/client.py in get_item at line 454
    aiodynamo/client.py in get_item at line 124
    
    opened by dimaqq 6
  • Raise asyncio.TimeoutError on httpx.TimeoutException

    Raise asyncio.TimeoutError on httpx.TimeoutException

    Now asyncio.TimeoutError is handled in the client logic. But httpx wrapper raises httpx.TimeoutException. This pull request convert the exception and make the client be able to handle timeouts in httpx.

    ref.) httpx timeout exceptions https://github.com/encode/httpx/blob/master/httpx/_exceptions.py#L7-L11

    ref.) Custom HTTP Client Adaptor document https://github.com/HENNGE/aiodynamo/blob/2e6c4716a3ac9fe5669bbdcaa73e6bbe0f73cfbb/docs/advanced.rst

    opened by gunyarakun 5
  • Count limits

    Count limits

    This was motivated by our real-life use case. We sometimes need to limit counting to preserve latency and Dynamo resources. I tried using query_single_page but it doesn't support count queries, so I just added limit to count.

    opened by Tinche 5
  • Use 3rd-party ddbcereal library for deserialize/serialize.

    Use 3rd-party ddbcereal library for deserialize/serialize.

    Figured I'd put this out here as an option. I've been working on a serializer/deserializer lib to supplement my use of aiobotocore, but it might make sense for this project too. It should be as fast or faster than this one.

    Overall, it reduces/simplifies the code, but the initialization of Client is a tad uglier due to ddbcereal needing a pre-constructed Deserializer and Client being frozen. It could be made pretty by unfreezing it or alternatively, making a seperate client-spawning function that passes the Deserializer to the Client constructor.

    opened by JustinTArthur 5
  • Don't send `StreamSpecification: null`

    Don't send `StreamSpecification: null`

    https://github.com/HENNGE/aiodynamo/blob/81b6f98111ce4bed41aeb8d07747695af836c76a/src/aiodynamo/client.py#L41-L49

    Sends StreamSpecification=None when none is supplied by the caller. Perhaps aiodynamo should not inject this key if the value is None.

    It seems that's OK for AWS SaaS dynamo, dynamodb-local and dynalite, but not ScyllaDB:

    https://github.com/scylladb/scylla/issues/5796

    opened by dimaqq 5
  • How to update map element using a pattern?

    How to update map element using a pattern?

    Described in https://dev.to/matthewvielkind/updating-values-in-dyanmodb-map-attributes-can

    UpdateExpression="SET players.#player_id.score = :score_val",  
    ExpressionAttributeNames={"#player_id": player_id},  
    

    I'm not sure what the API is in this case... Would I need to overload F?

    opened by dimaqq 4
  • Support consistent read

    Support consistent read

    Purpose

    Support consistent read in get_item, query, scan, count and batch get operations.

    Note: Tests do not actually check anything except that DynamoDb calls do not explode if using consistent_read option. Local DynamoDb is always strongly consistent. And might want to reduce copied code in them.

    opened by jarikujansuu 4
  • Empty-set safety

    Empty-set safety

    aiodynamo used to be empty-string-safe when DynamoDB didn't support empty string values. This mercifully got fixed in DynamoDB, but DynamoDB still doesn't accept empty sets (NS, SS, BS). There's an argument to be made that aiodynamo should handle this automatically, converting F("some_set").set(set()) to F("some_set").remove(), possibly with a warning.

    opened by ojii 3
  • Shall we try goodfirstissue.dev ?

    Shall we try goodfirstissue.dev ?

    links: https://goodfirstissue.dev/language/python https://github.com/deepsourcelabs/good-first-issue#adding-a-new-project

    I think this project may qualify, and would get some exposure and helping hands

    opened by dimaqq 0
  • Support AWS Web Identity Token File authentication

    Support AWS Web Identity Token File authentication

    aiodynamo ~~doesn't support AWS Web Identity Token File authentication~~ doesn't support AWS Web Identity Token File authentication out of the box.

    It looks like a typical case. For my team, we would like to use that authentication in our services in k8s by using IAM Roles for Service Accounts(IRSA)

    Related issue: https://github.com/HENNGE/aiodynamo/issues/128 Related PR: https://github.com/HENNGE/aiodynamo/pull/127

    There was a similar issue, but I decided to create another one because I think we can describe our goal and issue more directly.

    So, as discovered in a related PR, we should implement the AssumeRoleWithWebIdentity API for this authentication.

    opened by mrkovalchuk 2
  • Auth Error With No Default Credentials

    Auth Error With No Default Credentials

    I have access to several AWS accounts via SSO and no default profile for AWS. When I sign in to AWS via SSO the .aws/credentials file is updated with a fresh Access Key, Secret Key, and Security Token.

    I specify which credentials I will use with the AWS_PROFILE env var, which is checked as part of the credentials chain lookup. If using boto3 or the aws-cli this method works. With this method I can sign in via SSO to dev and prod accounts. Then if I AWS_PROFILE=dev aws s3 ls(or similar call in boto3) it will run the command against the dev account. Then if I AWS_PROFILE=prod aws s3 ls it will run against prod.

    This doesn't seem to work with aiodynamo. In order to get it to work I need to create a default profile and then copy/paste the credentials from the desired profile into the default profile. Aiodynamo should respect the AWS_PROFILE env var.

    opened by BTripp1986 1
  • Surprising behaviour in `Client.table_exists`

    Surprising behaviour in `Client.table_exists`

    We use Table.exists as part of a health check for the containers in our application, which in turn calls Client.table_exists. Today we had a short outage when all health checks began failing. It turned out the Dynamo table had entered UPDATING status.

    The implementation does the following:

    async def table_exists(self, name: TableName) -> bool:
            try:
                description = await self.describe_table(name)
            except TableNotFound:
                return False
    
            return description.status is TableStatus.active
    

    This feels like a bug to us: UPDATING is a valid state while a variety of operations are being applied to the table. So the code should do return description.status in [TableStatus.active, TableStatus.updating].

    Thoughts?

    opened by darylweir 1
Releases(21.9)
  • 21.9(Sep 1, 2021)

  • 21.8(Aug 24, 2021)

  • 21.7(Aug 24, 2021)

  • 21.6(Aug 24, 2021)

  • 21.5(Aug 24, 2021)

  • 20.4.2(Apr 21, 2020)

    Release Date: April 15th, 2020

    • Fix comparison conditions (equals, not_equals, gt, gte, lt, lte on aiodynamo.expressions.F and aiodynamo.expressions.Size via aiodynamo.expressions.F.size() to support referencing other fields (using aiodynamo.expressions.F)
    • Fix timeout handling in aiohttp based client.
    Source code(tar.gz)
    Source code(zip)
  • 20.4.1(Apr 21, 2020)

    Release Date: April 13th, 2020

    • Fixed put_item and delete_item with a condition which does not carry any values.
    • Wrap underlying HTTP client errors, such as connection issues, so networking issues during requests are retried.
    Source code(tar.gz)
    Source code(zip)
  • 20.4(Apr 21, 2020)

    Release Date: April 3rd, 2020

    • Fixed scan with a projection but no filter_expression.
    • Fixed logs leaking session tokens (request sending) and keys (metadata fetch).
    Source code(tar.gz)
    Source code(zip)
  • 20.3(Apr 21, 2020)

    Release Date: March 31st, 2020

    • Added TTL support
    • Added support for pluggable HTTP clients. Built in support for httpx and aiohttp.
    • Added custom client implementation.
    • Added custom credentials loaders, with support for custom credential loaders.
    • Fixed a typo in delete_item
    • Improved item deserialization performance
    • Improved overall client performance, especially for query, scan and count, which are now up to twice as fast.
    • Changed condition, key condition and filter expression APIs to not rely on boto3.
    • Moved aiodynamo.models.F to aiodynamo.expressions.F.
    • Removed boto3 dependency
    • Removed botocore dependency
    • Removed aiobotocore dependency
    Source code(tar.gz)
    Source code(zip)
  • 19.9(Apr 21, 2020)

  • 19.3(Mar 6, 2019)

  • 19.1(Jan 22, 2019)

Databank is an easy-to-use Python library for making raw SQL queries in a multi-threaded environment.

Databank Databank is an easy-to-use Python library for making raw SQL queries in a multi-threaded environment. No ORM, no frills. Thread-safe. Only ra

snapADDY GmbH 4 Apr 04, 2022
pandas-gbq is a package providing an interface to the Google BigQuery API from pandas

pandas-gbq pandas-gbq is a package providing an interface to the Google BigQuery API from pandas Installation Install latest release version via conda

Google APIs 348 Jan 03, 2023
google-cloud-bigtable Apache-2google-cloud-bigtable (🥈31 · ⭐ 3.5K) - Google Cloud Bigtable API client library. Apache-2

Python Client for Google Cloud Bigtable Google Cloud Bigtable is Google's NoSQL Big Data database service. It's the same database that powers many cor

Google APIs 39 Dec 03, 2022
Logica is a logic programming language that compiles to StandardSQL and runs on Google BigQuery.

Logica: language of Big Data Logica is an open source declarative logic programming language for data manipulation. Logica is a successor to Yedalog,

Evgeny Skvortsov 1.5k Dec 30, 2022
A Redis client library for Twisted Python

txRedis Asynchronous Redis client for Twisted Python. Install Install via pip. Usage examples can be found in the examples/ directory of this reposito

Dorian Raymer 127 Oct 23, 2022
Python client for Apache Kafka

Kafka Python client Python client for the Apache Kafka distributed stream processing system. kafka-python is designed to function much like the offici

Dana Powers 5.1k Jan 08, 2023
MariaDB connector using python and flask

MariaDB connector using python and flask This should work with flask and to be deployed on docker. Setting up stuff 1. Docker build and run docker bui

Bayangmbe Mounmo 1 Jan 11, 2022
MinIO Client SDK for Python

MinIO Python SDK for Amazon S3 Compatible Cloud Storage MinIO Python SDK is Simple Storage Service (aka S3) client to perform bucket and object operat

High Performance, Kubernetes Native Object Storage 582 Dec 28, 2022
GINO Is Not ORM - a Python asyncio ORM on SQLAlchemy core.

GINO - GINO Is Not ORM - is a lightweight asynchronous ORM built on top of SQLAlchemy core for Python asyncio. GINO 1.0 supports only PostgreSQL with

GINO Community 2.5k Dec 29, 2022
Micro ODM for MongoDB

Beanie - is an asynchronous ODM for MongoDB, based on Motor and Pydantic. It uses an abstraction over Pydantic models and Motor collections to work wi

Roman 993 Jan 03, 2023
TileDB-Py is a Python interface to the TileDB Storage Engine.

TileDB-Py TileDB-Py is a Python interface to the TileDB Storage Engine. Quick Links Installation Build Instructions TileDB Documentation Python API re

TileDB, Inc. 149 Nov 28, 2022
db.py is an easier way to interact with your databases

db.py What is it Databases Supported Features Quickstart - Installation - Demo How To Contributing TODO What is it? db.py is an easier way to interact

yhat 1.2k Jan 03, 2023
Pure-python PostgreSQL driver

pg-purepy pg-purepy is a pure-Python PostgreSQL wrapper based on the anyio library. A lot of this library was inspired by the pg8000 library. Credits

Lura Skye 11 May 23, 2022
A simple python package that perform SQL Server Source Control and Auto Deployment.

deploydb Deploy your database objects automatically when the git branch is updated. Production-ready! ⚙️ Easy-to-use 🔨 Customizable 🔧 Installation I

Mert Güvençli 10 Dec 07, 2022
Motor - the async Python driver for MongoDB and Tornado or asyncio

Motor Info: Motor is a full-featured, non-blocking MongoDB driver for Python Tornado and asyncio applications. Documentation: Available at motor.readt

mongodb 2.1k Dec 26, 2022
Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

AWS Data Wrangler Pandas on AWS Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretMana

Amazon Web Services - Labs 3.3k Dec 31, 2022
Python PostgreSQL database performance insights. Locks, index usage, buffer cache hit ratios, vacuum stats and more.

Python PG Extras Python port of Heroku PG Extras with several additions and improvements. The goal of this project is to provide powerful insights int

Paweł Urbanek 35 Nov 01, 2022
Python interface to Oracle Database conforming to the Python DB API 2.0 specification.

cx_Oracle version 8.2 (Development) cx_Oracle is a Python extension module that enables access to Oracle Database. It conforms to the Python database

Oracle 841 Dec 21, 2022
A Telegram Bot to manage Redis Database.

A Telegram Bot to manage Redis database. Direct deploy on heroku Manual Deployment python3, git is required Clone repo git clone https://github.com/bu

Amit Sharma 4 Oct 21, 2022
PostgreSQL database access simplified

Queries: PostgreSQL Simplified Queries is a BSD licensed opinionated wrapper of the psycopg2 library for interacting with PostgreSQL. The popular psyc

Gavin M. Roy 251 Oct 25, 2022