Familiar asyncio ORM for python, built with relations in mind

Overview

Tortoise ORM

https://badges.gitter.im/tortoise/community.png https://img.shields.io/pypi/v/tortoise-orm.svg?style=flat https://readthedocs.org/projects/tortoise-orm/badge/?version=latest https://pepy.tech/badge/tortoise-orm/month https://api.codacy.com/project/badge/Grade/b5b77021ba284e4a9e0c033a4611b046

Introduction

Tortoise ORM is an easy-to-use asyncio ORM (Object Relational Mapper) inspired by Django.

Tortoise ORM was build with relations in mind and admiration for the excellent and popular Django ORM. It's engraved in it's design that you are working not with just tables, you work with relational data.

You can find docs at ReadTheDocs

Note

Tortoise ORM is young project and breaking changes are to be expected. We keep a Changelog and it will have possible breakage clearly documented.

Tortoise ORM is supported on CPython >= 3.7 for SQLite, MySQL and PostgreSQL.

Why was Tortoise ORM built?

Python has many existing and mature ORMs, unfortunately they are designed with an opposing paradigm of how I/O gets processed. asyncio is relatively new technology that has a very different concurrency model, and the largest change is regarding how I/O is handled.

However, Tortoise ORM is not first attempt of building asyncio ORM, there are many cases of developers attempting to map synchronous python ORMs to the async world, initial attempts did not have a clean API.

Hence we started Tortoise ORM.

Tortoise ORM is designed to be functional, yet familiar, to ease the migration of developers wishing to switch to asyncio.

It also performs well when compared to other Python ORMs, trading places with Pony ORM:

https://raw.githubusercontent.com/tortoise/tortoise-orm/develop/docs/ORM_Perf.png

How is an ORM useful?

When you build an application or service that uses a relational database, there is a point when you can't just get away with just using parameterized queries or even query builder, you just keep repeating yourself, writing slightly different code for each entity. Code has no idea about relations between data, so you end up concatenating your data almost manually. It is also easy to make a mistake in how you access your database, making it easy for SQL-injection attacks to occur. Your data rules are also distributed, increasing the complexity of managing your data, and even worse, is applied inconsistently.

An ORM (Object Relational Mapper) is designed to address these issues, by centralising your data model and data rules, ensuring that your data is managed safely (providing immunity to SQL-injection) and keeps track of relationships so you don't have to.

Getting Started

Installation

First you have to install tortoise like this:

pip install tortoise-orm

You can also install with your db driver (aiosqlite is builtin):

pip install tortoise-orm[asyncpg]

Or MySQL:

pip install tortoise-orm[aiomysql]

Quick Tutorial

Primary entity of tortoise is tortoise.models.Model. You can start writing models like this:

from tortoise.models import Model
from tortoise import fields

class Tournament(Model):
    id = fields.IntField(pk=True)
    name = fields.TextField()

    def __str__(self):
        return self.name


class Event(Model):
    id = fields.IntField(pk=True)
    name = fields.TextField()
    tournament = fields.ForeignKeyField('models.Tournament', related_name='events')
    participants = fields.ManyToManyField('models.Team', related_name='events', through='event_team')

    def __str__(self):
        return self.name


class Team(Model):
    id = fields.IntField(pk=True)
    name = fields.TextField()

    def __str__(self):
        return self.name

After you defined all your models, tortoise needs you to init them, in order to create backward relations between models and match your db client with appropriate models.

You can do it like this:

from tortoise import Tortoise

async def init():
    # Here we connect to a SQLite DB file.
    # also specify the app name of "models"
    # which contain models from "app.models"
    await Tortoise.init(
        db_url='sqlite://db.sqlite3',
        modules={'models': ['app.models']}
    )
    # Generate the schema
    await Tortoise.generate_schemas()

Here we create connection to SQLite database in the local directory called db.sqlite3, and then we discover & initialise models.

Tortoise ORM currently supports the following databases:

  • SQLite (requires aiosqlite)
  • PostgreSQL (requires asyncpg)
  • MySQL (requires aiomysql)

generate_schema generates the schema on an empty database. Tortoise generates schemas in safe mode by default which includes the IF NOT EXISTS clause, so you may include it in your main code.

After that you can start using your models:

# Create instance by save
tournament = Tournament(name='New Tournament')
await tournament.save()

# Or by .create()
await Event.create(name='Without participants', tournament=tournament)
event = await Event.create(name='Test', tournament=tournament)
participants = []
for i in range(2):
    team = await Team.create(name='Team {}'.format(i + 1))
    participants.append(team)

# M2M Relationship management is quite straightforward
# (also look for methods .remove(...) and .clear())
await event.participants.add(*participants)

# You can query related entity just with async for
async for team in event.participants:
    pass

# After making related query you can iterate with regular for,
# which can be extremely convenient for using with other packages,
# for example some kind of serializers with nested support
for team in event.participants:
    pass


# Or you can make preemptive call to fetch related objects
selected_events = await Event.filter(
    participants=participants[0].id
).prefetch_related('participants', 'tournament')

# Tortoise supports variable depth of prefetching related entities
# This will fetch all events for team and in those events tournaments will be prefetched
await Team.all().prefetch_related('events__tournament')

# You can filter and order by related models too
await Tournament.filter(
    events__name__in=['Test', 'Prod']
).order_by('-events__participants__name').distinct()

Migration

Tortoise ORM use Aerich as database migrations tool, see more detail at it's docs.

Contributing

Please have a look at the Contribution Guide

License

This project is licensed under the Apache License - see the LICENSE.txt file for details

Comments
  • A unified, robust and bug-free connection management interface for the ORM

    A unified, robust and bug-free connection management interface for the ORM

    Description

    This PR provides for a much more robust implementation of the connection management interface of the ORM which is primarily geared towards improving performance and usability.

    Motivation and Context

    I was really excited that a native asyncIO based ORM was in town and implemented a lot of API constructs similar to the Django ORM since I personally like the way queries are expressed in Django. When I was tinkering around,

    • The first thing I noticed was that connections to the DB were being established on boot-up which I wasn't very comfortable with.
    • The second thing I noticed, after tinkering around with the source code, was that though the docs said that Tortoise.get_connection returns a connection for a given DB alias, there was another construct named current_transaction_map and another method named get_connection in tortoise/transactions.py that was being used internally for storing and retrieving connections.

    This PR tries to address the above problems/inconsistencies as follows:

    • A unified connection management interface for accessing, modifying and deleting connections using asyncio native constructs.
    • Lazy connection creation i.e the underlying connection to the DB gets established only upon execution of a query.
    • Fixes a lot of bugs that could occur due to connections being stored in two different places ( Tortoise._connections and current_transaction_map). Also, the usage of ContextVars has been done in the most optimal way.

    How Has This Been Tested?

    • All test cases present currently pass for all environments. Once the design and solution are agreed upon, I could write tests for the newly added code.
    • Any code attempting to access a specific connection using an alias has now been refactored to use the new connections interface (including all existing tests)

    Once the solution has been accepted and reviewed by the core team, I can update the tests, docs and changlog accordingly.

    Checklist:

    • [x] My code follows the code style of this project.
    • [x] My change requires a change to the documentation.
    • [ ] I have updated the documentation accordingly.
    • [ ] I have added the changelog accordingly.
    • [x] I have read the CONTRIBUTING document.
    • [ ] I have added tests to cover my changes.
    • [x] All new and existing tests passed.
    opened by blazing-gig 52
  • Migrations

    Migrations

    (edited by @grigi ) Migrations is currently planned as a post-v1.0 feature.

    Some useful resources for getting this working right now:

    • https://github.com/tortoise/tortoise-orm/issues/8#issuecomment-534946871 (Native python, MySQL)
    • https://github.com/tortoise/tortoise-orm/issues/8#issuecomment-575982472 (Go, MySQL/PostgreSQL/SQLite)

    Forward migrations

    Best guess at this time for a complete solution, not a "quick" solution:

    • [ ] Make the describe_model() contain all the data one needs to generate DDL from.
    • [ ] Generate a sequence of high-level DDL instructions from describe_model()
    • [ ] Port generate_schema to use this high-level DDL instructions to generate a schema in the requested dialect. (This would require some decoupling from the driver instance, look at #72)
    • [ ] Build a persistance model (read/write) for high-level DDL insctuctions much like Django Migrations
    • [ ] And need a way to run "management" commands. So possibly a standard command-line utility.
    • [ ] Build a full model (e.g. same format as describe_model())from a series of high-level DDL instructions (as in read the migration persistance model)
    • [ ] Diff models, and generate a diff DDL instruction set.
    • [ ] If a Diff requires a default not pre-specified, we need to ask the user for something.
    • [ ] Have some way of determining the "version" of the models persisted in the DB.
    • [ ] Put it all together and make it work

    I'm not particularily happy about many migration systems storing state in the DB itself, as for some environments that won't work, but it is a very good place from a consistency PoV. We should have a backup for when we generate DDL that one would pass on to the DB team in cases where DDL changes have an established process (e.g. enterprise/consulting)

    Data migrations

    Proper data migration management makes this very useful:

    • [ ] Allow custom user scripts as migrations
    • [ ] Allow data-migrations to get a version of a model as it was at that time. So ability to build a model class from the intermediate describe_model() data
    • [ ] Handle DB edge cases, like Postgres being able to do DDL changes in a transaction OR data changes in a transaction, but not both.

    Backward migrations

    There is some cases where backwards migrations cannot be done as a forward migration clobbered/destroyed some data. In cases where we detect that we need mising data, and request a sane default. What do we do about data migrations? Is it safe to just emit a warning?

    discussion Future Release 
    opened by abondar 32
  • pymysql.err.InternalError: Packet sequence number wrong - got X expected 1

    pymysql.err.InternalError: Packet sequence number wrong - got X expected 1

    Describe the bug I use aiohttp+tortoise-orm+mysql for simple API, which can only insert a row to a single mysql database table called "TableName" and read the rows from it. If my API is idle for some time, say, 3 hours, and I do API request which should return me table rows, it shows "pymysql.err.InternalError: Packet sequence number wrong - got 0 expected 1" (not necessarily 0, it could be other number) error and status 500 instead of returning rows.

    It shows this error when run this piece of code: rows = await TableName.filter(some_field=some_field)

    To Reproduce

    1. Create models.py file:
    from tortoise import Model, fields
    
    
    class TableName(Model):
        id = fields.IntField(pk=True)
        some_field = fields.CharField(250)
    
    1. Create main.py file:
    from aiohttp import web
    from tortoise.contrib.aiohttp import register_tortoise
    
    from models import TableName
    
    
    async def list_rows(request):
        data = await request.json()
        try:
            some_field = data['some_field']
        except KeyError:
            return web.json_response({'message': f'Not all fields were specified'}, status=400)
    
        rows = await TableName.filter(some_field=some_field)
    
        rows_json = [row.__dict__ for row in rows]
        return web.json_response(rows_json)
    
    
    app = web.Application()
    app.add_routes([
        web.post("/rows", list_rows)
    ])
    
    register_tortoise(
        app, db_url="mysql://user:[email protected]:3306/any-database-name", modules={"models": ["models"]},
        generate_schemas=True
    )
    
    if __name__ == "__main__":
        web.run_app(app, port=6000)
    
    1. pack it into docker container with such Dockerfile:
    FROM python:3.9.5
    COPY requirements.txt . 
    RUN pip install -r requirements.txt
    COPY . .
    CMD ["python", "main.py"]
    

    docker build -t image-name:tag-name .

    1. Run API with command:

    docker run -d -p 6000:6000 image-name:tag-name

    1. After 3 hours of working, run curl query:

    curl -i -X POST --data '{"some_field": "blabla"}' http://localhost:6000/rows

    1. curl will return 500 error, inside container's logs you will see error:
    pymysql.err.InternalError: Packet sequence number wrong - got 35 expected 1
    

    Expected behavior curl query should return json with rows' values in it.

    Additional context Full trace:

    ERROR:aiohttp.server:Error handling request
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/tortoise/backends/mysql/client.py", line 44, in translate_exceptions_
        return await func(self, *args)
      File "/usr/local/lib/python3.9/site-packages/tortoise/backends/mysql/client.py", line 199, in execute_query
        await cursor.execute(query, values)
      File "/usr/local/lib/python3.9/site-packages/aiomysql/cursors.py", line 239, in execute
        await self._query(query)
      File "/usr/local/lib/python3.9/site-packages/aiomysql/cursors.py", line 457, in _query
        await conn.query(q)
      File "/usr/local/lib/python3.9/site-packages/aiomysql/connection.py", line 428, in query
        await self._read_query_result(unbuffered=unbuffered)
      File "/usr/local/lib/python3.9/site-packages/aiomysql/connection.py", line 622, in _read_query_result
        await result.read()
      File "/usr/local/lib/python3.9/site-packages/aiomysql/connection.py", line 1105, in read
        first_packet = await self.connection._read_packet()
      File "/usr/local/lib/python3.9/site-packages/aiomysql/connection.py", line 574, in _read_packet
        raise InternalError(
    pymysql.err.InternalError: Packet sequence number wrong - got 0 expected 1
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/aiohttp/web_protocol.py", line 422, in _handle_request
        resp = await self._request_handler(request)
      File "/usr/local/lib/python3.9/site-packages/aiohttp/web_app.py", line 499, in _handle
        resp = await handler(request)
      File "//main.py", line 37, in list_rowa
        rows = await TableName.filter(some_field=some_field)
      File "/usr/local/lib/python3.9/site-packages/tortoise/queryset.py", line 879, in _execute
        instance_list = await self._db.executor_class(
      File "/usr/local/lib/python3.9/site-packages/tortoise/backends/base/executor.py", line 124, in execute_select
        _, raw_results = await self.db.execute_query(query.get_sql())
      File "/usr/local/lib/python3.9/site-packages/tortoise/backends/mysql/client.py", line 52, in translate_exceptions_
        raise OperationalError(exc)
    tortoise.exceptions.OperationalError: Packet sequence number wrong - got 0 expected 1
    
    opened by osintegrator 27
  • Builtin TestCase from python 3.8

    Builtin TestCase from python 3.8

    First and foremost, congrats, I really like this library and looking forward to contribute and follow the path of its evolution. I embraced this library mostly because it is async and it resembles django so it was very easy to get started with.

    Is your feature request related to a problem? Please describe. I am using tortoise with python 3.8, as of python 3.8 asyncio unittesting is possible without asynctest. https://docs.python.org/3/library/unittest.html#unittest.IsolatedAsyncioTestCase

    There are also AsyncMock and patch supports async methods.

    Describe the solution you'd like I would like not to include asynctest when using python3.8 and above.

    Describe alternatives you've considered I have considered no other alternatives, but I am open to consider other alternatives.

    Additional context I have been using a work around by copying some code from IsolatedTestCase

        async def asyncSetUp(self) -> None:
            config = generate_config(db_url='sqlite://:memory:', app_modules={'models': ['lib.models']})
            await Tortoise.init(config, _create_db=True)
            await Tortoise.generate_schemas(safe=False)
            self._connections = Tortoise._connections.copy()
    
        async def asyncTearDown(self) -> None:
            Tortoise._connections = self._connections.copy()
            await Tortoise._drop_databases()
    
            Tortoise.apps = {}
            Tortoise._connections = {}
            Tortoise._inited = False
    

    I am open to open a PR with the fixes.

    Another question, why the testing modules are in contrib? Testing is vital to every piece of software, is testing not fully supported or you are planning to change the api?

    enhancement 
    opened by WisdomPill 26
  • Add contains and contained_by filter to JSONField

    Add contains and contained_by filter to JSONField

    Description

    JSON is a native type in postgresql (since 9.2) and you can apply the filter on json object. The tortoise-orm not supported some filters such as contains , contained_by, filter by key and index in JSONField.

    Motivation and Context

    For example when we have an array of objects in json , we need to filter jsonfield if contains some objects or vice versa. Also to search inside the object by keys and index with equal , is_null, not_is_null, not options.

    How Has This Been Tested?

    I have added some methods in TestJSONFields class for testing.

    Checklist:

    • [x] My code follows the code style of this project.
    • [x] My change requires a change to the documentation.
    • [x] I have updated the documentation accordingly.
    • [x] I have read the CONTRIBUTING document.
    • [x] I have added tests to cover my changes.
    • [x] All new and existing tests passed.
    opened by ahmadgh74 22
  • fix bug with select_related not yielding null #up

    fix bug with select_related not yielding null #up

    Fixed (a bug ???) when select_related yielded unfilled instances of related objects instead of just nulls.

    Description

    Ran into this unexpected behaviour with something like this:

    class B(Model):
          id = fields.UUIDField(pk=True)
    class A(Model):
        the_b = fields.OneToOneField(
            "models.B", related_name="the_a", on_delete=fields.SET_NULL, null=True
        )
    

    then i got it like:

    a1 = await A.create(the_b=await B.create())
    a2 = await A.create(the_b=None)
    a = A.all().select_related('the_b').get(id=a2.id)
    # and for some reason the next checks failed
    assert a.the_b is None
    assert a.the_b == a2.the_b
    

    Motivation and Context

    Looks like this packages follows the ideas of django-orm and duplicates its best features in most cases, so i thought, like at least select_related behaviour should be same as it in django-orm. linked to this issue https://github.com/tortoise/tortoise-orm/issues/825.

    How Has This Been Tested?

    tested this with code like the one above. + as import dep in my pet proj, just assured that it don't gives me issues anymore.

    Checklist:

    • [x] My code follows the code style of this project.
    • [ ] My change requires a change to the documentation.
    • [ ] I have updated the documentation accordingly.
    • [x] I have read the CONTRIBUTING document.
    • [x] I have added tests to cover my changes.
    • [x] All new and existing tests passed.
    opened by urm8 19
  • Duplicate model instance

    Duplicate model instance

    Is your feature request related to a problem? Please describe. Usually in django to duplicate a model and to not copy every field there is a work around that is pretty straightforward and easy to do. This can be done via setting the pk to None. But with tortoise is not that easy, setting also the attribute ._saved_in_db is necessary.

    Describe the solution you'd like Not to access a private variables for this trick to work.

    Describe alternatives you've considered Maybe a copy method as well, that would be much more intuitive.

    Additional context Example of the code needed as of now

    book = Book(title='1984', author='George Orwell')
    
    print(book.pk)
    
    await book.save()
    
    print(book.pk)
    
    book.pk = None
    book._saved_in_db = False
    book.title = 'Animal farm'
    await book.save()
    
    print(book.pk)
    
    print(await Book.all().count())
    

    Example of the code that I would like to use instead

    book = Book(title='1984', author='George Orwell')
    
    print(book.pk)
    
    await book.save()
    
    print(book.pk)
    
    book.pk = None
    book.title = 'Animal farm'
    await book.save()
    
    print(book.pk)
    
    print(await Book.all().count())
    
    enhancement Next Release 
    opened by WisdomPill 18
  • Ability to SUM multiple columns

    Ability to SUM multiple columns

    Is your feature request related to a problem? Please describe. I'd like to be able to use the Sum function over multiple columns.

    Example for a model

    class Stats(Model):
        one_column = fields.IntField(default=0)
        two_column = fields.IntField(default=0)
    

    I'd like to be able to do run a query like select sum(one_column+two_column) from stats

    Describe the solution you'd like Stats.annotate(total_sum=Sum('one_column+two_column')) (or something similar)

    Additional context

    Pypika can already resolve this as seen in this test:

    https://github.com/kayak/pypika/blob/16339f85ed871d937609e9dbe26487b382ca5211/pypika/tests/test_formats.py

    fn.Sum(self.table_abc.fizz + self.table_abc.buzz)

    enhancement 
    opened by bbedward 18
  • Q-objects

    Q-objects

    after working withe the Q objects i have some questions, remarks and things that don't work as intended/expected

    • a blank Q object in AND mode works with just 1 argument to filter on, however OR mode requires a minimum of 2, quite inconvienient when your Q object is assembled based on client data
    • you can't combine keywords and nested Q objects into the same Q object but you got to wrap them into seperate Q objects to then combine into one master Q object to handle both, throws an explicit exception so seems to be intended behaviour but makes things more complex
    • are there any performance penalties to using nested Q objects several layers deep (other then ofc the overhead of python constructing objects)
    bug enhancement Next Release Waiting for feedback 
    opened by AEnterprise 18
  • New pool (WIP)

    New pool (WIP)

    Now that code base is simpler, and test runner should be more sane, attempt to add connection pooling for the third time.

    The plan is to change to connection pooling ONLY, as we currently implement persistent connections, but only one persistent connection. A connection pool should:

    • add robustness (if connection dies, then reconnect)
    • Allow multiple DB clients to operate at the same time (up to maxsize)
    • Allow more conflicts, so we need to handle rollback/retries explicitly.

    Things done:

    • [x] Change to a connection pooling system for MySQL.
    • [ ] Add tests for concurrency
    • [ ] Add tests for robustness (hackery allowed)
    • [ ] Add tests for handling conflicts.

    Concerns:

    • Can SQLite be concurrent at all? If difficult, should we limit it?
    • We need to add concurrency to the benchmarks, to manage performance
    opened by grigi 18
  • Tortoise orm not creating columns in postgres database in all tables

    Tortoise orm not creating columns in postgres database in all tables

    Hi i am having a serious issue i have read all docs and searched for the solution but my code is correct, acctually what is happening i have mades models and when i am running my python file its creating those tables in database but its not creating some columns in most of the tables here is the code of my one of the table:

    class GuildData(models.Model):
        class Meta:
            table = "server_configs"
        guild_id = fields.BigIntField(pk=True)
        is_bot_setuped = fields.BooleanField(default=False)
        scrims_manager = fields.BooleanField(default=False)
        autorole_toggle = fields.BooleanField(default=False)
        autorole_bot_toggle = fields.BooleanField(default=False)
        autorole_human_toggle = fields.BooleanField(default=False)
        autorole_human = fields.BigIntField(null = True)
        autorole_bot = fields.BigIntField(null = True)
        automeme_toogle = fields.BooleanField(default=False) 
        automeme_channel_id = fields.BigIntField(null = True)
        is_guild_premium = fields.BooleanField(default = False)
    

    and in this some of the columns are not getting created like i am naming the columns which have got created : guild_id,scrims_manager, autorole_bot, autorole_human, these columns are getting created but other columns dosen't get created here is the output logged bye toetoise logging:

    CREATE TABLE IF NOT EXISTS "server_configs" (
        "guild_id" BIGSERIAL NOT NULL PRIMARY KEY,
        "scrims_manager" BOOL NOT NULL  DEFAULT False,
        "autorole_human" BIGINT,
        "autorole_bot" BIGINT
    );
    

    you can see that most of the tables are missing same thing is happening in other tables too here is image of table in pg admin https://cdn.discordapp.com/attachments/754690035319701545/856917612395102218/Screenshot_from_2021-06-22_20-54-04.png.

    Kindly solve the issue fast because i am getting late with my bots update because of this.

    Thanks

    opened by TierGamerpy 17
  • fix: init multiple times will overwrite the previous db_config

    fix: init multiple times will overwrite the previous db_config

    Description

    When use Tortoise.init multiple times, last init will overwrite previous db_config.This will lead to tortoise.exceptions.OperationalError: no such table: xxx

    Motivation and Context

    In a plugin scenario, different plugins may do their own Tortoise.init.

    How Has This Been Tested?

    I tried Tortoise.init with different db_config several times in my project, and it worked fine, connecting to multiple databases.

    Checklist:

    • [x] My code follows the code style of this project.
    • [ ] My change requires a change to the documentation.
    • [ ] I have updated the documentation accordingly.
    • [ ] I have added the changelog accordingly.
    • [x] I have read the CONTRIBUTING document.
    • [ ] I have added tests to cover my changes.
    • [ ] All new and existing tests passed.
    opened by CMHopeSunshine 0
  • no such function: floor

    no such function: floor

    Describe the bug zh: 我使用的数据库是 sqlite 执行原始 sql 语句报错 en: The database I am using is sqlite executing the original sql statement with an error This quote comes from the translation

    tortoise.exceptions.OperationalError: no such function: floor
    

    To Reproduce

        sql = """SELECT * 
                FROM 
                    ImgPath 
                WHERE
                    id >= ( SELECT floor( RANDOM() * ( SELECT MAX( id ) FROM ImgPath ) ) ) 
                ORDER BY
                    id 
    	        LIMIT 10;"""
        res = await ImgPath.raw(sql)
    

    Expected behavior

    zh: 在 Navicat 执行是正常的,在代码中出现了错误 en: Execution is fine in Navicat, there is an error in the code This quote comes from the translation

    Additional context

    zh: 还是看下图片,这个问题好像是 sqlite 和 python 的问题。这里有什么解决方法吗? en: Looking at the picture again, this seems to be a problem with sqlite and python.Is there a solution here? This quote comes from the translation

    3eaf5981bbbe05b28af3ba2eea74e02

    opened by 522247020 1
  • Defer function call till after transaction

    Defer function call till after transaction

    I've been getting my hands dirty with tortoise recently, and I have been loving it. Porting from Django, I noticed (not sure if it isn't implemented already but with a different name) that I could not find something similar to transaction.on_commit in Django, something I could use to run comment only if the current transaction block is committed. I could easily go round this by moving the function call down, but in cases involving multiple models in different modules, that might up my imports and make my code more tightly coupled. The solution must not completely mirror Django, as tortoise is not Django, but could be something similar.

    opened by akpan-daniel 0
  • Pydantic typing fix

    Pydantic typing fix

    Description

    Changed config type for pydantic model creator.

    Motivation and Context

    I faced MRO issue when i tried to create type-correct config class for pydantic_model_creator. Example

    from tortoise import fields
    from tortoise.models import Model
    from pydantic import BaseConfig
    
    
    def to_upper(s: str) -> str:
        return s.upper()
    
    
    class ConfigClass(BaseConfig):
        alias_generator = to_upper
        allow_population_by_field_name = True
    
    
    class ModelClass(Model):
        name: str = fields.TextField()
        age: int = fields.IntField()
    
    
    pydantic_model = pydantic_model_creator(
        ModelClass,
        name="get_model",
        config_class=ConfigClass,
    )
    

    If you try to run this code you will face MRO issue

    Cannot create a consistent method resolution
    order (MRO) for bases BaseConfig, Config
    

    If ConfigClass does not inherits BaseConfig everything works fine, but typing is incorrect. So, this MR main aim is to fix some typing errors.

    How Has This Been Tested?

    1. Changed some tests to make config classes inherit BaseConfig
    2. To check code style and correct work used make check and make test.

    Checklist:

    • [x] My code follows the code style of this project.
    • [ ] My change requires a change to the documentation.
    • [ ] I have updated the documentation accordingly.
    • [ ] I have added the changelog accordingly.
    • [x] I have read the CONTRIBUTING document.
    • [x] I have added tests to cover my changes.
    • [x] All new and existing tests passed.
    opened by ZakharovDenis 0
  • fix: non-generic dict being subscribed

    fix: non-generic dict being subscribed

    Description

    In my prior PR, I used a non-generic dict which made ci fail on python >3.9.

    Motivation and Context

    Fix a critical bug

    How Has This Been Tested?

    This does not need any testing

    Checklist:

    • [X] My code follows the code style of this project.
    • [ ] My change requires a change to the documentation.
    • [ ] I have updated the documentation accordingly.
    • [X] I have added the changelog accordingly.
    • [X] I have read the CONTRIBUTING document.
    • [X] I have added tests to cover my changes.
    • [X] All new and existing tests passed.
    opened by Ovsyanka83 0
  • TypeError: error: Incompatible types in assignment (expression has type

    TypeError: error: Incompatible types in assignment (expression has type "Clan", variable has type "Optional[ForeignKeyFieldInstance[Clan]]")

    Describe the bug Getting a type error when updating the ForeignKey field to a different clan on a Nullable Foreign Key Field instance.

    To Reproduce

    if await Clan.get_or_none(tag=clantag):
    	clan = await Clan.get(tag=clantag)
    	if await Player.get_or_none(name=playername):
    		if await Season.active_seasons.all().exists():
    			await ctx.send("Cannot move players between clans during an active season.")
    		else:
    			player = await Player.get(name=playername)
    			if player.is_enabled():
    				player.clan = clan
    				await player.save()
    				await ctx.send(
    					f"Welcome @{playername} to the [{clan.tag}] {clan.name} Clan roster!"
    				)
    			else:
    				player.clan = clan
    				await player.save()
    				await ctx.send(
    					f"Welcome @{playername} to the [{clan.tag}] {clan.name} Clan roster!"
    				)
    	else:
    		await Player.create(name=playername, clan=clan, enabled=True)
    		await ctx.send(
    			f"Welcome @{playername} to the [{clan.tag}] {clan.name} Clan roster!"
    		)
    else:
    	await ctx.send(f"Clan {clantag} does not exist.")
    

    Expected behavior This shouldn't error as the code works fine and foreign key field gets updated fine.

    Additional context Add any other context about the problem here.

    opened by adambirds 0
Releases(0.19.2)
  • 0.19.2(Jul 11, 2022)

    Added

    • Added schema attribute to Model's Meta to specify exact schema to use with the model.

    Fixed

    • Mixin does not work. (#1133)
    • using_db wrong position in model shortcut methods. (#1150)
    • Fixed connection to Oracle database by adding database info to DBQ in connection string.
    • Fixed ORA-01435 error while using Oracle database (#1155)
    • Fixed processing of ssl option in MySQL connection string.
    • Fixed type hinting for QuerySetSingle.
    Source code(tar.gz)
    Source code(zip)
  • 0.19.1(May 20, 2022)

    Added

    • Added Postgres/SQLite partial indexes support. (#1103)
    • Added Microsoft SQL Server/Oracle support, powered by asyncodbc, note that which is not fully tested.
    • Added optional parameter to pydantic_model_creator. (#770)
    • Added using_db parameter to Model shortcut methods. (#1109)

    Fixed

    • TimeField for MySQL will return datetime.timedelta object instead of datetime.time object.
    • Fix on conflict do nothing. (#1122)
    • Fix _custom_generated_pk attribute not set in Model._init_from_db method. (#633)
    Source code(tar.gz)
    Source code(zip)
  • 0.19.0(Mar 27, 2022)

    Added

    • Added psycopg backend support.
    • Added a new unified and robust connection management interface to access DB connections which includes support for lazy connection creation and much more. For more details, check out this PR.
    • Added TimeField. (#1054).
    • Added ArrayField for postgres.

    Fixed

    • Fix bulk_create doesn't work correctly with more than 1 update_fields. (#1046)
    • Fix bulk_update errors when setting null for a smallint column on postgres. (#1086)

    Deprecated

    • Existing connection management interface and related public APIs which are deprecated:
    • Tortoise.get_connection
    • Tortoise.close_connections

    Changed

    • Refactored tortoise.transactions.get_connection method to tortoise.transactions._get_connection. Note that this method has now been marked private to this module and is not part of the public API
    Source code(tar.gz)
    Source code(zip)
  • 0.18.1(Jan 10, 2022)

    Added

    • Add on conflict do update for bulk_create. (#1024)

    Fixed

    • Fix bulk_create error. (#1012)
    • Fix unittest invalid.
    • Fix bulk_update in postgres with some type. (#968) (#1022)
    Source code(tar.gz)
    Source code(zip)
  • 0.18.0(Dec 20, 2021)

    Added

    • Add Case-When support. (#943)
    • Add Rand/Random function in contrib. (#944)
    • Add ON CONFLICT support in INSERT statements. (#428)

    Fixed

    • Fix bulk_update error when pk is uuid. (#986)
    • Fix mutable default value. (#969)

    Changed

    • Move Function, Aggregate from functions.py to expressions.py. (#943)
    • Move Q from query_utils.py to expressions.py.
    • Replace python-rapidjson to orjson.

    Removed

    • Remove asynctest and use unittest.IsolatedAsyncioTestCase. (#416)
    • Remove py37 support in tests.
    • Remove green and nose2 test runner.
    Source code(tar.gz)
    Source code(zip)
  • 0.17.8(Oct 6, 2021)

    Added

    • Add Model.raw method to support the raw sql query.
    • Add QuerySet.bulk_update method. (#924)
    • Add QuerySet.in_bulk method.
    • Add MaxValueValidator and MinValueValidator (#927)

    Fixed

    • Fix QuerySet subclass being lost when _clone is run on the instance.
    • Fix bug in .values with source_field. (#844)
    • Fix contrib.blacksheep exception handlers, use builtin json response. (#914)
    • Fix Indexes defined in Meta class do not make use of exists parameter in their template (#928)

    Changed

    • Allow negative values with IntEnumField. (#889)
    • Make .values() and .values_list() awaited return more consistent. (#899)
    Source code(tar.gz)
    Source code(zip)
  • 0.17.7(Aug 31, 2021)

    • Fix select_related behaviour for forward relation. (#825)
    • Fix bug in nested QuerySet and Manager. (#864)
    • Add Concat function for MySQL/PostgreSQL. (#873)
    • Patch for use_index/force_index mutable problem when making query. (#888)
    • Lift annotation field's priority in make query. (#883)
    • Make use/force index available in select type Query. (#893)
    • Fix all logging to use Tortoise's logger instead of root logger. (#879)
    • Rename db_client logger to tortoise.db_client.
    • Add indexes to Model.describe.
    Source code(tar.gz)
    Source code(zip)
  • 0.17.6(Jul 26, 2021)

    • Add RawSQL expression.
    • Fix columns count with annotations in _make_query. (#776)
    • Make functions nested. (#828)
    • Add db_constraint in field describe.
    Source code(tar.gz)
    Source code(zip)
  • 0.17.5(Jul 7, 2021)

    • Set field_type of fk and o2o same to which relation field type. (#443)
    • Fix error sql for .sql() call more than once. (#796)
    • Fix incorrect splitting of the import route when using Router (#798)
    • Fix filter error after annotate with F. (#806)
    • Fix select_related for reverse relation. (#808)
    Source code(tar.gz)
    Source code(zip)
  • 0.17.4(Jun 3, 2021)

  • 0.17.3(May 22, 2021)

    • Fix duplicates when using custom through association class on M2M relations
    • Fix update_or_create and get_or_create. (#721)
    • Fix refresh_from_db without fields pass. (#734)
    • Make update query work with limit and order_by. (#748)
    • Add Subquery expression. (#756) (#9) (#337)
    • Use JSON in JSONField.
    Source code(tar.gz)
    Source code(zip)
  • 0.17.2(Apr 9, 2021)

    • Add more index types.
    • Add force_index, use_index to queryset.
    • Fix F in update error with update_fields.
    • Make delete query work with limit and order_by. (#697)
    • Filter backward FK fields with IS NULL and NOT IS NULL filters (#700)
    • Add select_for_update in update_or_create. (#702)
    • Add Model.select_for_update.
    • Add __search full text search to queryset.
    Source code(tar.gz)
    Source code(zip)
  • 0.17.1(Mar 27, 2021)

    • Fix type for modules.
    • Fix select_related when related model specified more than once. (#679)
    • Add __iter__ to model, now can just return model/models in fastapi response.
    • Fix in_transaction bug caused by router. (#677) (#678)
    Source code(tar.gz)
    Source code(zip)
  • 0.17.0(Mar 20, 2021)

  • 0.16.21(Feb 4, 2021)

    • Fixed validating JSON before decoding. (#623)
    • Add model method update_or_create.
    • Add batch_size parameter for bulk_create method.
    • Fix save with F expression and field with source_field.
    Source code(tar.gz)
    Source code(zip)
  • 0.16.20(Jan 23, 2021)

  • 0.16.19(Dec 23, 2020)

    • Replace set TZ environment variable to TIMEZONE to avoid affecting global timezone.
    • Allow passing module objects to models_paths param of Tortoise.init_models(). (#561)
    • Implement PydanticMeta.backward_relations. (#536)
    • Allow overriding PydanticMeta in PydanticModelCreator. (#536)
    • Fixed make_native typo to make_naive in timezone module
    Source code(tar.gz)
    Source code(zip)
  • 0.16.18(Nov 16, 2020)

    • Support custom function in update. (#537)
    • Add Model.refresh_from_db. (#549)
    • Add timezone support, be careful to upgrade to this version, see docs for details. (#335)
    • Remove aerich in case of cyclic dependency. (#558)
    Source code(tar.gz)
    Source code(zip)
  • 0.16.17(Oct 23, 2020)

    • Add on_delete in ManyToManyField. (#508)
    • Support F expression in annotate. (#475)
    • Fix QuerySet.select_related in case of join same table twice. (#525)
    • Integrate Aerich into the install. (#530)
    Source code(tar.gz)
    Source code(zip)
  • 0.16.16(Sep 24, 2020)

  • 0.16.15(Sep 16, 2020)

    • Make DateField accept valid date str.
    • Add QuerySet.select_for_update().
    • check default for not None on pydantic model creation
    • propagate default to pydantic model
    • Add QuerySet.select_related().
    • Add custom attribute name for Prefetch instruction.
    • Add db_constraint for RelationalField family.
    Source code(tar.gz)
    Source code(zip)
  • 0.16.14(Jul 25, 2020)

    • We now do CI runs on a Windows VM as well, to try and prevent Windows specific regressions.
    • Make F expression work with QuerySet.filter().
    • Include py.typed in source distribution.
    • Added datetime parsing from int for fields.DatetimeField.
    • get_or_create passes the using_db= on if provided.
    • Allow custom loop and connection_class parameters to be passed on to asyncpg.
    Source code(tar.gz)
    Source code(zip)
  • 0.16.13(Jun 2, 2020)

    • Default install of tortoise-orm now installs with no C-dependencies, if you want to use the C accelerators, please do a pip install tortoise-orm[accel] instead.
    • Added <instance>.clone() method that will create a cloned instance in memory. To persist it you still need to call .save()
    • .clone() will raise a ParamsError if tortoise can't generate a primary key. In that case do a .clone(pk=<newval>)
    • If manually setting the primary key value to None and the primary key can be automatically generated, this will create a new record. We however still recommend the .clone() method instead.
    • .save() can be forced to do a create by setting force_create=True
    • .save() can be forced to do an update by setting force_update=True
    • Setting update_fields for a .save() operation will strongly prefer to do an update if possible
    Source code(tar.gz)
    Source code(zip)
  • 0.16.12(May 22, 2020)

    • Make Field.default effect on db level when generate table
    • Add converters instead of importing from pymysql
    • Fix PostgreSQL BooleanField default value convertion
    • Fix JSONField typed in pydantic_model_creator
    • Add .sql() method on QuerySet
    Source code(tar.gz)
    Source code(zip)
  • 0.16.11(May 14, 2020)

    • fix: sqlite://:memory: in Windows thrown OSError: [WinError 123]
    • Support bulk_create() insertion of records with overridden primary key when the primary key is DB-generated
    • Add queryset.exists() and Model.exists().
    • Add model subscription lookup, Model[<pkval>] that will return the object or raise KeyError
    Source code(tar.gz)
    Source code(zip)
  • 0.16.10(Apr 30, 2020)

    • Fix bad import of basestring
    • Better handling of NULL characters in strings. Fixes SQLite, raises better error for PostgreSQL.
    • Support .group_by() with join now
    Source code(tar.gz)
    Source code(zip)
  • 0.16.9(Apr 26, 2020)

    • Support F expression in .save() now
    • IntEnumField accept valid int value and CharEnumField accept valid str value
    • Pydantic models get created with globally unique identifier
    • Leaf-detection to minimize duplicate Pydantic model creation
    • Pydantic models with a Primary Key that is also a raw field of a relation is now not hidden when exclude_raw_fields=True as it is a critically important field
    • Raise an informative error when a field is set as nullable and primary key at the same time
    • Foreign key id's are now described to have the positive-integer range of the field it is related to
    • Fixed prefetching over OneToOne relations
    • Fixed __contains for non-text fields (e.g. JSONB)
    Source code(tar.gz)
    Source code(zip)
  • 0.16.8(Apr 22, 2020)

    • Allow Q expression to function with _filter parameter on aggregations
    • Add manual .group_by() support
    • Fixed regression where GROUP BY class is missing for an aggregate with a specified order.
    Source code(tar.gz)
    Source code(zip)
  • 0.15.24(Apr 22, 2020)

  • 0.16.7(Apr 19, 2020)

    • Added preliminary support for Python 3.9
    • TruncationTestCase now properly quotes table names when it clears them out.
    • Add model signals support
    • Added app_label to test initializer(...) and TORTOISE_TEST_APP as test environment variable.
    Source code(tar.gz)
    Source code(zip)
Owner
Tortoise
Familiar asyncio ORM for python, built with relations in mind
Tortoise
A fast MySQL driver written in pure C/C++ for Python. Compatible with gevent through monkey patching.

:: Description :: A fast MySQL driver written in pure C/C++ for Python. Compatible with gevent through monkey patching :: Requirements :: Requires P

ESN Social Software 549 Nov 18, 2022
This is a repository for a task assigned to me by Bilateral solutions!

Processing-Files-using-MySQL This is a repository for a task assigned to me by Bilateral solutions! Task: Make Folders named Processing,queue and proc

Kandal Khandeka 1 Nov 07, 2022
High level Python client for Elasticsearch

Elasticsearch DSL Elasticsearch DSL is a high-level library whose aim is to help with writing and running queries against Elasticsearch. It is built o

elastic 3.6k Jan 03, 2023
Lazydata: Scalable data dependencies for Python projects

lazydata: scalable data dependencies lazydata is a minimalist library for including data dependencies into Python projects. Problem: Keeping all data

629 Nov 21, 2022
Asynchronous interface for peewee ORM powered by asyncio

peewee-async Asynchronous interface for peewee ORM powered by asyncio. Important notes Since version 0.6.0a only peewee 3.5+ is supported If you still

05Bit 666 Dec 30, 2022
MinIO Client SDK for Python

MinIO Python SDK for Amazon S3 Compatible Cloud Storage MinIO Python SDK is Simple Storage Service (aka S3) client to perform bucket and object operat

High Performance, Kubernetes Native Object Storage 582 Dec 28, 2022
AWS SDK for Python

Boto3 - The AWS SDK for Python Boto3 is the Amazon Web Services (AWS) Software Development Kit (SDK) for Python, which allows Python developers to wri

the boto project 7.8k Jan 04, 2023
SQL for Humans™

Records: SQL for Humans™ Records is a very simple, but powerful, library for making raw SQL queries to most relational databases. Just write SQL. No b

Kenneth Reitz 6.9k Jan 07, 2023
New generation PostgreSQL database adapter for the Python programming language

Psycopg 3 -- PostgreSQL database adapter for Python Psycopg 3 is a modern implementation of a PostgreSQL adapter for Python. Installation Quick versio

The Psycopg Team 880 Jan 08, 2023
Py2neo is a comprehensive toolkit for working with Neo4j from within Python applications or from the command line.

Py2neo v3 Py2neo is a client library and toolkit for working with Neo4j from within Python applications and from the command line. The core library ha

64 Oct 14, 2022
db.py is an easier way to interact with your databases

db.py What is it Databases Supported Features Quickstart - Installation - Demo How To Contributing TODO What is it? db.py is an easier way to interact

yhat 1.2k Jan 03, 2023
Kafka Connect JDBC Docker Image.

kafka-connect-jdbc This is a dockerized version of the Confluent JDBC database connector. Usage This image is running the connect-standalone command w

Marc Horlacher 1 Jan 05, 2022
This repository is for active development of the Azure SDK for Python.

Azure SDK for Python This repository is for active development of the Azure SDK for Python. For consumers of the SDK we recommend visiting our public

Microsoft Azure 3.4k Jan 02, 2023
Motor - the async Python driver for MongoDB and Tornado or asyncio

Motor Info: Motor is a full-featured, non-blocking MongoDB driver for Python Tornado and asyncio applications. Documentation: Available at motor.readt

mongodb 2.1k Dec 26, 2022
sync/async MongoDB ODM, yes.

μMongo: sync/async ODM μMongo is a Python MongoDB ODM. It inception comes from two needs: the lack of async ODM and the difficulty to do document (un)

Scille 428 Dec 29, 2022
Create a database, insert data and easily select it with Sqlite

sqliteBasics create a database, insert data and easily select it with Sqlite Watch on YouTube a step by step tutorial explaining this code: https://yo

Mariya 27 Dec 27, 2022
MongoDB data stream pipeline tools by YouGov (adopted from MongoDB)

mongo-connector The mongo-connector project originated as a MongoDB mongo-labs project and is now community-maintained under the custody of YouGov, Pl

YouGov 1.9k Jan 04, 2023
GINO Is Not ORM - a Python asyncio ORM on SQLAlchemy core.

GINO - GINO Is Not ORM - is a lightweight asynchronous ORM built on top of SQLAlchemy core for Python asyncio. GINO 1.0 supports only PostgreSQL with

GINO Community 2.5k Dec 29, 2022
Python PostgreSQL database performance insights. Locks, index usage, buffer cache hit ratios, vacuum stats and more.

Python PG Extras Python port of Heroku PG Extras with several additions and improvements. The goal of this project is to provide powerful insights int

Paweł Urbanek 35 Nov 01, 2022
Python client for InfluxDB

InfluxDB-Python InfluxDB-Python is a client for interacting with InfluxDB. Development of this library is maintained by: Github ID URL @aviau (https:/

InfluxData 1.6k Dec 24, 2022