Pre-commit hook for upgrading type hints

Overview

supported python versions code coverage pre-commit

PEP585 Upgrade

This is a pre-commit hook configured to automatically upgrade your type hints to the new native types implemented in PEP 585.

This will work for any Python version above 3.7, though if you're not using 3.9 you will need to run the hook with futures-imports=true.

A complete type list is shown below.

See the complete list
Used to be Will be upgraded to
typing.Tuple tuple
typing.List list
typing.Dict dict
typing.Set set
typing.FrozenSet frozenset
typing.Type type
typing.Deque collections.deque
typing.DefaultDict collections.defaultdict
typing.OrderedDict collections.OrderedDict
typing.Counter collections.Counter
typing.ChainMap collections.ChainMap
typing.Awaitable collections.abc.Awaitable
typing.Coroutine collections.abc.Coroutine
typing.AsyncIterable collections.abc.AsyncIterable
typing.AsyncIterator collections.abc.AsyncIterator
typing.AsyncGenerator collections.abc.AsyncGenerator
typing.Iterable collections.abc.Iterable
typing.Iterator collections.abc.Iterator
typing.Generator collections.abc.Generator
typing.Reversible collections.abc.Reversible
typing.Container collections.abc.Container
typing.Collection collections.abc.Collection
typing.Callable collections.abc.Callable
typing.AbstractSet collections.abc.Set
typing.MutableSet collections.abc.MutableSet
typing.Mapping collections.abc.Mapping
typing.MutableMapping collections.abc.MutableMapping
typing.Sequence collections.abc.Sequence
typing.MutableSequence collections.abc.MutableSequence
typing.ByteString collections.abc.ByteString
typing.MappingView collections.abc.MappingView
typing.KeysView collections.abc.KeysView
typing.ItemsView collections.abc.ItemsView
typing.ValuesView collections.abc.ValuesView
typing.ContextManager contextlib.AbstractContextManager
typing.AsyncContextManager contextlib.AbstractAsyncContextManager
typing.re.Pattern re.Pattern
typing.re.Match re.Match

I'm a visual learner

In a nutshell, this code

from typing import List, Tuple, Dict, Set, FrozenSet

def do_thing(x: List[Tuple[str, ...]], y: Dict[str, Set[str]]) -> FrozenSet:

becomes this

def do_thing(x: list[tuple[str, ...]], y: dict[str, set[str]]) -> frozenset:

or this, if you're running python < 3.9 or enable the futures option

from __future__ import annotations

def do_thing(x: list[tuple[str, ...]], y: dict[str, set[str]]) -> frozenset:

Features:

  • Performs in-line substitution for new types
  • Adds new imports for upgrade types which need them
  • Adds __futures__ imports if the futures flag is enabled
  • Removes no longer needed typing imports

Note: even though we remove and add imports reasonably well, I would recommend running this in tandem with hooks like isort to aggregate and sort your imports, and flake8 to discover any unused imports neither were able to remove. Otherwise you risk needing to do some manual cleanup from time to time (though it should be pretty rare).

Config

To use this with pre-commit, simply add this to your config file:

- repo: https://github.com/sondrelg/pep585-upgrade
  rev: ''  # Use the sha / tag you want to point at
  hooks:
  - id: upgrade-type-hints

and while futures imports are added automatically if you're running Python older than 3.9, you can also enable them explicitly, by adding a --futures arg.

This is required, e.g., when maintaining code that needs to support older Python versions.

- repo: https://github.com/sondrelg/pep585-upgrade
  rev: ''  # Use the sha / tag you want to point at
  hooks:
  - id: upgrade-type-hints
    args: [ '--futures=true' ]

For more information about available arguments, see the function definitions.

Running this once on my codebase

If you wish to run this once on your codebase, that's not easy to do without pre-commit, as it piggybacks on that process quite a bit.

However, installing pre-commit and configuring the hook to run will take you less than a minute. These are the steps:

  • Run pip install pre-commit
  • Run touch .pre-commit-config.yaml
  • Copy the configuration shown above into the file
  • Run pre-commit run --all-files

Running this once on a single file

To run the upgrade on a single file, you can clone the repo and run python -m upgrade_type_hints from the src folder, or something equivalent.

Known imperfections

  • We have a hard time removing common import typing imports, since we don't have a full inventory of all possible places typing could be used. Seeing something like this, you might think this is easy to handle

    import typing
    
    x: typing.List

    but extending this example to a thousand-line file, the way we've structured the logic, there is no way to know whether there is a valid typing.Optional somewhere in the file.

  • We might remove typing imports in a file where you needed them for more than just type annotations. An example of this is custom type declarations:

    from typing import List
    
    x: List  # this will be upgraded and the import will be removed
    y = List[str]  # this will be left without its required import

    The reason for this is that custom type declarations are not a part of the ast objects we look at.

Both points are resolved by running flake8.

Supporting the project

Please leave a ✭ if this project helped you 👏 and contributions are always welcome!

Comments
  • Don't upgrade generics - part 2

    Don't upgrade generics - part 2

    I observe the same issue as #22, using v1.0.1.

    Starting with the file

    from __future__ import annotations
    
    from typing import Any, Callable, Mapping
    
    MessageCallback = Callable[[Mapping[str, Any], Any], None]
    
    def func(x: Mapping[str, Any], y: Any) -> None:
        return None
    
    callback: MessageCallback
    callback = func
    

    This originally runs without errors, but not after running pep585-upgrade has run:

    $ python -V
    Python 3.8.6
    $ python test.py
    $ pre-commit run
    Upgrade type hints.......................................................Failed
    - hook id: upgrade-type-hints
    - exit code: 1
    - files were modified by this hook
    
    Fixing src/workflows/test.py
    
    isort....................................................................Failed
    - hook id: isort
    - files were modified by this hook
    
    $ git diff
    index 72539fc..56bc282 100644
    --- a/test.py
    +++ b/test.py
    @@ -1,6 +1,7 @@
     from __future__ import annotations
    
    -from typing import Any, Callable, Mapping
    +from collections.abc import Mapping
    +from typing import Any, Callable
    
     MessageCallback = Callable[[Mapping[str, Any], Any], None]
    
    $ python test.py
    Traceback (most recent call last):
      File "test.py", line 6, in <module>
        MessageCallback = Callable[[Mapping[str, Any], Any], None]
    TypeError: 'ABCMeta' object is not subscriptable
    
    opened by Anthchirp 11
  • Support for typing.cast call

    Support for typing.cast call

    The code-base I'm modernizing has the following code:

    project_acl_list = cast(Optional[List[str]], project.acl)
    

    Unfortunately, this is not transformed.

    I'm willing to fix that, but I don't know yet how...

    Could we discuss how to implement the transformation for this code?

    opened by rominf 9
  • end_lineno error while running python 3.7

    end_lineno error while running python 3.7

    Adding hook via:

      - repo: https://github.com/sondrelg/pep585-upgrade
        rev: ''
        hooks:
          - id: upgrade-type-hints
            args: [ '--futures=true' ]
    

    When running in a python 3.9 virtualenv everything works, but switching to python 3.7 causes this error:

    Upgrade type hints.......................................................Failed
    - hook id: upgrade-type-hints
    - exit code: 1
    
    Traceback (most recent call last):
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/bin/upgrade-type-hints-script", line 8, in <module>
        sys.exit(main())
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/main.py", line 33, in main
        annotation_list, imports, futures_import_found = find_annotations_and_imports_in_file(filename)
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/checker.py", line 128, in find_annotations_and_imports_in_file
        imports, futures_import_found = map_imports(tree)
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/checker.py", line 113, in map_imports
        imports[item.lineno]['end_lineno'] = item.end_lineno
    AttributeError: 'Import' object has no attribute 'end_lineno'
    Traceback (most recent call last):
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/bin/upgrade-type-hints-script", line 8, in <module>
        sys.exit(main())
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/main.py", line 33, in main
        annotation_list, imports, futures_import_found = find_annotations_and_imports_in_file(filename)
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/checker.py", line 128, in find_annotations_and_imports_in_file
        imports, futures_import_found = map_imports(tree)
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/checker.py", line 113, in map_imports
        imports[item.lineno]['end_lineno'] = item.end_lineno
    AttributeError: 'Import' object has no attribute 'end_lineno'
    
    opened by ArcLightSlavik 8
  • pre-commit autoupdate warning

    pre-commit autoupdate warning

    Running pre-commit autoupdate leads to the following result: Updating https://github.com/sondrelg/pep585-upgrade ... updating v1.0.1 -> v1.

    Running pre-commit autoupdate again will result in the following warning:

    [WARNING] The 'rev' field of repo 'https://github.com/sondrelg/pep585-upgrade' appears to be a mutable reference (moving tag / branch).
    Mutable references are never updated after first install and are not supported. 
    See https://pre-commit.com/#using-the-latest-version-for-a-repository for more details. 
    Hint: `pre-commit autoupdate` often fixes this.
    

    It would be cool if pre-commit autoupdate would work on this hook as expected. I suspect that the problem is that all 3 tags point to the same commit.

    opened by alkatar21 7
  • Consider module docstring while inserting future import

    Consider module docstring while inserting future import

    Summary

    When the python file consists of module docstring, the formatter inserts the from __future__ import annotation on the top of the docstring.

    Previous Behavior

    Source

    #! /usr/env/bin python
    """don't touch the doc string
    
    multi line boss
    """
    from typing import List
    
    a: List[int] = [1, 2]
    
    
    def foo():
        """I'm a doc string"""
        pass
    

    Generated

    from __future__ import annotations
    
    
    #! /usr/env/bin python
    """don't touch the doc string
    
    multi line boss
    """
    
    a: list[int] = [1, 2]
    
    
    def foo():
        """I'm a doc string"""
    

    The PR adds a function to find the module docstring. It will insert future import after the module docstring.

    opened by kracekumar 4
  • Shebang is not respected

    Shebang is not respected

    Input:

    #!/usr/bin/env python3
    from typing import AbstractSet
    
    s: AbstractSet
    

    Output:

    from collections.abc import Set
    #!/usr/bin/env python3
    
    s: Set
    

    I didn't prepare the fix yet, but I think that the best solution would be to save the first occurrence of non-comment, non-docstring expressions while scanning AST and put the imports before that. What do you think?

    opened by rominf 4
  • Create/push a tag for the package

    Create/push a tag for the package

    Hi guys, Thanks a lot for this great package, it makes the code much more pythonic! I have one favor: could you create/push a tag for the package? Else I have this annoying warning:

    [WARNING] The 'rev' field of repo 'https://github.com/snok/pep585-upgrade' appears to be a mutable reference (moving tag / branch).  Mutable references are never updated after first install and are not supported.  See https://pre-commit.com/#using-the-latest-version-for-a-repository for more details.
    
    opened by MRigal 3
  • don't add another import line from same package

    don't add another import line from same package

    For an input file like this:

    from collections.abc import Callable
    
    from typing import Collection
    
    def a(a: Callable, b: Collection):
        pass
    

    when applying pep585 hook I get:

    from collections.abc import Collection
    from collections.abc import Callable
    
    def a(a: Callable, b: Collection):
        pass
    

    while I expect:

    from collections.abc import Callable, Collection
    
    def a(a: Callable, b: Collection):
        pass
    

    It could be argued this is the job of e.g. isort, however - pep585-upgrade already tries to be nice in terms of imports - e.g. in a larger import set - this new import line gets added nearby other import from same package and not at the end (which would be simplest I guess)

    opened by kretes 1
  • Don't Upgrade Generics

    Don't Upgrade Generics

    The typing package supports generics, whereas collections.abc does not. Hence, when this tool replaces typing types that are used as generics with their equivalent types in collections.abc, it breaks the build. I found it helpful to run this tool as a one-off and then manually undo these cases though, so thank you!

    opened by Kurt-von-Laven 1
  • Imports removal corrupts imports

    Imports removal corrupts imports

    Minimal working example:

    Input:

    from typing import Any, Type, TypeVar
    
    t: Type
    

    Output:

    from typing import AnyVar
    
    t: type
    

    I have a fix for this already. Please wait for a PR with a fix and tests.

    opened by rominf 1
  • Correct name of .pre-commit-config.yaml

    Correct name of .pre-commit-config.yaml

    I got an error from pre-commit which suggested it was looking for a .pre-commit-config.yaml file, when I changed the filename, as in this commit.

    Possibly OS specific, this was on MacOS Big Sur, Python 3.9.

    opened by james-d-f 1
Releases(v1.0.1)
Owner
snok
Open source collaboration organization made by @jonasks & @sondrelg
snok
Python samples for Google Cloud Platform products.

Google Cloud Platform Python Samples Python samples for Google Cloud Platform products. Setup Install pip and virtualenv if you do not already have th

Google Cloud Platform 6k Jan 03, 2023
A collection of tips for using MISP.

MISP Tip of the Week A collection of tips for using MISP. Published via BelgoMISP (todo) and this repository. Available in MD and JSON. Do you want to

Koen Van Impe 52 Jan 07, 2023
Cvdl-hw2 - Find Contour, Camera Calibration, Augmented Reality and Stereo Disparity Map

opevcvdl-hw2 This project uses openCV and Qt to achieve the requirements. Version Python 3.7 opencv-contrib-python 3.4.2.17 Matplotlib 3.1.1 pyqt5 5.1

Kenny Cheng 3 Aug 17, 2022
A short course on Julia and open-source software development

Advanced Scientific Computing: producing better code This course is taught as a 6-session "nanocourse" at Washington University in St. Louis. See the

Tim Holy 230 Jan 07, 2023
Sync SiYuanNote & Yuque.

SiyuanYuque Sync SiYuanNote & Yuque. Install Use pip to install. pip install SiyuanYuque Execute like this: python -m SiyuanYuque Remember to create a

Clouder 23 Nov 25, 2022
Module for remote in-memory Python package/module loading through HTTP/S

httpimport Python's missing feature! The feature has been suggested in Python Mailing List Remote, in-memory Python package/module importing through H

John Torakis 220 Dec 17, 2022
poro is a LCU interface to change some lol's options.

poro is a LCU interface to change some lol's options. with this program you can: change your profile icon change your profiel background image ch

João Dematte 2 Jan 05, 2022
Penelope Shell Handler

penelope Penelope is an advanced shell handler. Its main aim is to replace netcat as shell catcher during exploiting RCE vulnerabilities. It works on

293 Dec 30, 2022
run-js Goal: The Easiest Way to Run JavaScript in Python

run-js Goal: The Easiest Way to Run JavaScript in Python features Stateless Async JS Functions No Intermediary Files Functional Programming CommonJS a

Daniel J. Dufour 9 Aug 16, 2022
The presented desktop application was made to solve 1d schrodinger eqation

schrodinger_equation_1d_solver The presented desktop application was made to solve 1d schrodinger eqation. It implements Numerov's algorithm (step by

Artem Kashapov 2 Dec 29, 2021
Repositório para estudo do airflow

airflow-101 Repositório para estudo do airflow Docker criado baseado no tutorial Exemplo de API da pokeapi Para executar clone o repo execute as confi

Gabriel (Gabu) Bellon 1 Nov 23, 2021
A tool to quickly create codeforces contest directories with templates.

Codeforces Template Tool I created this tool to help me quickly set up codeforces contests/singular problems with templates. Tested for windows, shoul

1 Jun 02, 2022
Roman numeral conversion with python

Roman numeral conversion Discipline: Programming Languages Student: Paulo Henrique Diniz de Lima Alencar. Language: Python Description Responsible for

Paulo Alencar 1 Jul 11, 2022
Statistics Calculator module for all types of Stats calculations.

Statistics-Calculator This Calculator user the formulas and methods to find the statistical values listed. Statistics Calculator module for all types

2 May 29, 2022
Digitales Raumbuch

Helios Digitales Raumbuch Settings Moved to settings. Basic Commands Setting Up Your Users To create a normal user account, just go to Sign Up and fil

1 Nov 19, 2021
flake8 plugin which forbids match statements (PEP 634)

flake8-match flake8 plugin which forbids match statements (PEP 634)

Anthony Sottile 25 Nov 01, 2022
Runtime fault injection platform by Daniele Rizzieri (2021)

GDBitflip [v1.04] Runtime fault injection platform by Daniele Rizzieri (2021) This platform executes N times a binary and during each execution it inj

Daniele Rizzieri 1 Dec 07, 2021
In this project we will be using OpenCV to virtually drag a rectangle and drop it at a different location. It will be further used for Virtual Mouse.

Virtual Drag & Drog using OpenCV In this project we will be using OpenCV to virtually drag a rectangle and drop it at a different location. It will be

Hassan Shahzad 5 Sep 27, 2021
Model synchronization from dbt to Metabase.

dbt-metabase Model synchronization from dbt to Metabase. If dbt is your source of truth for database schemas and you use Metabase as your analytics to

Mike Gouline 270 Jan 08, 2023
Package to provide translation methods for pyramid, and means to reload translations without stopping the application

Package to provide translation methods for pyramid, and means to reload translations without stopping the application

Grzegorz Śliwiński 4 Nov 20, 2022