Pre-commit hook for upgrading type hints

Overview

supported python versions code coverage pre-commit

PEP585 Upgrade

This is a pre-commit hook configured to automatically upgrade your type hints to the new native types implemented in PEP 585.

This will work for any Python version above 3.7, though if you're not using 3.9 you will need to run the hook with futures-imports=true.

A complete type list is shown below.

See the complete list
Used to be Will be upgraded to
typing.Tuple tuple
typing.List list
typing.Dict dict
typing.Set set
typing.FrozenSet frozenset
typing.Type type
typing.Deque collections.deque
typing.DefaultDict collections.defaultdict
typing.OrderedDict collections.OrderedDict
typing.Counter collections.Counter
typing.ChainMap collections.ChainMap
typing.Awaitable collections.abc.Awaitable
typing.Coroutine collections.abc.Coroutine
typing.AsyncIterable collections.abc.AsyncIterable
typing.AsyncIterator collections.abc.AsyncIterator
typing.AsyncGenerator collections.abc.AsyncGenerator
typing.Iterable collections.abc.Iterable
typing.Iterator collections.abc.Iterator
typing.Generator collections.abc.Generator
typing.Reversible collections.abc.Reversible
typing.Container collections.abc.Container
typing.Collection collections.abc.Collection
typing.Callable collections.abc.Callable
typing.AbstractSet collections.abc.Set
typing.MutableSet collections.abc.MutableSet
typing.Mapping collections.abc.Mapping
typing.MutableMapping collections.abc.MutableMapping
typing.Sequence collections.abc.Sequence
typing.MutableSequence collections.abc.MutableSequence
typing.ByteString collections.abc.ByteString
typing.MappingView collections.abc.MappingView
typing.KeysView collections.abc.KeysView
typing.ItemsView collections.abc.ItemsView
typing.ValuesView collections.abc.ValuesView
typing.ContextManager contextlib.AbstractContextManager
typing.AsyncContextManager contextlib.AbstractAsyncContextManager
typing.re.Pattern re.Pattern
typing.re.Match re.Match

I'm a visual learner

In a nutshell, this code

from typing import List, Tuple, Dict, Set, FrozenSet

def do_thing(x: List[Tuple[str, ...]], y: Dict[str, Set[str]]) -> FrozenSet:

becomes this

def do_thing(x: list[tuple[str, ...]], y: dict[str, set[str]]) -> frozenset:

or this, if you're running python < 3.9 or enable the futures option

from __future__ import annotations

def do_thing(x: list[tuple[str, ...]], y: dict[str, set[str]]) -> frozenset:

Features:

  • Performs in-line substitution for new types
  • Adds new imports for upgrade types which need them
  • Adds __futures__ imports if the futures flag is enabled
  • Removes no longer needed typing imports

Note: even though we remove and add imports reasonably well, I would recommend running this in tandem with hooks like isort to aggregate and sort your imports, and flake8 to discover any unused imports neither were able to remove. Otherwise you risk needing to do some manual cleanup from time to time (though it should be pretty rare).

Config

To use this with pre-commit, simply add this to your config file:

- repo: https://github.com/sondrelg/pep585-upgrade
  rev: ''  # Use the sha / tag you want to point at
  hooks:
  - id: upgrade-type-hints

and while futures imports are added automatically if you're running Python older than 3.9, you can also enable them explicitly, by adding a --futures arg.

This is required, e.g., when maintaining code that needs to support older Python versions.

- repo: https://github.com/sondrelg/pep585-upgrade
  rev: ''  # Use the sha / tag you want to point at
  hooks:
  - id: upgrade-type-hints
    args: [ '--futures=true' ]

For more information about available arguments, see the function definitions.

Running this once on my codebase

If you wish to run this once on your codebase, that's not easy to do without pre-commit, as it piggybacks on that process quite a bit.

However, installing pre-commit and configuring the hook to run will take you less than a minute. These are the steps:

  • Run pip install pre-commit
  • Run touch .pre-commit-config.yaml
  • Copy the configuration shown above into the file
  • Run pre-commit run --all-files

Running this once on a single file

To run the upgrade on a single file, you can clone the repo and run python -m upgrade_type_hints from the src folder, or something equivalent.

Known imperfections

  • We have a hard time removing common import typing imports, since we don't have a full inventory of all possible places typing could be used. Seeing something like this, you might think this is easy to handle

    import typing
    
    x: typing.List

    but extending this example to a thousand-line file, the way we've structured the logic, there is no way to know whether there is a valid typing.Optional somewhere in the file.

  • We might remove typing imports in a file where you needed them for more than just type annotations. An example of this is custom type declarations:

    from typing import List
    
    x: List  # this will be upgraded and the import will be removed
    y = List[str]  # this will be left without its required import

    The reason for this is that custom type declarations are not a part of the ast objects we look at.

Both points are resolved by running flake8.

Supporting the project

Please leave a ✭ if this project helped you 👏 and contributions are always welcome!

Comments
  • Don't upgrade generics - part 2

    Don't upgrade generics - part 2

    I observe the same issue as #22, using v1.0.1.

    Starting with the file

    from __future__ import annotations
    
    from typing import Any, Callable, Mapping
    
    MessageCallback = Callable[[Mapping[str, Any], Any], None]
    
    def func(x: Mapping[str, Any], y: Any) -> None:
        return None
    
    callback: MessageCallback
    callback = func
    

    This originally runs without errors, but not after running pep585-upgrade has run:

    $ python -V
    Python 3.8.6
    $ python test.py
    $ pre-commit run
    Upgrade type hints.......................................................Failed
    - hook id: upgrade-type-hints
    - exit code: 1
    - files were modified by this hook
    
    Fixing src/workflows/test.py
    
    isort....................................................................Failed
    - hook id: isort
    - files were modified by this hook
    
    $ git diff
    index 72539fc..56bc282 100644
    --- a/test.py
    +++ b/test.py
    @@ -1,6 +1,7 @@
     from __future__ import annotations
    
    -from typing import Any, Callable, Mapping
    +from collections.abc import Mapping
    +from typing import Any, Callable
    
     MessageCallback = Callable[[Mapping[str, Any], Any], None]
    
    $ python test.py
    Traceback (most recent call last):
      File "test.py", line 6, in <module>
        MessageCallback = Callable[[Mapping[str, Any], Any], None]
    TypeError: 'ABCMeta' object is not subscriptable
    
    opened by Anthchirp 11
  • Support for typing.cast call

    Support for typing.cast call

    The code-base I'm modernizing has the following code:

    project_acl_list = cast(Optional[List[str]], project.acl)
    

    Unfortunately, this is not transformed.

    I'm willing to fix that, but I don't know yet how...

    Could we discuss how to implement the transformation for this code?

    opened by rominf 9
  • end_lineno error while running python 3.7

    end_lineno error while running python 3.7

    Adding hook via:

      - repo: https://github.com/sondrelg/pep585-upgrade
        rev: ''
        hooks:
          - id: upgrade-type-hints
            args: [ '--futures=true' ]
    

    When running in a python 3.9 virtualenv everything works, but switching to python 3.7 causes this error:

    Upgrade type hints.......................................................Failed
    - hook id: upgrade-type-hints
    - exit code: 1
    
    Traceback (most recent call last):
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/bin/upgrade-type-hints-script", line 8, in <module>
        sys.exit(main())
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/main.py", line 33, in main
        annotation_list, imports, futures_import_found = find_annotations_and_imports_in_file(filename)
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/checker.py", line 128, in find_annotations_and_imports_in_file
        imports, futures_import_found = map_imports(tree)
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/checker.py", line 113, in map_imports
        imports[item.lineno]['end_lineno'] = item.end_lineno
    AttributeError: 'Import' object has no attribute 'end_lineno'
    Traceback (most recent call last):
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/bin/upgrade-type-hints-script", line 8, in <module>
        sys.exit(main())
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/main.py", line 33, in main
        annotation_list, imports, futures_import_found = find_annotations_and_imports_in_file(filename)
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/checker.py", line 128, in find_annotations_and_imports_in_file
        imports, futures_import_found = map_imports(tree)
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/checker.py", line 113, in map_imports
        imports[item.lineno]['end_lineno'] = item.end_lineno
    AttributeError: 'Import' object has no attribute 'end_lineno'
    
    opened by ArcLightSlavik 8
  • pre-commit autoupdate warning

    pre-commit autoupdate warning

    Running pre-commit autoupdate leads to the following result: Updating https://github.com/sondrelg/pep585-upgrade ... updating v1.0.1 -> v1.

    Running pre-commit autoupdate again will result in the following warning:

    [WARNING] The 'rev' field of repo 'https://github.com/sondrelg/pep585-upgrade' appears to be a mutable reference (moving tag / branch).
    Mutable references are never updated after first install and are not supported. 
    See https://pre-commit.com/#using-the-latest-version-for-a-repository for more details. 
    Hint: `pre-commit autoupdate` often fixes this.
    

    It would be cool if pre-commit autoupdate would work on this hook as expected. I suspect that the problem is that all 3 tags point to the same commit.

    opened by alkatar21 7
  • Consider module docstring while inserting future import

    Consider module docstring while inserting future import

    Summary

    When the python file consists of module docstring, the formatter inserts the from __future__ import annotation on the top of the docstring.

    Previous Behavior

    Source

    #! /usr/env/bin python
    """don't touch the doc string
    
    multi line boss
    """
    from typing import List
    
    a: List[int] = [1, 2]
    
    
    def foo():
        """I'm a doc string"""
        pass
    

    Generated

    from __future__ import annotations
    
    
    #! /usr/env/bin python
    """don't touch the doc string
    
    multi line boss
    """
    
    a: list[int] = [1, 2]
    
    
    def foo():
        """I'm a doc string"""
    

    The PR adds a function to find the module docstring. It will insert future import after the module docstring.

    opened by kracekumar 4
  • Shebang is not respected

    Shebang is not respected

    Input:

    #!/usr/bin/env python3
    from typing import AbstractSet
    
    s: AbstractSet
    

    Output:

    from collections.abc import Set
    #!/usr/bin/env python3
    
    s: Set
    

    I didn't prepare the fix yet, but I think that the best solution would be to save the first occurrence of non-comment, non-docstring expressions while scanning AST and put the imports before that. What do you think?

    opened by rominf 4
  • Create/push a tag for the package

    Create/push a tag for the package

    Hi guys, Thanks a lot for this great package, it makes the code much more pythonic! I have one favor: could you create/push a tag for the package? Else I have this annoying warning:

    [WARNING] The 'rev' field of repo 'https://github.com/snok/pep585-upgrade' appears to be a mutable reference (moving tag / branch).  Mutable references are never updated after first install and are not supported.  See https://pre-commit.com/#using-the-latest-version-for-a-repository for more details.
    
    opened by MRigal 3
  • don't add another import line from same package

    don't add another import line from same package

    For an input file like this:

    from collections.abc import Callable
    
    from typing import Collection
    
    def a(a: Callable, b: Collection):
        pass
    

    when applying pep585 hook I get:

    from collections.abc import Collection
    from collections.abc import Callable
    
    def a(a: Callable, b: Collection):
        pass
    

    while I expect:

    from collections.abc import Callable, Collection
    
    def a(a: Callable, b: Collection):
        pass
    

    It could be argued this is the job of e.g. isort, however - pep585-upgrade already tries to be nice in terms of imports - e.g. in a larger import set - this new import line gets added nearby other import from same package and not at the end (which would be simplest I guess)

    opened by kretes 1
  • Don't Upgrade Generics

    Don't Upgrade Generics

    The typing package supports generics, whereas collections.abc does not. Hence, when this tool replaces typing types that are used as generics with their equivalent types in collections.abc, it breaks the build. I found it helpful to run this tool as a one-off and then manually undo these cases though, so thank you!

    opened by Kurt-von-Laven 1
  • Imports removal corrupts imports

    Imports removal corrupts imports

    Minimal working example:

    Input:

    from typing import Any, Type, TypeVar
    
    t: Type
    

    Output:

    from typing import AnyVar
    
    t: type
    

    I have a fix for this already. Please wait for a PR with a fix and tests.

    opened by rominf 1
  • Correct name of .pre-commit-config.yaml

    Correct name of .pre-commit-config.yaml

    I got an error from pre-commit which suggested it was looking for a .pre-commit-config.yaml file, when I changed the filename, as in this commit.

    Possibly OS specific, this was on MacOS Big Sur, Python 3.9.

    opened by james-d-f 1
Releases(v1.0.1)
Owner
snok
Open source collaboration organization made by @jonasks & @sondrelg
snok
Curses frontend for Canto daemon

Canto Curses The curses (text) client for canto-daemon. Canto-daemon is required to work and is found at: http://github.com/themoken/canto-next Requir

Jack Miller 86 Dec 28, 2022
Script para generar automatización de registro de formularios IEEH

Formularios_IEEH Script para generar automatización de registro de formularios IEEH Corresponde a un conjunto de script en python que permiten la auto

vhevia11 1 Jan 06, 2022
Курс про техническое совершенство для нетехнарей

Technical Excellence 101 Курс про техническое совершенство для нетехнарей. Этот курс представлят из себя серию воркшопов, при помощи которых можно объ

Anton Bevzuk 11 Nov 13, 2022
This code can help you with auto update for-TV-advertisements in the store.

Auto-update-files-for-TV-advertisements-in-the-store This code can help you with auto update for-TV-advertisements in the store. It was write for Rasp

Max 2 Feb 20, 2022
A beautiful and useful prompt for your shell

A Powerline style prompt for your shell A beautiful and useful prompt generator for Bash, ZSH, Fish, and tcsh: Shows some important details about the

Buck Ryan 6k Jan 08, 2023
The newest contender in Server Gateway Interface.

nsgi The newest contender in Server Gateway Interface. Why use this webserver? This webserver is made with the newest version of asyncio, and sockets,

OpenRobot 1 Feb 12, 2022
Blender Addon for Snapping a UV to a specific part of a Tilemap

UVGridSnapper A simple Blender Addon for easier texturing. A menu in the UV editor allows a square UV to be snapped to an Atlas texture, or Tilemap. P

2 Jul 17, 2022
Script to check if your Bistromatic handle everything as it should.

Bistromatic Checker Script to check if your Bistromatic handle everything as it should. The bistromatic is the project marking the end of the CPool at

Mathias 1 Dec 27, 2021
Meower a social media platform written in Scratch 3.0 and Python

Meower Meower is a social media platform written in Scratch 3.0 and Python, ported to HTML for self-hosting. Try Beta 4.6 Changelog for 4.6 Start impl

Meower Media Co. 23 Dec 02, 2022
A Python script to parse Fortinet products serial numbers, and detect the associated model and version.

ParseFortinetSerialNumber A Python script to parse Fortinet products serial numbers, and detect the associated model and version. Example $ ./ParseFor

Podalirius 10 Oct 28, 2022
Two predictive attributes (Speed and Angle) and one attribute target (Power)

Two predictive attributes (Speed and Angle) and one attribute target (Power). A container crane has the function of transporting containers from one point to another point. The difficulty of this tas

Astitva Veer Garg 1 Jan 11, 2022
This project intends to take the user's CEP (brazilian adress code) and return the local in which the CEP is placed.

This project aims to simply return the CEP's (the brazilian resident adress code) User of the application. The project uses a request and passes on to

Daniel Soares Saldanha 4 Nov 17, 2021
Aerial Ace is a helper bot for poketwo which provide various functionalities on top of being a pokedex.

Aerial Ace is a helper bot for poketwo which provide various functionalities on top of being a pokedex.

Devanshu Mishra 1 Dec 01, 2021
Automated Content Feed Curator

Gathers posts from content feeds, filters, formats, delivers to you.

Alper S. Soylu 2 Jan 22, 2022
Just some mtk tool for exploitation, reading/writing flash and doing crazy stuff

Just some mtk tool for exploitation, reading/writing flash and doing crazy stuff. For linux, a patched kernel is needed (see Setup folder) (except for read/write flash). For windows, you need to inst

Bjoern Kerler 1.1k Dec 31, 2022
Script Repository for the ICGM-CNRS FRANCE

Here you will find my Python Work repesitory for the ICGM institute - Montpellier - France.

CABOS Matthieu 1 Apr 13, 2022
Files for QMC Workshop 2021

QMC Workshop 2021 This repository contains the presented slides and example files for the Quantum Monte Carlo (QMC) Workshop 5 October - 23 November,

QMCPACK 39 Nov 04, 2022
Bitflip Fault Simulation Platform by Daniele Rizzieri (2021)

BFSP [v1.05] Bitflip Fault Simulation Platform by Daniele Rizzieri (2021) The platform injects a random bitflip in each of N copies of a binary file.

Daniele Rizzieri 2 Nov 05, 2022
This repository containing cross-section cut and fill calculations using Python programming language.

cross-section This repository is containing cut and fill calculations for cross-section using Python programming language. This codes is made to calcu

3 Jun 15, 2022
Stori QA Automation Challenge

Stori-QA-Automation-Challenge This is the repository is created for the Stori QA Intern Automation Engineer Challenge! In this you can find the Requir

Daniel Castañeda 0 Feb 20, 2022