pytest plugin for testing mypy types, stubs, and plugins

Overview

mypy logo

pytest plugin for testing mypy types, stubs, and plugins

Tests Status Checked with mypy Gitter PyPI Conda Version

Installation

This package is available on PyPI

pip install pytest-mypy-plugins

and conda-forge

conda install -c conda-forge pytest-mypy-plugins

Usage

Running

Plugin, after installation, is automatically picked up by pytest therefore it is sufficient to just execute:

pytest

Paths

The PYTHONPATH and MYPYPATH environment variables, if set, are passed to mypy on invocation. This may be helpful if you are testing a local plugin and need to provide an import path to it.

Be aware that when mypy is run in a subprocess (the default) the test cases are run in temporary working directories where relative paths such as PYTHONPATH=./my_plugin do not reference the directory which you are running pytest from. If you encounter this, consider invoking pytest with --mypy-same-process or make your paths absolute, e.g. PYTHONPATH=$(pwd)/my_plugin pytest.

You can also specify PYTHONPATH, MYPYPATH, or any other environment variable in env: section of yml spec:

- case: mypy_path_from_env
  main: |
    from pair import Pair

    instance: Pair
    reveal_type(instance)  # N: Revealed type is 'pair.Pair'
  env:
    - MYPYPATH=./pytest_mypy_plugins/tests/fixtures

What is a test case?

In general each test case is just an element in an array written in a properly formatted YAML file. On top of that, each case must comply to following types:

Property Type Description
case str Name of the test case, complies to [a-zA-Z0-9] pattern
main str Portion of the code as if written in .py file
files Optional[List[File]]=[]* List of extra files to simulate imports if needed
disable_cache Optional[bool]=False Set to true disables mypy caching
mypy_config Optional[Dict[str, Union[str, int, bool, float]]]={} Inline mypy configuration, passed directly to mypy as --config-file option
env Optional[Dict[str, str]]={} Environmental variables to be provided inside of test run
parametrized Optional[List[Parameter]]=[]* List of parameters, similar to @pytest.mark.parametrize
skip str Expression evaluated with following globals set: sys, os, pytest and platform
expect_fail bool Mark test case as an expected failure, like @pytest.mark.xfail
regex str Allow regular expressions in comments to be matched against actual output. Defaults to "no", i.e. matches full text.

(*) Appendix to pseudo types used above:

class File:
    path: str
    content: Optional[str] = None
Parameter = Mapping[str, Any]

Implementation notes:

  • main must be non-empty string that evaluates to valid Python code,
  • content of each of extra files must evaluate to valid Python code,
  • parametrized entries must all be the objects of the same type. It simply means that each entry must have exact same set of keys,
  • skip - an expression set in skip is passed directly into eval. It is advised to take a peek and learn about how eval works.

Example

1. Inline type expectations

# typesafety/test_request.yml
- case: request_object_has_user_of_type_auth_user_model
  main: |
    from django.http.request import HttpRequest
    reveal_type(HttpRequest().user)  # N: Revealed type is 'myapp.models.MyUser'
    # check that other fields work ok
    reveal_type(HttpRequest().method)  # N: Revealed type is 'Union[builtins.str, None]'
  files:
    - path: myapp/__init__.py
    - path: myapp/models.py
      content: |
        from django.db import models
        class MyUser(models.Model):
            pass

2. @parametrized

- case: with_params
  parametrized:
    - val: 1
      rt: builtins.int
    - val: 1.0
      rt: builtins.float
  main: |
    reveal_type({[ val }})  # N: Revealed type is '{{ rt }}'

3. Longer type expectations

- case: with_out
  main: |
    reveal_type('abc')
  out: |
    main:1: note: Revealed type is 'builtins.str'

4. Regular expressions in expectations

- case: expected_message_regex_with_out
  regex: yes
  main: |
    a = 'abc'
    reveal_type(a)
  out: |
    main:2: note: .*str.*

5. Regular expressions specific lines of output.

- case: expected_single_message_regex
  main: |
    a = 'hello'
    reveal_type(a)  # NR: .*str.*

Options

mypy-tests:
  --mypy-testing-base=MYPY_TESTING_BASE
                        Base directory for tests to use
  --mypy-ini-file=MYPY_INI_FILE
                        Which .ini file to use as a default config for tests
  --mypy-same-process   Run in the same process. Useful for debugging, will create problems with import cache
  --mypy-extension-hook=MYPY_EXTENSION_HOOK
                        Fully qualified path to the extension hook function, in case you need custom yaml keys. Has to be top-level.
  --mypy-only-local-stub
                        mypy will ignore errors from site-packages

Further reading

License

MIT

Comments
  • Cut down on noise in expected strings

    Cut down on noise in expected strings

    Many times the full expectation string is long and includes low value information. I'd like to introduce some way to cut down on that noise. One thought is to make the expected string a fullmatch regex. This could also handle #45.

    - case: test_logic_promotion
      main: |
        from hdltypes.logic import StdLogic, X01Z, Bit
    
        a: StdLogic
        b: X01Z
        c: Bit
    
        reveal_type(c & a) # N: .* "hdltypes.logic.StdLogic(?:\\*|`-?\\d+)"
        reveal_type(c & b) # N: .* "hdltypes.logic.X01Z(?:\\*|`-?\\d+)"
        reveal_type(c & c) # N: .* "hdltypes.logic.Bit(?:\\*|`-?\\d+)"
    

    It might also be possible to make the expected strings templates to enable reuse, and perhaps package some of the common patterns with the tool.

    - case: test_logic_promotion
      main: |
        # ...
    
        reveal_type(c & a) # N: {reveal} "hdltypes.logic.StdLogic{_}"
        reveal_type(c & b) # N: {reveal} "hdltypes.logic.X01Z{_}"
        reveal_type(c & c) # N: {reveal} "hdltypes.logic.Bit{_}"
      meta:
        reveal: "Revealed type is"
        _: "(?:\\*|`-?\\d+)"
    

    The language here is strings, so using the common tools for that: regexes and templates, makes a lot of sense.

    opened by ktbarrett 19
  • Refactor and fix assert_expected_matched_actual

    Refactor and fix assert_expected_matched_actual

    This PR:

    • Refactors assert_expected_matched_actual function to avoid repeated matching between expected and actual output
    • Fixes #63
    • Fixes #64

    The following test file:

    - case: all_mismatched
      main: |
        reveal_type(42)  # N: Revealed type is "Literal['foo']?"
        reveal_type("foo")  # N: Revealed type is "Literal[42]?"
    
    - case: missing_message_then_match
      main: |
        reveal_type(42)
        reveal_type("foo")  # N: Revealed type is "Literal['foo']?"
    
    - case: match_then_missing_message
      main: |
        reveal_type(42)  # N: Revealed type is "Literal[42]?"
        reveal_type("foo")
    
    - case: missing_message
      main: |
        42 + "foo"
    
    - case: mismatched_message_inline
      main: |
        1 + 1  # E: Unsupported operand types for + ("int" and "int")
    
    - case: mismatched_messaged_in_out
      main: |
        1 + "foo"
      out: |
        main:1: error: Unsupported operand types for + ("int" and "int")
    
    - case: match_then_mismatched_message
      main: |
        reveal_type(42)  # N: Revealed type is "Literal[42]?"
        reveal_type("foo")  # N: Revealed type is "builtins.int"
    
    - case: mismatched_message_then_match
      main: |
        reveal_type("foo")  # N: Revealed type is "builtins.int"
        reveal_type(42)  # N: Revealed type is "Literal[42]?"
    
    - case: match_between_mismatched_messages
      main: |
        reveal_type(42.0)  # N: Revealed type is "builtins.float"
        reveal_type("foo")  # N: Revealed type is "builtins.int"
        reveal_type(42)  # N: Revealed type is "Literal[42]?"
    

    has been used to check for expected failures and gives output as shown below

    test-expect-fail.yaml FFFFFFFFF                                                     [100%]
    
    ======================================== FAILURES =========================================
    _____________________________________ all_mismatched ______________________________________
    /home/zero323/Workspace/test-reg/failing/test-expect-fail.yaml:3: 
    E   pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output: 
    E   Actual:
    E     main:1: note: Revealed type is "Literal[42]?" (diff)
    E     main:2: note: Revealed type is "Literal['foo']?" (diff)
    E   Expected:
    E     main:1: note: Revealed type is "Literal['foo']?" (diff)
    E     main:2: note: Revealed type is "Literal[42]?" (diff)
    E   Alignment of first line difference:
    E     E: ...ed type is "Literal['foo']?"
    E     A: ...ed type is "Literal[42]?"
    E                               ^
    _______________________________ missing_message_then_match ________________________________
    /home/zero323/Workspace/test-reg/failing/test-expect-fail.yaml:9: 
    E   pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output: 
    E   Actual:
    E     main:1: note: Revealed type is "Literal[42]?" (diff)
    E     main:2: note: Revealed type is "Literal['foo']?" (diff)
    E   Expected:
    E     main:2: note: Revealed type is "Literal['foo']?" (diff)
    E   Alignment of first line difference:
    E     E: main:2: note: Revealed type is "Literal['foo']?"
    E     A: main:1: note: Revealed type is "Literal[42]?"
    E             ^
    _______________________________ match_then_missing_message ________________________________
    /home/zero323/Workspace/test-reg/failing/test-expect-fail.yaml:12: 
    E   pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output: 
    E   Actual:
    E     main:2: note: Revealed type is "Literal['foo']?" (diff)
    E   Expected:
    E     (empty)
    _____________________________________ missing_message _____________________________________
    /home/zero323/Workspace/test-reg/failing/test-expect-fail.yaml:17: 
    E   pytest_mypy_plugins.utils.TypecheckAssertionError: Output is not expected: 
    E   Actual:
    E     main:1: error: Unsupported operand types for + ("int" and "str") (diff)
    E   Expected:
    E     (empty)
    ________________________________ mismatched_message_inline ________________________________
    /home/zero323/Workspace/test-reg/failing/test-expect-fail.yaml:22: 
    E   pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output: 
    E   Actual:
    E     (empty)
    E   Expected:
    E     main:1: error: Unsupported operand types for + ("int" and "int") (diff)
    _______________________________ mismatched_messaged_in_out ________________________________
    /home/zero323/Workspace/test-reg/failing/test-expect-fail.yaml:26: 
    E   pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output: 
    E   Actual:
    E     main:1: error: Unsupported operand types for + ("int" and "str") (diff)
    E   Expected:
    E     main:1: error: Unsupported operand types for + ("int" and "int") (diff)
    E   Alignment of first line difference:
    E     E: ...rand types for + ("int" and "int")
    E     A: ...rand types for + ("int" and "str")
    E                                        ^
    ______________________________ match_then_mismatched_message ______________________________
    /home/zero323/Workspace/test-reg/failing/test-expect-fail.yaml:33: 
    E   pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output: 
    E   Actual:
    E     ...
    E     main:2: note: Revealed type is "Literal['foo']?" (diff)
    E   Expected:
    E     ...
    E     main:2: note: Revealed type is "builtins.int" (diff)
    E   Alignment of first line difference:
    E     E: ...te: Revealed type is "builtins.int"
    E     A: ...te: Revealed type is "Literal['foo']?"
    E                                 ^
    ______________________________ mismatched_message_then_match ______________________________
    /home/zero323/Workspace/test-reg/failing/test-expect-fail.yaml:37: 
    E   pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output: 
    E   Actual:
    E     main:1: note: Revealed type is "Literal['foo']?" (diff)
    E     ...
    E   Expected:
    E     main:1: note: Revealed type is "builtins.int" (diff)
    E     ...
    E   Alignment of first line difference:
    E     E: ...te: Revealed type is "builtins.int"
    E     A: ...te: Revealed type is "Literal['foo']?"
    E                                 ^
    ____________________________ match_between_mismatched_messages ____________________________
    /home/zero323/Workspace/test-reg/failing/test-expect-fail.yaml:43: 
    E   pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output: 
    E   Actual:
    E     ...
    E     main:2: note: Revealed type is "Literal['foo']?" (diff)
    E     ...
    E   Expected:
    E     ...
    E     main:2: note: Revealed type is "builtins.int" (diff)
    E     ...
    E   Alignment of first line difference:
    E     E: ...te: Revealed type is "builtins.int"
    E     A: ...te: Revealed type is "Literal['foo']?"
    E                                 ^
    ================================= short test summary info =================================
    FAILED test-expect-fail.yaml::all_mismatched - 
    FAILED test-expect-fail.yaml::missing_message_then_match - 
    FAILED test-expect-fail.yaml::match_then_missing_message - 
    FAILED test-expect-fail.yaml::missing_message - 
    FAILED test-expect-fail.yaml::mismatched_message_inline - 
    FAILED test-expect-fail.yaml::mismatched_messaged_in_out - 
    FAILED test-expect-fail.yaml::match_then_mismatched_message - 
    FAILED test-expect-fail.yaml::mismatched_message_then_match - 
    FAILED test-expect-fail.yaml::match_between_mismatched_messages - 
    ==================================== 9 failed in 2.77s ====================================
    
    opened by zero323 12
  • Test that snippet passes

    Test that snippet passes

    In mypy internal testing suite I can omit output to indicate that I expect snippet to pass

    [case import]
    from collections import namedtuple
    [out]
    

    This is very useful for basics checks.

    However, I cannot get similar behaviour here. If I create a package

    ├── mypackage
    │   └── __init__.py
    ├── mypy.ini
    └── test
        └── test_mypackage.yaml
    

    with

    # ./mypackage/__init__.py
    def bar(x: int) -> None:
        pass
    

    and

    ./test/test_mypackage.yaml
    - case: test_mypackage
      main: |
        from mypackage import foo
    

    tests run without errors (pytest-mypy-plugins==1.9.1, mypy==0.910) despite incorrect import foo.

    I have to put some output check

    ./test/test_mypackage.yaml
    - case: test_mypackage
      main: |
        from mypackage import foo      
        reveal_type(True)  # N: Revealed type is "Literal[True]?"
    
    

    to get an exception.

    I also tried using empty out block

    - case: test_mypackage
      main: |
        from mypackage import foo      
        reveal_type(True)
      out: ""
    

    but it still silently ignores the broken import.

    opened by 0x143 10
  • pdb not catching although --mypy-same-process

    pdb not catching although --mypy-same-process

    Hey @sobolevn, hope you're doing great in those crazy days !

    Anyway, I want to keep this short. I am trying to debug a certain issue in my code but I have the problem. Neither pycharm nor pdb is able to break in a place I want. It makes no difference whether I make a breakpoint or do pdb.set_trace(). It just ignores that.

    Here's relevant portion of my setup https://github.com/kornicameister/axion/blob/master/pytest.ini#L9 Any hints are welcome.

    opened by kornicameister 10
  • Errors reported for 3rd-party dependencies

    Errors reported for 3rd-party dependencies

    When I run type-checking of my code with mypy - everything works correctly. But, when I try to run type-tests on some module, then it breaks, example:

    E     ../../../../../../Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/hypothesis/vendor/pretty:706: error: Call to untyped function "_safe_getattr" in typed context (diff)
    
    opened by sobolevn 8
  • Update path argument

    Update path argument

    This PR introduces compatibility with pytest 7.0.0rc1. As described in the link included in the original issue. fspath with LocalPath is deprecated in favor of path with pathlib.Path.

    Closes #88

    opened by zero323 7
  • Output-file rewriting removes '.py' in too many places

    Output-file rewriting removes '.py' in too many places

    The last line of pytest_mypy_plugins.items.replace_fpath_with_module_name() does a blanket removal of all ".py" strings:

    return line.strip().replace(".py", "")
    

    This destroys paths in log information when mypy.verbosity > 0. Paths like "$HOME/.pyenv" and ".../python3.6/.../__init__.pyi" are converted into random strings.

    I suspect you are only trying to replace "main.py: 1: note: ..." -type strings at the beginning of the line. If that is true (and I really haven't explored enough to be sure, so please check this) then a simple replacement my be replace(".py:", ":") to ensure the offending .py looks like a filespec.

    Otherwise maybe only do the replacement in the first colon-delimited field?

    bug 
    opened by aghast 7
  • AttributeError: type object 'YamlTestFile' has no attribute 'from_parent'  in pytest_collect_file

    AttributeError: type object 'YamlTestFile' has no attribute 'from_parent' in pytest_collect_file

    The latest update to pytest_mypy_plugins broke my CI tests with the following error

    ==================================== ERRORS ====================================
    
    ______________________ ERROR collecting tests/__init__.py ______________________
    
    ../../../virtualenv/python3.8.0/lib/python3.8/site-packages/pluggy/hooks.py:286: in __call__
    
        return self._hookexec(self, self.get_hookimpls(), kwargs)
    
    ../../../virtualenv/python3.8.0/lib/python3.8/site-packages/pluggy/manager.py:92: in _hookexec
    
        return self._inner_hookexec(hook, methods, kwargs)
    
    ../../../virtualenv/python3.8.0/lib/python3.8/site-packages/pluggy/manager.py:83: in <lambda>
    
        self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
    
    ../../../virtualenv/python3.8.0/lib/python3.8/site-packages/pytest_mypy_plugins/collect.py:139: in pytest_collect_file
    
        return YamlTestFile.from_parent(parent, fspath=path)
    
    E   AttributeError: type object 'YamlTestFile' has no attribute 'from_parent'
    
    !!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!
    
    =============================== 1 error in 0.11s ===============================
    
    The command "pytest" exited with 2.
    

    view the log outputs here: https://travis-ci.org/github/lovasoa/marshmallow_dataclass/jobs/705848682

    opened by lovasoa 7
  • How can I test that an assignment is legal? i.e that no output is generated

    How can I test that an assignment is legal? i.e that no output is generated

    Thanks for this great tool, its been indispensable to my project!

    I have a plugin that defines a protocol intersection type, and I want to test that it is invariant to argument order. My test case looks like this:

    from typing import Protocol
    
    from my_package import Intersection
    
    class P1(Protocol):
        pass
    
    class P1(Protocol):
        pass
    
    i: Intersection[P1, P2]
    i2: Intersection[P2, P1] = i  # I want to assert that no output is generated on this line
    
    opened by suned 6
  • Test failures using Python 3.11 and Mypy 0.981

    Test failures using Python 3.11 and Mypy 0.981

    Hi!

    Tests are failing on Python 3.11 due to a Mypy error and I'm not sure why. :thinking:

    I forked pytest-mypy-plugins and added 3.11-dev Python version to CI tests: https://github.com/Delgan/pytest-mypy-plugins/commit/27f9037897507026519c67fcf016758bd2b3d7a6 You can see the failing test here: https://github.com/Delgan/pytest-mypy-plugins/actions/runs/3166219555/jobs/5155788328

    E     ../../opt/hostedtoolcache/Python/3.11.0-rc.2/x64/lib/python3.11/site-packages/mypy/typeshed/stdlib/builtins.pyi:1865: error: Overloaded function signatures 1 and 2 overlap with incompatible return types (diff)
    E     ../../opt/hostedtoolcache/Python/3.11.0-rc.2/x64/lib/python3.11/site-packages/mypy/typeshed/stdlib/builtins.pyi:1885: error: Overloaded function signatures 1 and 2 overlap with incompatible return types (diff)
    

    When I manually run mypy on the code base no error is reported, so I'm a bit confused about why it's failing when tests are ran through pytest and this plugin.

    opened by Delgan 4
  • Does not work with pytest-cov

    Does not work with pytest-cov

    » pytest
    ================================ test session starts =================================
    platform darwin -- Python 3.7.3, pytest-4.6.3, py-1.7.0, pluggy-0.12.0
    Using --randomly-seed=1560429839
    rootdir: /Users/sobolev/Documents/github/returns, inifile: setup.cfg
    plugins: mypy-plugins-0.3.0, asyncio-0.10.0, cov-2.7.1, randomly-3.0.0
    collected 49 items / 3 errors / 46 selected                                          
    INTERNALERROR> Traceback (most recent call last):
    INTERNALERROR>   File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/_pytest/main.py", line 206, in wrap_session
    INTERNALERROR>     session.exitstatus = doit(config, session) or 0
    INTERNALERROR>   File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/_pytest/main.py", line 250, in _main
    INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
    INTERNALERROR>   File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/pluggy/hooks.py", line 289, in __call__
    INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
    INTERNALERROR>   File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/pluggy/manager.py", line 87, in _hookexec
    INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
    INTERNALERROR>   File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/pluggy/manager.py", line 81, in <lambda>
    INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
    INTERNALERROR>   File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/pluggy/callers.py", line 203, in _multicall
    INTERNALERROR>     gen.send(outcome)
    INTERNALERROR>   File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/pytest_cov/plugin.py", line 229, in pytest_runtestloop
    INTERNALERROR>     self.cov_controller.finish()
    INTERNALERROR>   File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/pytest_cov/engine.py", line 171, in finish
    INTERNALERROR>     self.cov.stop()
    INTERNALERROR>   File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/coverage/control.py", line 818, in combine
    INTERNALERROR>     self.data, aliases=aliases, data_paths=data_paths, strict=strict,
    INTERNALERROR>   File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/coverage/data.py", line 736, in combine_parallel_data
    INTERNALERROR>     data.update(new_data, aliases=aliases)
    INTERNALERROR>   File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/coverage/data.py", line 488, in update
    INTERNALERROR>     raise CoverageException("Can't combine line data with arc data")
    INTERNALERROR> coverage.misc.CoverageException: Can't combine line data with arc data
    
    

    Configuration: https://github.com/dry-python/returns/blob/master/setup.cfg

    Test: test-data/typecheck/return.test

    [CASE no_incompatible_meta_nested_class_false_positive]
    1 + '2'
    [/CASE]
    
    opened by sobolevn 4
  • Feature proposal: Allow specifying out as list

    Feature proposal: Allow specifying out as list

    I think the current format for output could be iterated on, and be made a little friendlier to work with. Instead of today's format using multiline strings:

    out: |
      main:7: error: Argument 1 has incompatible type "int"; expected "str"  [arg-type]
      main:9: note: Revealed type is "builtins.bool*"
    

    We could allow specifying out as a list of objects with properties:

    out:
      - error: 'Argument 1 has incompatible type "int"; expected "str"  [arg-type]'
        line: 7
      - note: 'Revealed type is "builtins.bool"'
        line: 9
    

    This would unlock a few different things we could then implement at a higher level of abstraction. There are probably more that will make sense, these are just the ones I can think of for now.

    Special support for error-codes

    out:
      - error: 'Argument 1 has incompatible type "int"; expected "str"'
        code: "arg-type"
        line: 7
    

    Special support for revealed type

    out:
      - revealed_type: "builtins.bool"
        line: 9
    

    Allow mixing regex with non-regex

    out:
      - error: 'Argument 1 has incompatible type "int\*?"; expected "str\*?"'
        line: 7
        regex: true
    

    Omitting line numbers

    This probably requires more knowledge of the internals than I currently have, but I think it might make sense to allow omitting line numbers. I'm proposing that this example would match if there's revealed type on line nine, followed by an error message on any line after that (strictly greater than line 9).

    out:
      - note: 'Revealed type is "builtins.bool"'
        line: 9
      - error: 'Argument 1 has incompatible type "int"; expected "str"'
        code: "arg-type"
    

    What do you think? 🙂

    opened by antonagestam 3
  • mypy_config documentation is confusing

    mypy_config documentation is confusing

    The mypy_config field accepts a string as below.

    - case: foo
      mypy_config:
        python_version = 3.8
    

    So, if we want to provide multi fields, it would be like below, where the value of mypy_config in the YAML format is of type str.

    - case: foo
      mypy_config: |
        python_version = 3.8
        ignore_missing_imports = True
    

    However, the document says its type is Optional[Dict[str, Union[str, int, bool, float]]]={} which I think means the following YAML format (dict syntax) is accepted.

    - case: foo
      mypy_config:
        python_version: 3.8
        ignore_missing_imports: true
    

    but actually it is not.


    My suggestion is either

    • to change the documentation to say the type of mypy_config is str, or
    • to change the implementation to accept the YAML dict.

    because the files field, which is documented to accept Optional[List[File]]=[] type, can be specified with the YAML list syntax like

        - path: myapp/__init__.py
    

    , which is inconsistent and confusing.


    The same problem is there on the env field. Its type is documented as Optional[Dict[str, str]]={}, but the YAML syntax for the field is list as follows:

      env:
        - MYPYPATH=./pytest_mypy_plugins/tests/fixtures
    
    opened by whitphx 2
  • Document cache directory usage

    Document cache directory usage

    When tests are executed with typecheck_in_new_subprocess and disable_cache is False, --cache-dir is passed directly to mypy call as a path relative to root directory.

    This means that the following are ignored

    • MYPY_CACHE_DIRin execution environment orenv` block of the test case.
    • cache_dir in mypy configuration.

    are ignored.

    This behavior is confusing, so it might be a good idea to document it.

    In a long run, YamlTestItem should check if any of these are provided, and omit --cache-dir in such cases (this would be useful for example with remote cache).

    opened by zero323 0
  • Broken  mypy cache?

    Broken mypy cache?

    I am trying to investigate some issues related to caching behavior. When testing project with complex dependencies, I see serious performance degradation (roughly 20 fold on just 36 tests) compared to using mypy.test.testcheckTypeCheckSuite directly.

    I thought the issue was a simple logic mistake (#82), but it seems it might be actually with find_dependent_paths

    https://github.com/typeddjango/pytest-mypy-plugins/blob/a2d4adde12b0024e62f2e1661fd0dd5abb4f9191/pytest_mypy_plugins/item.py#L187

    Since it uses at least main.py it includes all kinds of packages using main as a name (not necessarily as a module name, could be even an argument), for example

    ['/tmp/.mypy_cache/3.9/pdb',
     '/tmp/.mypy_cache/3.9/unittest/main',
     '/tmp/.mypy_cache/3.9/unittest/__init__',
     '/tmp/.mypy_cache/3.9/_pytest/pytester',
     '/tmp/.mypy_cache/3.9/_pytest/config/__init__',
     '/tmp/.mypy_cache/3.9/pytest/__init__',
     '/tmp/.mypy_cache/3.9/asyncio/runners']
    

    This seems to escalate (in my case, to numpy annotations, for reason yet to be determined), and break caching in general.

    Possibly related to #37

    Originally posted by @zero323 in https://github.com/typeddjango/pytest-mypy-plugins/issues/82#issuecomment-945904250

    opened by zero323 2
  • Support for mypy  .test files.

    Support for mypy .test files.

    This is a feature request.

    It might be nice to get support for handling mypy-style .test files:

    [case foo]
    
    reveal_type(1 + 1)  # N: Revealed type is "builtins.int"
    
    [out]]
    
    -- Comment
    
    [case bar]
    
    reveal_type(1 + 1)
    
    [out]
    main:2: note: Revealed type is "builtins.int"
    
    
    opened by zero323 3
  • Improve line matching behavior

    Improve line matching behavior

    This is a feature request.

    Summary:

    Currently rows are matched by their position in the file / output.

    User experience could be improved, if matching was performed by (file, line-number).

    Details:

    Let's assume that I have a test case like this

    - case: break_following_2
      main: |
        reveal_type(1 + 1) 
        reveal_type(1.0 + 2.0) # N: Revealed type is "builtins.float"
        reveal_type("foo" + "bar") # N: Revealed type is "builtins.str"
    

    When I run tests I see:

    E   pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output: 
    E   Expected:
    E     main:2: note: Revealed type is "builtins.float" (diff)
    E     main:3: note: Revealed type is "builtins.str" (diff)
    E   Actual:
    E     main:1: note: Revealed type is "builtins.int" (diff)
    E     main:2: note: Revealed type is "builtins.float" (diff)
    E     main:3: note: Revealed type is "builtins.str" (diff)
    E   
    E   Alignment of first line difference:
    E     E: main:2: note: Revealed type is "builtins.float"
    E     A: main:1: note: Revealed type is "builtins.int"
    E
    

    If you analyze the test case, you'll see that actual state is like this:

    | line| actual | expected | match | | ---- | ----------------------------------------- | ------------------------------------------- | ------------ | | 1 | Revealed type is "builtins.int" | | ✘ | | 2 | Revealed type is "builtins.float" | Revealed type is "builtins.float" | ✓ | | 3 | Revealed type is "builtins.str" | Revealed type is "builtins.str" | ✓ |

    however alignment message

    E   Alignment of first line difference:
    E     E: main:2: note: Revealed type is "builtins.float"
    E     A: main:1: note: Revealed type is "builtins.int"
    

    clearly shows that we start with comparing line 2 of expected and line 1 of actual.

    This escalates to all the following lines and probably gets worse with multi-line messages (I wanted to investigate that, hence #66).

    I am aware that this is consistent with behavior of the internal mypy test suite, which returns

    Expected:
      main:2: note: Revealed type is "builtins.float" (diff)
      main:3: note: Revealed type is "builtins.str" (diff)
    Actual:
      main:1: note: Revealed type is "builtins.int" (diff)
      main:2: note: Revealed type is "builtins.float" (diff)
      main:3: note: Revealed type is "builtins.str" (diff)
    
    Alignment of first line difference:
      E: main:2: note: Revealed type is "builtins.float"
      A: main:1: note: Revealed type is "builtins.int"
              ^
    

    for equivalent input, but it seems a bit counter-intuitive. While it detects presence of output mismatch, it cannot do much beyond that.

    Ideally, I'd like to see something around these lines:

    E   pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output: 
    E   Expected:
    E     (Empty)
    E      ...
    E   Actual:
    E     main:1: note: Revealed type is "builtins.int" (diff)
    E     ...
    

    (no alignment needed for an empty line).

    This should generalize to multiple interleaved blocks of matching and not matching lines, where matching blocks are indicated, but omitted.

    Furthermore, errors shouldn't propagate beyond current line, in case of multline output.

    Originally posted by @zero323 in https://github.com/typeddjango/pytest-mypy-plugins/issues/65#issuecomment-938144873

    opened by zero323 3
Releases(1.10.1)
Owner
TypedDjango
We make types for Django framework!
TypedDjango
Baseball Discord bot that can post up-to-date scores, lineups, and home runs.

Sunny Day Discord Bot Baseball Discord bot that can post up-to-date scores, lineups, and home runs. Uses webscraping techniques to scrape baseball dat

Benjamin Hammack 1 Jun 20, 2022
Lightweight, scriptable browser as a service with an HTTP API

Splash - A javascript rendering service Splash is a javascript rendering service with an HTTP API. It's a lightweight browser with an HTTP API, implem

Scrapinghub 3.8k Jan 03, 2023
Turn any OpenAPI2/3 and Postman Collection file into an API server with mocking, transformations and validations.

Prism is a set of packages for API mocking and contract testing with OpenAPI v2 (formerly known as Swagger) and OpenAPI v3.x. Mock Servers: Life-like

Stoplight 3.3k Jan 05, 2023
Make Selenium work on Github Actions

Make Selenium work on Github Actions Scraping with BeautifulSoup on GitHub Actions is easy-peasy. But what about Selenium?? After you jump through som

Jonathan Soma 33 Dec 27, 2022
Integration layer between Requests and Selenium for automation of web actions.

Requestium is a Python library that merges the power of Requests, Selenium, and Parsel into a single integrated tool for automatizing web actions. The

Tryolabs 1.7k Dec 27, 2022
A Python Selenium library inspired by the Testing Library

Selenium Testing Library Slenium Testing Library (STL) is a Python library for Selenium inspired by Testing-Library. Dependencies Python 3.6, 3.7, 3.8

Anže Pečar 12 Dec 26, 2022
bulk upload files to libgen.lc (Selenium script)

LibgenBulkUpload bulk upload files to http://libgen.lc/librarian.php (Selenium script) Usage ./upload.py to_upload uploaded rejects So title and autho

8 Jul 07, 2022
PyBuster A directory busting tool for web application penetration tester, written in python

PyBuster A directory busting tool for web application penetration tester, written in python. Supports custom wordlist,recursive search. Screenshots Pr

Anukul Pandey 4 Jan 30, 2022
Tools for test driven data-wrangling and data validation.

datatest: Test driven data-wrangling and data validation Datatest helps to speed up and formalize data-wrangling and data validation tasks. It impleme

269 Dec 16, 2022
✅ Python web automation and testing. 🚀 Fast, easy, reliable. 💠

Build fast, reliable, end-to-end tests. SeleniumBase is a Python framework for web automation, end-to-end testing, and more. Tests are run with "pytes

SeleniumBase 3k Jan 04, 2023
A modern API testing tool for web applications built with Open API and GraphQL specifications.

Schemathesis Schemathesis is a modern API testing tool for web applications built with Open API and GraphQL specifications. It reads the application s

Schemathesis.io 1.6k Jan 06, 2023
Codeforces Test Parser for C/C++ & Python on Windows

Codeforces Test Parser for C/C++ & Python on Windows Installation Run pip instal

Minh Vu 2 Jan 05, 2022
Rerun pytest when your code changes

A simple watcher for pytest Overview pytest-watcher is a tool to automatically rerun pytest when your code changes. It looks for the following events:

Olzhas Arystanov 74 Dec 29, 2022
Fills out the container extension form automatically. (Specific to IIT Ropar)

automated_container_extension Fills out the container extension form automatically. (Specific to IIT Ropar) Download the chrome driver from the websit

Abhishek Singh Sambyal 1 Dec 24, 2021
Pynguin, The PYthoN General UnIt Test geNerator is a test-generation tool for Python

Pynguin, the PYthoN General UnIt test geNerator, is a tool that allows developers to generate unit tests automatically.

Chair of Software Engineering II, Uni Passau 997 Jan 06, 2023
WomboAI Art Generator

WomboAI Art Generator Automate AI art generation using wombot.art. Also integrated into SnailBot for you to try out. Setup Install Python Go to the py

nbee 7 Dec 03, 2022
输入Google Hacking语句,自动调用Chrome浏览器爬取结果

Google-Hacking-Crawler 该脚本可输入Google Hacking语句,自动调用Chrome浏览器爬取结果 环境配置 python -m pip install -r requirements.txt 下载Chrome浏览器

Jarcis 4 Jun 21, 2022
Code for "SUGAR: Subgraph Neural Network with Reinforcement Pooling and Self-Supervised Mutual Information Mechanism"

SUGAR Code for "SUGAR: Subgraph Neural Network with Reinforcement Pooling and Self-Supervised Mutual Information Mechanism" Overview train.py: the cor

41 Nov 08, 2022
Auto Click by pyautogui and excel operations.

Auto Click by pyautogui and excel operations.

Janney 2 Dec 21, 2021
PacketPy is an open-source solution for stress testing network devices using different testing methods

PacketPy About PacketPy is an open-source solution for stress testing network devices using different testing methods. Currently, there are only two c

4 Sep 22, 2022