Get information about what a Python frame is currently doing, particularly the AST node being executed

Related tags

Miscellaneouspython
Overview

executing

Build Status Coverage Status Supports Python versions 2.7 and 3.4+, including PyPy

This mini-package lets you get information about what a frame is currently doing, particularly the AST node being executed.

Usage

Getting the AST node

import executing

node = executing.Source.executing(frame).node

Then node will be an AST node (from the ast standard library module) or None if the node couldn't be identified (which may happen often and should always be checked).

node will always be the same instance for multiple calls with frames at the same point of execution.

If you have a traceback object, pass it directly to Source.executing() rather than the tb_frame attribute to get the correct node.

Getting the source code of the node

For this you will need to separately install the asttokens library, then obtain an ASTTokens object:

executing.Source.executing(frame).source.asttokens()

or:

executing.Source.for_frame(frame).asttokens()

or use one of the convenience methods:

executing.Source.executing(frame).text()
executing.Source.executing(frame).text_range()

Getting the __qualname__ of the current function

executing.Source.executing(frame).code_qualname()

or:

executing.Source.for_frame(frame).code_qualname(frame.f_code)

The Source class

Everything goes through the Source class. Only one instance of the class is created for each filename. Subclassing it to add more attributes on creation or methods is recommended. The classmethods such as executing will respect this. See the source code and docstrings for more detail.

Installation

pip install executing

If you don't like that you can just copy the file executing.py, there are no dependencies (but of course you won't get updates).

How does it work?

Suppose the frame is executing this line:

self.foo(bar.x)

and in particular it's currently obtaining the attribute self.foo. Looking at the bytecode, specifically frame.f_code.co_code[frame.f_lasti], we can tell that it's loading an attribute, but it's not obvious which one. We can narrow down the statement being executed using frame.f_lineno and find the two ast.Attribute nodes representing self.foo and bar.x. How do we find out which one it is, without recreating the entire compiler in Python?

The trick is to modify the AST slightly for each candidate expression and observe the changes in the bytecode instructions. We change the AST to this:

(self.foo ** 'longuniqueconstant')(bar.x)

and compile it, and the bytecode will be almost the same but there will be two new instructions:

LOAD_CONST 'longuniqueconstant'
BINARY_POWER

and just before that will be a LOAD_ATTR instruction corresponding to self.foo. Seeing that it's in the same position as the original instruction lets us know we've found our match.

Is it reliable?

Yes - if it identifies a node, you can trust that it's identified the correct one. The tests are very thorough - in addition to unit tests which check various situations directly, there are property tests against a large number of files (see the filenames printed in this build) with real code. Specifically, for each file, the tests:

  1. Identify as many nodes as possible from all the bytecode instructions in the file, and assert that they are all distinct
  2. Find all the nodes that should be identifiable, and assert that they were indeed identified somewhere

In other words, it shows that there is a one-to-one mapping between the nodes and the instructions that can be handled. This leaves very little room for a bug to creep in.

Furthermore, executing checks that the instructions compiled from the modified AST exactly match the original code save for a few small known exceptions. This accounts for all the quirks and optimisations in the interpreter.

Which nodes can it identify?

Currently it works in almost all cases for the following ast nodes:

  • Call, e.g. self.foo(bar)
  • Attribute, e.g. point.x
  • Subscript, e.g. lst[1]
  • BinOp, e.g. x + y (doesn't include and and or)
  • UnaryOp, e.g. -n (includes not but only works sometimes)
  • Compare e.g. a < b (not for chains such as 0 < p < 1)

The plan is to extend to more operations in the future.

Libraries that use this

My libraries

  • stack_data: Extracts data from stack frames and tracebacks, particularly to display more useful tracebacks than the default. Also uses another related library of mine: pure_eval.
  • snoop: A feature-rich and convenient debugging library. Uses executing to show the operation which caused an exception and to allow the pp function to display the source of its arguments.
  • heartrate: A simple real time visualisation of the execution of a Python program. Uses executing to highlight currently executing operations, particularly in each frame of the stack trace.
  • sorcery: Dark magic delights in Python. Uses executing to let special callables called spells know where they're being called from.

Libraries I've contributed to

  • IPython: Highlights the executing node in tracebacks using executing via stack_data.
  • icecream: 🍦 Sweet and creamy print debugging. Uses executing to identify where ic is called and print its arguments.
  • python-devtools: Uses executing for print debugging similar to icecream.
  • sentry_sdk: Add the integration sentry_sdk.integrations.executingExecutingIntegration() to show the function __qualname__ in each frame in sentry events. Highlighting the executing node is hopefully coming soon.
  • varname: Dark magics about variable names in python. Uses executing to find where its various magical functions like varname and nameof are called from.
Comments
  • feat: support for python 3.11

    feat: support for python 3.11

    • This provides a new implementation, which uses co_positions() to lookup the ast node.
    • It has a simpler implementation and better performance.
    • Some limitations in the unit tests are removed for 3.11.
      • support for and and or
      • no ambiguities for generators
      • fixes #30
    opened by 15r10nk 80
  • Let GH index who's using this package

    Let GH index who's using this package

    After experimenting a little bit, I found that for python packages, including a requirements.txt in the repo would enable GH to index who's using the package.

    The list can be seen on both the main page of the repo or Insights -> Dependency graph -> Dependents

    opened by pwwang 20
  • Use only pytest

    Use only pytest

    I was able to execute all tests with pytest. However assertion rewriting is still disabled for test_main.py.

    But this has already several benefits:

    • tox -e py310 -- --sw works
    • no need to maintain two test environments

    I also converted the test_sample_files to an parametrized test which makes it more useful for pytest --sw

    I also don't know how I should handle the timeouts. I removed them for test_sample_files because they are not really useful for single file checks. I kept them for test_module_files because this takes already a bit longer. What's your opinion there?

    I wanted to get your feedback before I continue.

    opened by 15r10nk 12
  • Small samples

    Small samples

    generate small samples to improve test coverage.

    • The samples where generated with tox -e mutmut (which calls tests/mutmut_workflow.py, which class tests/generate_small_sample.py).
    • The samples are placed in tests/small_samples

    Every sample is the result of a code mutation detected by mutmut which was not covered by any other test (excluding the long running tests).

    generate_small_sample.py tries to find a source file with some code which fails with the mutated implementation and removes code from this file with pysource_minimize.

    I have also found some issues:

    • The ExceptHandler cleanup is now an KnownIssue. Before it was verified as an valid mapping, which is not ok.
    • DELETE/STORE_DEREF had some issues.

    I had to reimplement the Deadcode analysis. The problem was that I always tried to reimplement the deadcode analysis of the python interpreter, which did not scale very well. The new implementation replaces the node with an sentinel compiles the source and checks if the sentinel is in the bytecode. The downside is that I had to remove the inverse deadcode check, because the new implementation is a bit more complex ( compiles the whole file for every node to test).

    Let me know what you think.

    closes #55

    opened by 15r10nk 10
  • fix decorator detection bug

    fix decorator detection bug

    try to fix #49

    I also reported frame.f_lasti and the problem is that the instruction is a CACHE instruction in the deco case (f_lasti == 130).

    The Tester case gets the expected instruction and maps the bytecode correctly.

    You can find my source here https://github.com/15r10nk/executing/tree/decorator-detection-bug

    grafik

    I could ignore CACHE instructions until now. I will try to figure out what they are good for.

    opened by 15r10nk 9
  • Support for Python 3.11?

    Support for Python 3.11?

    With today's release of Python 3.11.0a3, I suddenly have many failing tests for friendly-traceback that were still passing with 3.11.0a2. It looks like the nodes are no longer properly identified.

    For example, if I have

    a = (1, 2)(3, 4)
    

    the node was previously identified as (1, 2)(3, 4) whereas now it is the entire line that is identified as the problematic node.

    I can provide more examples if needed, but I suspect that your own unit tests would likely give you some results. It might be that the problem is upstream (with asttoken) instead of in executing.

    opened by aroberge 9
  • Bug:  Incorrect node text when __slots__ and class variables are used (3.11)

    Bug: Incorrect node text when __slots__ and class variables are used (3.11)

    Apologies for not providing a self-contained test with executing for the following, as I don't know how. The following raises a ValueError exception:

    class F:
        __slots__ = ["a", "b"]
        a = 1
    

    With Python 3.10 (and prior), the text of the node identified causing the exception is:

    class F:
    

    With Python 3.11, we get this:

    class F: __slots__ = ["a", "b"] a = 1
    

    This is not a typo: the text of the node identified a single line, with no \n, which is clearly a SyntaxError.

    This is the only bug I found when running the unit tests for friendly_traceback with executing 1.1.0 and Python 3.11. Everything still works for me with Python 3.6 to 3.10, although I found that the tests I run take about 2 to 3 times as long as compared with using executing 0.8.3. The slowdown seemed to be already present with executing 1.0.0.

    opened by aroberge 7
  • Executing cannot find node inside Jupyter lab/notebooks

    Executing cannot find node inside Jupyter lab/notebooks

    Using friendly, I found that excuting could not locate the correct location of a node causing a problem inside Jupyter notebooks (or Jupyter lab) but could do so with IPython. First, here's a screen capture showing the correct result using IPython, with some additional outputs from print statements inserted for this specific report.

    image

    Next, the same code run within a Jupyter notebook image

    Perhaps this is is caused by the new way that Jupyter runs code, using temporary files instead of using exec() on code.

    With the "old" friendly-traceback (before it made use of stack_data), when Jupyter was not using temporary files, I see from screen captures on https://aroberge.github.io/friendly-traceback-docs/docs/html/jupyter.html that the location was correctly picked up.

    opened by aroberge 6
  • no source distribution on pypi

    no source distribution on pypi

    Would it be possible to upload the sdist in addition to the wheel file on pypi? Currently the conda-forge distribution does not automatically pick up updates unless the sources are present.

    opened by Anthchirp 6
  • 0.8.3: pytest is failing

    0.8.3: pytest is failing

    I'm trying to package your module as an rpm package. So I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

    • python3 -sBm build -w --no-isolation
    • because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
    • install .whl file in </install/prefix>
    • run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

    Here is pytest output:

    + PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-executing-0.8.3-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-executing-0.8.3-2.fc35.x86_64/usr/lib/python3.8/site-packages
    + /usr/bin/pytest -ra
    =========================================================================== test session starts ============================================================================
    platform linux -- Python 3.8.13, pytest-7.1.1, pluggy-1.0.0
    rootdir: /home/tkloczko/rpmbuild/BUILD/executing-0.8.3
    collected 2 items / 1 error
    
    ================================================================================== ERRORS ==================================================================================
    ___________________________________________________________________ ERROR collecting tests/test_main.py ____________________________________________________________________
    executing/executing.py:317: in executing
        args = executing_cache[key]
    E   KeyError: (<code object <module> at 0x7f98855a0ea0, file "/home/tkloczko/rpmbuild/BUILD/executing-0.8.3/tests/test_main.py", line 2>, 140293049028256, 418)
    
    During handling of the above exception, another exception occurred:
    executing/executing.py:346: in find
        node_finder = NodeFinder(frame, stmts, tree, lasti)
    executing/executing.py:636: in __init__
        matching = list(self.matching_nodes(exprs))
    executing/executing.py:702: in matching_nodes
        original_index = only(
    executing/executing.py:164: in only
        raise NotOneValueFound('Expected one value, found 0')
    E   executing.executing.NotOneValueFound: Expected one value, found 0
    
    During handling of the above exception, another exception occurred:
    tests/test_main.py:682: in <module>
        assert tester([1, 2, 3]) == [1, 2, 3]
    tests/utils.py:40: in __call__
        ex = self.get_executing(inspect.currentframe().f_back)
    tests/utils.py:28: in get_executing
        return Source.executing(frame)
    executing/executing.py:369: in executing
        args = find(source=cls.for_frame(frame), retry_cache=True)
    executing/executing.py:355: in find
        return find(
    executing/executing.py:346: in find
        node_finder = NodeFinder(frame, stmts, tree, lasti)
    executing/executing.py:636: in __init__
        matching = list(self.matching_nodes(exprs))
    executing/executing.py:702: in matching_nodes
        original_index = only(
    executing/executing.py:164: in only
        raise NotOneValueFound('Expected one value, found 0')
    E   executing.executing.NotOneValueFound: Expected one value, found 0
    ========================================================================= short test summary info ==========================================================================
    ERROR tests/test_main.py - executing.executing.NotOneValueFound: Expected one value, found 0
    !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    ============================================================================= 1 error in 0.44s =============================================================================
    
    opened by kloczek 5
  • Is it possible to know whether a frame is from a method call?

    Is it possible to know whether a frame is from a method call?

    For example

    def do_func(self):
        frame = sys._getframe()
    
    class A:
        def do_method(self):
            frame = sys._getframe()
    

    Is it possible to know whether frame is inside a regular function call or method call? From https://stackoverflow.com/q/2203424/2142577 it seems this is no way except relying on the convention that self is the first argument of method signature. Do we have a better way?

    I also tried using gc.get_referrers(frame.f_code)), but it turns out the code is referrenced by the unbound function, no matter whether it is actually a method.

    opened by laike9m 4
  • Index node finder

    Index node finder

    This is still work in progress.

    The goal is an NodeFinder with similar capabilities like PositionNodeFinder.

    todo:

    • [ ] workaround for https://github.com/python/cpython/issues/100537
    • [ ] integrate node verification ( create Verifier which can be used for python 3.11 and 3.10)
    • expand to:
      • [ ] 3.9
      • [ ] 3.8
      • [ ] 3.7
      • [ ] 3.6 (if possible)
    opened by 15r10nk 0
  • parent pointers in AST nodes hurt deepcopy performance

    parent pointers in AST nodes hurt deepcopy performance

    Obligatory "I'm a huge fan of your work".

    Background: I maintain a library called ipyflow, which uses another library I maintain called pyccolo. I noticed that on ipython >= 8.0, which uses executing for better stack traces, ipyflow would have really bad performance regressions the first time after a cell throws some exception.

    Eventually I traced it to pyccolo's use of copy.deepcopy in a few places -- performance was bad because the parent pointers that executing added to AST nodes were causing deepcopy to do a lot of extra unnecessary work.

    I ended up working around it on the pyccolo side, but I figured you may be interested in this for other libraries that may want to use executing and get surprised when deepcopy has bad perf. The way we maintain deepcopy-ability in pyccolo is to maintain a mapping from id(node) to parent for parent pointers, which avoids setting an attribute on the AST node directly.

    Thanks for this great library!

    opened by smacke 0
  • Include new library files in coverage measurement

    Include new library files in coverage measurement

    Currently GHA has coverage run --include=executing/executing.py <run tests> which doesn't measure coverage in the recently added files in the executing folder. The original motivation was lines in __init__.py that would never be covered, but that should be handled by a # pragma: no cover comment or something.

    opened by alexmojaki 3
Releases(v0.5.4)
Owner
Alex Hall
Python metaprogrammer
Alex Hall
🥦 Send and receive nano with 2 simple functions

easy_nano Send and receive nano (without having to understand the nano protocol).

1 Feb 14, 2022
A lighweight screen color picker tool

tkpick A lighweigt screen color picker tool Availability Only GNU/Linux 🐧 Installing Install via pip (No auto-update): [sudo] pip install tkpick Usa

Adil Gürbüz 7 Aug 30, 2021
STAC in Jupyter Notebooks

stac-nb STAC in Jupyter Notebooks Install pip install stac-nb Usage To use stac-nb in a project, start Jupyter Lab (jupyter lab), create a new noteboo

Darren Wiens 32 Oct 04, 2022
jmespath.rs Python binding

rjmespath-py jmespath.rs Python binding.

messense 3 Dec 14, 2022
Create a simple program by applying the use of class

TUGAS PRAKTIKUM 8 💻 Nama : Achmad Mahfud NIM : 312110520 Kelas : TI.21.C5 Perintah : Buat program sederhana dengan mengaplikasikan pengguna

Achmad Mahfud 1 Dec 23, 2021
Animations made using manim-ce

ManimCE Animations Animations made using manim-ce The code turned out to be a bit complicated than expected.. It can be greatly simplified, will work

sparshg 17 Jan 06, 2023
Tool for running a high throughput data ingestion/transformation workload with MongoDB

Mongo Mangler The mongo-mangler tool is a lightweight Python utility, which you can run from a low-powered machine to execute a high throughput data i

Paul Done 9 Jan 02, 2023
A repository containing an introduction to Panel made to be support videos and talks.

👍 Awesome Panel - Introduction to Panel THIS REPO IS WORK IN PROGRESS. PRE-ALPHA Panel is a very powerful framework for exploratory data analysis and

Marc Skov Madsen 51 Nov 17, 2022
python scripts - mostly automation scripts

python python scripts - mostly automation scripts You can set your environment in various ways bash #!/bin/bash python - locally on remote host #!/bi

Enyi 1 Jan 05, 2022
A tool to determine optimal projects for Gridcoin crunchers. Maximize your magnitude!

FindTheMag FindTheMag helps optimize your BOINC client for Gridcoin mining. You can group BOINC projects into two groups: "preferred" projects and "mi

7 Oct 04, 2022
Python bindings for the Plex API.

Python-PlexAPI Overview Unofficial Python bindings for the Plex API. Our goal is to match all capabilities of the official Plex Web Client. A few of t

Michael Shepanski 931 Jan 07, 2023
An account generator for guilded.gg that I made a while back and decided to bring back up

An account generator for guilded.gg that I made a while back and decided to bring back up

8 Nov 17, 2022
From "fixed RAnDom CRashes" to "[FIX] Fixed random crashes."

Clean Commit From fixed RAnDom CRashes to [FIX] Fixed random crashes. Clean commit helps you by auto-formating your commits to make your repos better

Mathias 3 Dec 26, 2021
This project intends to take the user's CEP (brazilian adress code) and return the local in which the CEP is placed.

This project aims to simply return the CEP's (the brazilian resident adress code) User of the application. The project uses a request and passes on to

Daniel Soares Saldanha 4 Nov 17, 2021
This is an implementation of PEP 557, Data Classes.

This is an implementation of PEP 557, Data Classes. It is a backport for Python 3.6. Because dataclasses will be included in Python 3.7, any discussio

Eric V. Smith 561 Dec 06, 2022
En este repositorio pondré archivos graciositos de python que hago de vez en cuando

🐍 Apuntes de python 🐍 ¿Quién soy? 👽 Saludos,mi nombre es Carlos Lara. Pero mi nickname en internet es Hercules Kan. Soy un programador autodidacta

Carlos E. Lara 3 Nov 16, 2021
Python for Microscopists and other image processing enthusiasts

The YouTube channel associated with this code walks you through the entire process of learning to code in Python; all the way from basics to advanced machine learning and deep learning.

Dr. Sreenivas Bhattiprolu 2.3k Jan 01, 2023
Transform Python source code into it's most compact representation

Python Minifier Transforms Python source code into it's most compact representation. Try it out! python-minifier currently supports Python 2.7 and Pyt

Daniel Flook 403 Jan 02, 2023
Multtable is a collection of multiplication table generators in various languages.

Multtable Multtable is a collection of multiplication table generators in various languages. This project was created as a joke based on one of my bro

pollen__ 7 Mar 05, 2022
Roman numeral conversion with python

Roman numeral conversion Discipline: Programming Languages Student: Paulo Henrique Diniz de Lima Alencar. Language: Python Description Responsible for

Paulo Alencar 1 Jul 11, 2022