Yet Another Python Profiler, but this time thread&coroutine&greenlet aware.

Overview

yappi

Yappi

Yet Another Python Profiler, but this time thread&coroutine&greenlet aware.

Highlights

  • Fast: Yappi is fast. It is completely written in C and lots of love&care went into making it fast.
  • Unique: Yappi supports multithreaded, asyncio and gevent profiling. Tagging/filtering multiple profiler results has interesting use cases.
  • Intuitive: Profiler can be started/stopped and results can be obtained from any time and any thread.
  • Standards Complaint: Profiler results can be saved in callgrind or pstat formats.
  • Rich in Feature set: Profiler results can show either Wall Time or actual CPU Time and can be aggregated from different sessions. Various flags are defined for filtering and sorting profiler results.
  • Robust: Yappi had seen years of production usage.

Motivation

CPython standard distribution comes with three deterministic profilers. cProfile, Profile and hotshot. cProfile is implemented as a C module based on lsprof, Profile is in pure Python and hotshot can be seen as a small subset of a cProfile. The major issue is that all of these profilers lack support for multi-threaded programs and CPU time.

If you want to profile a multi-threaded application, you must give an entry point to these profilers and then maybe merge the outputs. None of these profilers are designed to work on long-running multi-threaded applications. It is also not possible to profile an application that start/stop/retrieve traces on the fly with these profilers.

Now fast forwarding to 2019: With the latest improvements on asyncio library and asynchronous frameworks, most of the current profilers lacks the ability to show correct wall/cpu time or even call count information per-coroutine. Thus we need a different kind of approach to profile asynchronous code. Yappi, with v1.2 introduces the concept of coroutine profiling. With coroutine-profiling, you should be able to profile correct wall/cpu time and call count of your coroutine. (including the time spent in context switches, too). You can see details here.

Installation

Can be installed via PyPI

$ pip install yappi

OR from the source directly.

$ pip install git+https://github.com/sumerc/yappi#egg=yappi

Examples

A simple example:

import yappi

def a():
    for _ in range(10000000):  # do something CPU heavy
        pass

yappi.set_clock_type("cpu") # Use set_clock_type("wall") for wall time
yappi.start()
a()

yappi.get_func_stats().print_all()
yappi.get_thread_stats().print_all()
'''

Clock type: CPU
Ordered by: totaltime, desc

name                                  ncall  tsub      ttot      tavg      
doc.py:5 a                            1      0.117907  0.117907  0.117907

name           id     tid              ttot      scnt        
_MainThread    0      139867147315008  0.118297  1
'''

Profile a multithreaded application:

You can profile a multithreaded application via Yappi and can easily retrieve per-thread profile information by filtering on ctx_id with get_func_stats API.

import yappi
import time
import threading

_NTHREAD = 3


def _work(n):
    time.sleep(n * 0.1)


yappi.start()

threads = []
# generate _NTHREAD threads
for i in range(_NTHREAD):
    t = threading.Thread(target=_work, args=(i + 1, ))
    t.start()
    threads.append(t)
# wait all threads to finish
for t in threads:
    t.join()

yappi.stop()

# retrieve thread stats by their thread id (given by yappi)
threads = yappi.get_thread_stats()
for thread in threads:
    print(
        "Function stats for (%s) (%d)" % (thread.name, thread.id)
    )  # it is the Thread.__class__.__name__
    yappi.get_func_stats(ctx_id=thread.id).print_all()
'''
Function stats for (Thread) (3)

name                                  ncall  tsub      ttot      tavg
..hon3.7/threading.py:859 Thread.run  1      0.000017  0.000062  0.000062
doc3.py:8 _work                       1      0.000012  0.000045  0.000045

Function stats for (Thread) (2)

name                                  ncall  tsub      ttot      tavg
..hon3.7/threading.py:859 Thread.run  1      0.000017  0.000065  0.000065
doc3.py:8 _work                       1      0.000010  0.000048  0.000048


Function stats for (Thread) (1)

name                                  ncall  tsub      ttot      tavg
..hon3.7/threading.py:859 Thread.run  1      0.000010  0.000043  0.000043
doc3.py:8 _work                       1      0.000006  0.000033  0.000033
'''

Different ways to filter/sort stats:

You can use filter_callback on get_func_stats API to filter on functions, modules or whatever available in YFuncStat object.

import package_a
import yappi
import sys

def a():
    pass

def b():
    pass

yappi.start()
a()
b()
package_a.a()
yappi.stop()

# filter by module object
current_module = sys.modules[__name__]
stats = yappi.get_func_stats(
    filter_callback=lambda x: yappi.module_matches(x, [current_module])
)  # x is a yappi.YFuncStat object
stats.sort("name", "desc").print_all()
'''
Clock type: CPU
Ordered by: name, desc

name                                  ncall  tsub      ttot      tavg
doc2.py:10 b                          1      0.000001  0.000001  0.000001
doc2.py:6 a                           1      0.000001  0.000001  0.000001
'''

# filter by function object
stats = yappi.get_func_stats(
    filter_callback=lambda x: yappi.func_matches(x, [a, b])
).print_all()
'''
name                                  ncall  tsub      ttot      tavg
doc2.py:6 a                           1      0.000001  0.000001  0.000001
doc2.py:10 b                          1      0.000001  0.000001  0.000001
'''

# filter by module name
stats = yappi.get_func_stats(filter_callback=lambda x: 'package_a' in x.module
                             ).print_all()
'''
name                                  ncall  tsub      ttot      tavg
package_a/__init__.py:1 a             1      0.000001  0.000001  0.000001
'''

# filter by function name
stats = yappi.get_func_stats(filter_callback=lambda x: 'a' in x.name
                             ).print_all()
'''
name                                  ncall  tsub      ttot      tavg
doc2.py:6 a                           1      0.000001  0.000001  0.000001
package_a/__init__.py:1 a             1      0.000001  0.000001  0.000001
'''

Profile an asyncio application:

You can see that coroutine wall-time's are correctly profiled.

import asyncio
import yappi

async def foo():
    await asyncio.sleep(1.0)
    await baz()
    await asyncio.sleep(0.5)

async def bar():
    await asyncio.sleep(2.0)

async def baz():
    await asyncio.sleep(1.0)

yappi.set_clock_type("WALL")
with yappi.run():
    asyncio.run(foo())
    asyncio.run(bar())
yappi.get_func_stats().print_all()
'''
Clock type: WALL
Ordered by: totaltime, desc

name                                  ncall  tsub      ttot      tavg      
doc4.py:5 foo                         1      0.000030  2.503808  2.503808
doc4.py:11 bar                        1      0.000012  2.002492  2.002492
doc4.py:15 baz                        1      0.000013  1.001397  1.001397
'''

Profile a gevent application:

You can use yappi to profile greenlet applications now!

import yappi
from greenlet import greenlet
import time

class GreenletA(greenlet):
    def run(self):
        time.sleep(1)

yappi.set_context_backend("greenlet")
yappi.set_clock_type("wall")

yappi.start(builtins=True)
a = GreenletA()
a.switch()
yappi.stop()

yappi.get_func_stats().print_all()
'''
name                                  ncall  tsub      ttot      tavg
tests/test_random.py:6 GreenletA.run  1      0.000007  1.000494  1.000494
time.sleep                            1      1.000487  1.000487  1.000487
'''

Documentation

Related Talks

Special thanks to A.Jesse Jiryu Davis:

PyCharm Integration

Yappi is the default profiler in PyCharm. If you have Yappi installed, PyCharm will use it. See the official documentation for more details.

Issues
  • Coroutine-aware wall-time profiling

    Coroutine-aware wall-time profiling

    For the sake of profiling ASGI apps, it would be really helpful if there was a way to make use of python 3.7's contextvars while filtering via yappi.get_func_stats. If yappi supported this, you could get per-request-response-cycle profiling through an ASGI middleware.

    I'm not sure if the necessary functionality already exists or not, but wanted to open an issue in case anyone else has thought about this, or in case someone is aware of an existing way to accomplish this.

    See https://github.com/tiangolo/fastapi/issues/701 for more context on how we'd like to use this.

    1.2 
    opened by dmontagu 46
  • Make yappi grok greenlets

    Make yappi grok greenlets

    Proposal to make yappi grok greenlets

    This PR is a proposal to make yappi understand greenlets. The change allows yappi to map each greenlet to a unique execution context. Further, it also adjusts CPU stats measured to account for interleaving greenlet execution on one thread.

    Changes required

    The following changes are required by the fix:

    • ensure _profile_thread is called only from the target thread to be profiled.

      _profile_thread invokes the _current_context_id operation which retrieves the execution context of a thread.

      Using _current_context_id to retrieve the context information from another thread is not guaranteed to work and is hard to get right when yappi is configured with set_context_id_callback because the callback is not passed the information about what thread it has to fetch information for.

      It is also to hard to achieve the above when using greenlets because there is no obvious way to retrieve the greenlet currently executing on another thread. i.e there is no simple way to retrieve the greenlet currently executing on threadA from threadB.

      To get around this, this change guarantees that _profile_thread and _ensure_thread_profiled is only called from the target thread by removing their invocations from the 'start' sequence when 'multithreading' is enabled and delaying it. This is done by attaching a separate profiler func callback (profile_event and profile_event_c) to the threads. These functions first run _ensure_thread_profiled in the target thread context and change the profiler func callback to _yapp_callback. This way the work done by the start sequence is reduced and _ensure_thread_profiled is delayed to the first time a callback is invoked on the target thread.

    • add set_ctx_backend and allow specifying threading backend as greenlets as suggested by @sumerc

    • modify _current_context_id to identify the greenlet currently being executed when backend is greenlets. This is done the same way as it is for threads, except here we use the dictionary of the greenlet rather than that of the thread.

    • account for interleaving greenlet executions on the same native thread

      greenlet is a co-operative multitasking framework that introduces user controlled threads called greenlets. Each native python thread can run a number of greenlets. Greenlets on the same thread co-operatively switch between one another using greenlet.switch. it is important to note that a greenlet can only execute on the thread it was spawned on. As a result, greenlets spawned on one thread can only switch between each other.

      Greenlets do not play well with yappi. Yappi uses thread local stats to track the CPU usage of each frame. It tracks the usage at the entry of each function and at the exit of each function and subtracts the values to get the total clock cycles spent in that frame. This does not work well with greenlets. Let's take an example of two greenlets A and B and view the following sequence of events.

      • greenletA is currently running
      • greenletA enters function foo
      • greenletA co-operatively switches to greenletB
      • greenletB consumes 1 second of CPU
      • greenletB switches over back to greenletA
      • greenletA exits function foo

      The CPU usage of function foo will be 1 second more than it should be because greenletB's execution is interleaved.

      To account for this we have to track the context switches and adjust stats. The solution implemented in this change is a derivative of https://github.com/ajdavis/GreenletProfiler with some modifications to account for multiple native threads

      To track context switches:

      • declare a new thread local variable tl_prev_ctx. This needs to be thread local since multiple threads can exist. Greenlet context switches should be tracked at a thread level rather than at a global level.
      • tl_prev_ctx is initialised in _ensure_thread_profiled. It is initialised to the greenlet currently running on that thread
      • use tl_prev_ctx in _yapp_callback to determine if the context changed since the last time it was set.

      To adjust stats, if a switch is noticed from prev->current, then:

      • pause prev. Store the time it was paused under the paused_at field of the context associated with prev
      • resume the current greenlet. use the paused_at field and the current value of tickcount to determine how long the greenlet was paused for. Since a greenlet can only execute on one thread, subtracting these values is an accurate measure.

    NOTE: Another way to track context switches is by using greenlet.settrace (see https://greenlet.readthedocs.io/en/latest/). However installing this trace function for each native thread and uninstalling it did not seem easy to implement.

    What is remaining?

    This is not a complete fix, here's what is remaining:

    • make this work for wall clock time or document limitations of wall clock time measurement
    • make declaration of tl_prev_ctx platform independent. __thread is not portable to windows AFAIK
    • fix build system to install greenlet before compilation of yappi C extension. greenlet.h is required by yappi.
    • enumerate and add more test cases
    • there are a few TODOs added within the code. They are to be complete before closing. Some of them require discussion with @sumerc for a better understanding.

    Tests performed

    • The tests provided under https://github.com/sumerc/yappi/pull/53 have been modified to test correct CPU usage measurement
    • The script provided by @sumerc under https://github.com/sumerc/yappi/pull/53 was run. Here are the results:
    import gevent
    import yappi
    
    def burn_cpu(sec):
        t0 = yappi.get_clock_time()
        elapsed = 0
        while (elapsed < sec):
            for _ in range(1000):
                pass
            elapsed = yappi.get_clock_time() - t0
    
    
    def foo():
        burn_cpu(0.1)
        gevent.sleep(1.0)
    
    
    def bar():
        burn_cpu(0.1)
        gevent.sleep(1.0)
    
    
    #yappi.set_clock_type("wall")
    yappi.set_ctx_backend("greenlet")
    yappi.start()
    g1 = gevent.spawn(foo)
    g2 = gevent.spawn(bar)
    gevent.wait([g1, g2])
    
    yappi.stop()
    yappi.get_func_stats(
        filter_callback=lambda x: yappi.func_matches(x, [foo, bar])
    ).print_all()
    
    Clock type: CPU
    Ordered by: totaltime, desc
    
    name                                  ncall  tsub      ttot      tavg
    rand_test.py:19 bar                   1      0.000032  0.100202  0.100202
    rand_test.py:14 foo                   1      0.000040  0.100171  0.100171
    
    • automated tests were executed with both python2.7 and python3.8
    opened by Suhail-MOHD 40
  • Feature request: Filter stats by function descriptor

    Feature request: Filter stats by function descriptor

    As part of an application profiler plugin project for the ASGI protocol, where users can just pass a list of functions to profile, I was using a simple yappi.get_func_stats({"name": somefunction.__qualname__, "tag": ctx_tag}) filter, thinking it would be enough to unambigously select those functions. I later found out that __qualname__ is only unambiguous within the scope of a module, so I should in all likelyhood specify somefunction.__module__ as part of the filter as well, but that got me to look at the code yappi uses to identify the functions being called:

    https://github.com/sumerc/yappi/blob/6c97f55ae69a0979bdea8e557510a5d349da340c/yappi/_yappi.c#L546-L567

    It uses ml_name as the function name, which apparently corresponds to __name__, not __qualname__ like I initially figured yappi would use, and has a few edge-cases for both function and module name that could make it difficult/unreliable to build the correct filter automatically.

    Shouldn't yappi have some kind of filter_from_function_descriptor function to generate a detailled filter from a function descriptor automatically, instead of leaving users guessing as to whether they're accidentally profiling other functions with the same name as the one they want?

    opened by sm-Fifteen 26
  • Add async profiling support for gevent

    Add async profiling support for gevent

    First of all, thanks a lot for this project, yappi is really the best Python profiler !

    In my projects I use gevent extensively. I was full of hope when I found this blog article:

    https://emptysqua.re/blog/greenletprofiler/

    Someone made a greenlet profiler a few years ago on top of yappi...

    But the project is Python 2, and it was coming with a modified, bundled version of yappi.

    Now that yappi supports coroutines (with asyncio, unfortunately), could you please give me some guidance how to redo what was done with greenletprofiler ?

    I would be happy to contribute to yappi with gevent support, but I need some help - my C++ skills are rusty and I don't know the code.

    I saw in this discussion https://github.com/sumerc/yappi/issues/21 that it was something you thought about once... But I am afraid set_ctx_backend() was finally not implemented ?

    Thanks a lot

    opened by mguijarr 11
  • Add basic contextvar-based test of asyncio

    Add basic contextvar-based test of asyncio

    I'm not really sure how you want to handle the fact that this requires python 3.7.

    Figured I'd open this to share; feel free to scavenge it into the coroutine-profiling branch in your own way, or to provide me with some guidance about how you'd like it refactored. (No pressure to merge this PR now or ever, but I'm happy to add more if desired.)

    Also, let me know if there are additional tests you'd like me to add.

    opened by dmontagu 11
  • BufferError when using memoryview

    BufferError when using memoryview

    I noticed this when trying to use yappi with tornado. Here is a simple repro:

    import yappi
    yappi.start()
    
    
    def test_mem():
        buf = bytearray()
        buf += b't' * 200
        view = memoryview(buf)[10:].tobytes()
        del buf[:10]
        return view
    
    
    if __name__ == "__main__":
        test_mem()
    

    Without yappi, the code completes successfully. With yappi I get the following error:

    Traceback (most recent call last):
      File "yappi_repro.py", line 14, in <module>
        test_mem()
      File "yappi_repro.py", line 9, in test_mem
        del buf[:10]
    BufferError: Existing exports of data: object cannot be re-sized
    

    Reproducible on python 3.6, 3.7 and 3.8. I have not tried python 2.7

    bug 
    opened by coldeasy 10
  • Internal Error 15

    Internal Error 15

    Good day,

    I'm author of pptop, which has a plugin for yappi profiler. After updating to 1.2.1, Yappi prints a lot of "Internal Error 15" messages when working. I believe the issue is because pptop plugin starts yappi from the own thread, however everything seems to work fine except this message.

    p.s. it would be also nice to have an ability obtaining data fields from get_func_stats() by keywords, as I see some of their indexes are changed from version to version.

    1.2 
    opened by divi255 7
  • Feature request: Filter stats by package

    Feature request: Filter stats by package

    Hi there,

    It would be lovely to filter stats by package. Because you're using PyObject_RichCompareBool as part of the filter it's already possible to achieve this with a helper object. Would you be interested in integrating this as a feature with a stable API?

    My current implementation is:

    import dataclasses
    import importlib
    import os
    
    @dataclasses.dataclass
    class PackageModule:
        package: str
        
        def __post_init__(self):
            mod = importlib.import_module(self.package)
            self.fn = mod.__file__
            if self.fn.endswith("__init__.py"):
                self.fn = os.path.dirname(self.fn)
        
        def __eq__(self, other):
            return other.startswith(self.fn)
            
    
    yappi.get_func_stats(filter={"modname": PackageModule("apd.aggregation")}).print_all()
    

    There are caveats to this, mainly that it requires the module to be importable, and not undesirable to import (such as potential import-time side-effects), but I think it improves the usability a fair bit.

    What do you think? If you're interested I'm happy to put together a PR.

    Matt

    opened by MatthewWilkes 6
  • Error installing through pip on windows

    Error installing through pip on windows

    I tried to install yappi and got this error. I use Windows 10 1909 and Python 3.7. I cloned the repository from the github and removed the slash symbol from https://github.com/sumerc/yappi/blob/master/setup.py#L82. Then I installed yappi via python setup.py install

    Collecting yappi
      Using cached https://files.pythonhosted.org/packages/90/8e/ac718e8ffaffb7c92c09eb2cec33557f8f1b3ec32af5b7599a993a7700c8/yappi-1.2.1.tar.gz
        Complete output from command python setup.py egg_info:
        running egg_info
        creating pip-egg-info\yappi.egg-info
        writing pip-egg-info\yappi.egg-info\PKG-INFO
        writing dependency_links to pip-egg-info\yappi.egg-info\dependency_links.txt
        writing entry points to pip-egg-info\yappi.egg-info\entry_points.txt
        writing top-level names to pip-egg-info\yappi.egg-info\top_level.txt
        writing manifest file 'pip-egg-info\yappi.egg-info\SOURCES.txt'
        Traceback (most recent call last):
          File "<string>", line 1, in <module>
          File "C:\Users\dimam\AppData\Local\Temp\pip-install-b6ze0_h7\yappi\setup.py", line 95, in <module>
            url=HOMEPAGE,
          File "C:\Users\dimam\Desktop\Programming\exchange_bot\venv\lib\site-packages\setuptools\__init__.py", line 145, in setup
            return distutils.core.setup(**attrs)
          File "C:\Users\dimam\AppData\Local\Programs\Python\Python37\lib\distutils\core.py", line 148, in setup
            dist.run_commands()
          File "C:\Users\dimam\AppData\Local\Programs\Python\Python37\lib\distutils\dist.py", line 966, in run_commands
            self.run_command(cmd)
          File "C:\Users\dimam\AppData\Local\Programs\Python\Python37\lib\distutils\dist.py", line 985, in run_command
            cmd_obj.run()
          File "C:\Users\dimam\Desktop\Programming\exchange_bot\venv\lib\site-packages\setuptools\command\egg_info.py", line 296, in run
            self.find_sources()
          File "C:\Users\dimam\Desktop\Programming\exchange_bot\venv\lib\site-packages\setuptools\command\egg_info.py", line 303, in find_sources
            mm.run()
          File "C:\Users\dimam\Desktop\Programming\exchange_bot\venv\lib\site-packages\setuptools\command\egg_info.py", line 534, in run
            self.add_defaults()
          File "C:\Users\dimam\Desktop\Programming\exchange_bot\venv\lib\site-packages\setuptools\command\egg_info.py", line 570, in add_defaults
            sdist.add_defaults(self)
          File "C:\Users\dimam\AppData\Local\Programs\Python\Python37\lib\distutils\command\sdist.py", line 226, in add_defaults
            self._add_defaults_python()
          File "C:\Users\dimam\Desktop\Programming\exchange_bot\venv\lib\site-packages\setuptools\command\sdist.py", line 127, in _add_defaults_python
            build_py = self.get_finalized_command('build_py')
          File "C:\Users\dimam\AppData\Local\Programs\Python\Python37\lib\distutils\cmd.py", line 299, in get_finalized_command
            cmd_obj.ensure_finalized()
          File "C:\Users\dimam\AppData\Local\Programs\Python\Python37\lib\distutils\cmd.py", line 107, in ensure_finalized
            self.finalize_options()
          File "C:\Users\dimam\Desktop\Programming\exchange_bot\venv\lib\site-packages\setuptools\command\build_py.py", line 34, in finalize_options
            orig.build_py.finalize_options(self)
          File "C:\Users\dimam\AppData\Local\Programs\Python\Python37\lib\distutils\command\build_py.py", line 55, in finalize_options
            self.package_dir[name] = convert_path(path)
          File "C:\Users\dimam\AppData\Local\Programs\Python\Python37\lib\distutils\util.py", line 112, in convert_path
            raise ValueError("path '%s' cannot end with '/'" % pathname)
        ValueError: path 'yappi/' cannot end with '/'
    
        ----------------------------------------
    Command "python setup.py egg_info" failed with error code 1 in C:\Users\dimam\AppData\Local\Temp\pip-install-b6ze0_h7\yappi\
    
    
    opened by klaipher 6
  • Current release 1.3.0 doesn't support python 3.10

    Current release 1.3.0 doesn't support python 3.10

    Python 3.10 will be included in Fedora 35. To make that update smoother, we're building Fedora packages with early pre-releases of Python 3.10 and yappi 1.3.0 is failing to build.

    https://bugzilla.redhat.com/show_bug.cgi?id=1899557

    I see some patches in master branch to support python 3.10 but it's not included in any release. May you create a new release which includes the required patches?

    release 
    opened by amoralej 5
  • Bug when installed with pip

    Bug when installed with pip

    When yappi is installed with pip, it crashes with the following error: image

    TEXT VERSION: Collecting yappi Using cached https://files.pythonhosted.org/packages/d2/92/7cd637a19fa2a10c0e55a44f8b36bcb83f0e1943ba8f1fb5edb15c819f2e/yappi-1.0.tar.gz Installing collected packages: yappi Running setup.py install for yappi ... error ERROR: Command errored out with exit status 1: command: 'c:\users\58394\appdata\local\programs\python\python37\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\58394\AppData\Local\Temp\pip-install-zsdo72o2\yappi\setup.py'"'"'; file='"'"'C:\Users\58394\AppData\Local\Temp\pip-install-zsdo72o2\yappi\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record 'C:\Users\58394\AppData\Local\Temp\pip-record-1uijfgsm\install-record.txt' --single-version-externally-managed --compile cwd: C:\Users\58394\AppData\Local\Temp\pip-install-zsdo72o2\yappi
    Complete output (9 lines): running install running build running build_py creating build creating build\lib.win-amd64-3.7 copying yappi.py -> build\lib.win-amd64-3.7 running build_ext building '_yappi' extension error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": https://visualstudio.microsoft.com/downloads/ ---------------------------------------- ERROR: Command errored out with exit status 1: 'c:\users\58394\appdata\local\programs\python\python37\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\58394\AppData\Local\Temp\pip-install-zsdo72o2\yappi\setup.py'"'"'; file='"'"'C:\Users\58394\AppData\Local\Temp\pip-install-zsdo72o2\yappi\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record 'C:\Users\58394\AppData\Local\Temp\pip-record-1uijfgsm\install-record.txt' --single-version-externally-managed --compile Check the logs for full command output.

    opened by xkcdjerry 5
  • suggestion: add a column showing max execution time for a single call

    suggestion: add a column showing max execution time for a single call

    This tool is really useful but it would be great if it also showed what the highest recorded execution time was for a measured call when a call was made multiple times. This would basically point to bottlenecks. It has the average and total at the moment. (I tried already to modify the code to add this feature before posting this but I don't understand the code well enough yet.)

    opened by robertsdotpm 0
  • yappi fails to build with python 3.11

    yappi fails to build with python 3.11

    When trying to build yappi with 3.11, it fails with following trace:

    /tmp/timer_createf6naxck_.c: In function ‘main’: /tmp/timer_createf6naxck_.c:2:5: warning: implicit declaration of function ‘timer_create’ [-Wimplicit-function-declaration] 2 | timer_create(); | ^~~~~~~~~~~~ running build running build_py running build_ext building '_yappi' extension gcc -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DLIB_RT_AVAILABLE=1 -I/usr/include/python3.11 -c yappi/_yappi.c -o build/temp.linux-x86_64-3.11/yappi/_yappi.o yappi/_yappi.c: In function ‘IS_SUSPENDED’: yappi/_yappi.c:220:18: error: invalid use of incomplete typedef ‘PyFrameObject’ {aka ‘struct _frame’} 220 | return (frame->f_state == FRAME_SUSPENDED); | ^~ yappi/_yappi.c:220:31: error: ‘FRAME_SUSPENDED’ undeclared (first use in this function); did you mean ‘IS_SUSPENDED’? 220 | return (frame->f_state == FRAME_SUSPENDED); | ^~~~~~~~~~~~~~~ | IS_SUSPENDED yappi/_yappi.c:220:31: note: each undeclared identifier is reported only once for each function it appears in yappi/_yappi.c: In function ‘IS_ASYNC’: yappi/_yappi.c:232:19: error: invalid use of incomplete typedef ‘PyFrameObject’ {aka ‘struct _frame’} 232 | result = frame->f_code->co_flags & CO_COROUTINE || | ^~ yappi/_yappi.c:233:14: error: invalid use of incomplete typedef ‘PyFrameObject’ {aka ‘struct _frame’} 233 | frame->f_code->co_flags & CO_ITERABLE_COROUTINE; | ^~ yappi/_yappi.c:236:29: error: invalid use of incomplete typedef ‘PyFrameObject’ {aka ‘struct _frame’} 236 | result = result || frame->f_code->co_flags & CO_ASYNC_GENERATOR; | ^~ yappi/_yappi.c: In function ‘_code2pit’: yappi/_yappi.c:653:16: error: invalid use of incomplete typedef ‘PyFrameObject’ {aka ‘struct _frame’} 653 | cobj = fobj->f_code; | ^~ yappi/_yappi.c:673:72: error: ‘PyCodeObject’ has no member named ‘co_varnames’; did you mean ‘co_names’? 673 | const char firstarg = PyStr_AS_CSTRING(PyTuple_GET_ITEM(cobj->co_varnames, 0)); | ^~~~~~~~~~~ yappi/_yappi.c:182:46: note: in definition of macro ‘PyStr_AS_CSTRING’ 182 | #define PyStr_AS_CSTRING(s) PyUnicode_AsUTF8(s) | ^ /usr/include/python3.11/cpython/tupleobject.h:18:33: note: in expansion of macro ‘_Py_CAST’ 18 | (assert(PyTuple_Check(op)), _Py_CAST(PyTupleObject, (op))) | ^~~~~~~~ /usr/include/python3.11/cpython/tupleobject.h:30:38: note: in expansion of macro ‘_PyTuple_CAST’ 30 | #define PyTuple_GET_ITEM(op, index) (_PyTuple_CAST(op)->ob_item[index]) | ^~~~~~~~~~~~~ yappi/_yappi.c:673:49: note: in expansion of macro ‘PyTuple_GET_ITEM’ 673 | const char *firstarg = PyStr_AS_CSTRING(PyTuple_GET_ITEM(cobj->co_varnames, 0)); | ^~~~~~~~~~~~~~~~ yappi/_yappi.c: In function ‘IS_SUSPENDED’: yappi/_yappi.c:224:1: warning: control reaches end of non-void function [-Wreturn-type] 224 | } | ^ error: command '/usr/bin/gcc' failed with exit code 1

    Errors seems related to documented changes in release notes:

    https://docs.python.org/3.11/whatsnew/3.11.html

    opened by amoralej 0
  • Tests are broken on Python 3.11 due to @asyncio.coroutine use

    Tests are broken on Python 3.11 due to @asyncio.coroutine use

    The @asyncio.coroutine decorator has been deprecated since Python 3.8, and is removed entirely in 3.11. This causes the tests to fail immediately:

    $ python3.11 run_tests.py 
    Traceback (most recent call last):
      File "/tmp/yappi/run_tests.py", line 29, in <module>
        test_suite = test_loader.loadTestsFromNames(tests)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/unittest/loader.py", line 220, in loadTestsFromNames
        suites = [self.loadTestsFromName(name, module) for name in names]
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/unittest/loader.py", line 220, in <listcomp>
        suites = [self.loadTestsFromName(name, module) for name in names]
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/unittest/loader.py", line 154, in loadTestsFromName
        module = __import__(module_name)
                 ^^^^^^^^^^^^^^^^^^^^^^^
      File "/tmp/yappi/tests/test_asyncio.py", line 8, in <module>
        @asyncio.coroutine
         ^^^^^^^^^^^^^^^^^
    AttributeError: module 'asyncio' has no attribute 'coroutine'. Did you mean: 'coroutines'?
    

    IIUC the replacement is async def, with await instead of yield from.

    opened by mgorny 0
  • Fastapi support

    Fastapi support

    I am new to yappi and fastapi, I want to use yappi to profile my fastapi app with uvicorn as WSGI server, I googled it for a while and also checked yappi official site, but failed to find a direct solution for this. Does anyone share to me how to use yappi in fastapi? Super thx.

    opened by fansonfong 2
  • Error building wheel for yappi

    Error building wheel for yappi

    Hey I got the following error while installing the yappi wheel

    ERROR: Command errored out with exit status 1:
       command: /mnt/c/Users/USER/Documents/gitprojects/cs-api-dashboard-crud/venv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-x9pq_pue/yappi/setup.py'"'"'; __file__='"'"'/tmp/pip-install-x9pq_pue/yappi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-8cv9dxrc
           cwd: /tmp/pip-install-x9pq_pue/yappi/
      Complete output (14 lines):
      unable to execute 'cc': No such file or directory
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.8
      copying yappi/yappi.py -> build/lib.linux-x86_64-3.8
      running build_ext
      building '_yappi' extension
      creating build/temp.linux-x86_64-3.8
      creating build/temp.linux-x86_64-3.8/yappi
      x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/mnt/c/Users/USER/Documents/gitprojects/cs-api-dashboard-crud/venv/include -I/usr/include/python3.8 -c yappi/_yappi.c -o build/temp.linux-x86_64-3.8/yappi/_yappi.o
      unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
      error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
      ----------------------------------------
      ERROR: Failed building wheel for yappi
      Running setup.py clean for yappi
    Failed to build yappi
    Installing collected packages: yappi
        Running setup.py install for yappi ... error
        ERROR: Command errored out with exit status 1:
         command: /mnt/c/Users/USER/Documents/gitprojects/cs-api-dashboard-crud/venv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-x9pq_pue/yappi/setup.py'"'"'; __file__='"'"'/tmp/pip-install-x9pq_pue/yappi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-bmoz0ey7/install-record.txt --single-version-externally-managed --compile --install-headers /mnt/c/Users/USER/Documents/gitprojects/cs-api-dashboard-crud/venv/include/site/python3.8/yappi
             cwd: /tmp/pip-install-x9pq_pue/yappi/
        Complete output (14 lines):
        unable to execute 'cc': No such file or directory
        running install
        running build
        running build_py
        creating build
        creating build/lib.linux-x86_64-3.8
        copying yappi/yappi.py -> build/lib.linux-x86_64-3.8
        running build_ext
        building '_yappi' extension
        creating build/temp.linux-x86_64-3.8
        creating build/temp.linux-x86_64-3.8/yappi
        x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/mnt/c/Users/USER/Documents/gitprojects/cs-api-dashboard-crud/venv/include -I/usr/include/python3.8 -c yappi/_yappi.c -o build/temp.linux-x86_64-3.8/yappi/_yappi.o
        unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
        error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
        ----------------------------------------
    ERROR: Command errored out with exit status 1: /mnt/c/Users/USER/Documents/gitprojects/cs-api-dashboard-crud/venv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-x9pq_pue/yappi/setup.py'"'"'; __file__='"'"'/tmp/pip-install-x9pq_pue/yappi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-bmoz0ey7/install-record.txt --single-version-externally-managed --compile --install-headers /mnt/c/Users/USER/Documents/gitprojects/cs-api-dashboard-crud/venv/include/site/python3.8/yappi Check the logs for full command output.
    
    opened by EKivutha 0
  • Callgrind format shows lot of unknown function

    Callgrind format shows lot of unknown function

    When saving the yappi profile output as callgrind format, qcachegrind shows a major chunk of time spent in unknown function with output on a console similar to this

    Loading "./file_name.profile" : 3808 :  "Undefined compressed file index 1887"
    Loading "./file_name.profile" : 3808 :  "Invalid specification of called file, setting to unknown"
    Loading "./file_name.profile" : 3809 :  "Undefined compressed function index 1887"
    Loading "./file_name.profile" : 3809 :  "Invalid called function, setting to unknown"
    

    In _save_as_CALLGRIND function, while adding the function definitions we only iterate over function stats

    for func_stat in self:
      file_ids += ['fl=(%d) %s' % (func_stat.index, func_stat.module)]
      func_ids += [
          'fn=(%d) %s %s:%s' % (
              func_stat.index, func_stat.name, func_stat.module,
              func_stat.lineno
          )
      ]
    

    Function/File definitions of some child functions were not specified in the callgrind format which was causing this issue.

    opened by saurabhkgp21 0
Owner
Sümer Cip
Senior Software Engineer @blackfireio
Sümer Cip
🚴 Call stack profiler for Python. Shows you why your code is slow!

pyinstrument Pyinstrument is a Python profiler. A profiler is a tool to help you 'optimize' your code - make it faster. It sounds obvious, but to get

Joe Rickerby 4.6k Jul 13, 2022
Visual profiler for Python

vprof vprof is a Python package providing rich and interactive visualizations for various Python program characteristics such as running time and memo

Nick Volynets 3.8k Jul 12, 2022
Scalene: a high-performance, high-precision CPU and memory profiler for Python

scalene: a high-performance CPU and memory profiler for Python by Emery Berger 中文版本 (Chinese version) About Scalene % pip install -U scalene Scalen

Emery Berger 88 Jul 12, 2022
Was an interactive continuous Python profiler.

☠ This project is not maintained anymore. We highly recommend switching to py-spy which provides better performance and usability. Profiling The profi

What! Studio 3k Jul 9, 2022
Tracy Profiler module for the Godot Engine

GodotTracy Tracy Profiler module for the Godot Engine git clone --recurse-submodules https://github.com/Pineapple/GodotTracy.git Copy godot_tracy fold

Pineapple Works 16 Mar 10, 2022
System monitor - A python-based real-time system monitoring tool

System monitor A python-based real-time system monitoring tool Screenshots Installation Run My project with these commands pip install -r requiremen

Sachit Yadav 4 Feb 11, 2022
Real-time metrics for nginx server

ngxtop - real-time metrics for nginx server (and others) ngxtop parses your nginx access log and outputs useful, top-like, metrics of your nginx serve

Binh Le 6.3k Jul 15, 2022
GoAccess is a real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser.

GoAccess What is it? GoAccess is an open source real-time web log analyzer and interactive viewer that runs in a terminal on *nix systems or through y

Gerardo O. 14.9k Jul 15, 2022
Development tool to measure, monitor and analyze the memory behavior of Python objects in a running Python application.

README for pympler Before installing Pympler, try it with your Python version: python setup.py try If any errors are reported, check whether your Pyt

null 936 Jul 16, 2022
Prometheus instrumentation library for Python applications

Prometheus Python Client The official Python 2 and 3 client for Prometheus. Three Step Demo One: Install the client: pip install prometheus-client Tw

Prometheus 2.8k Jul 11, 2022
Automatically monitor the evolving performance of Flask/Python web services.

Flask Monitoring Dashboard A dashboard for automatic monitoring of Flask web-services. Key Features • How to use • Live Demo • Feedback • Documentatio

null 633 Jul 15, 2022
Cross-platform lib for process and system monitoring in Python

Home Install Documentation Download Forum Blog Funding What's new Summary psutil (process and system utilities) is a cross-platform library for retrie

Giampaolo Rodola 8.5k Jul 15, 2022
Monitor Memory usage of Python code

Memory Profiler This is a python module for monitoring memory consumption of a process as well as line-by-line analysis of memory consumption for pyth

null 3.5k Jul 14, 2022
Line-by-line profiling for Python

line_profiler and kernprof NOTICE: This is the official line_profiler repository. The most recent version of line-profiler on pypi points to this repo

OpenPyUtils 1.3k Jul 19, 2022
Monitor Memory usage of Python code

Memory Profiler This is a python module for monitoring memory consumption of a process as well as line-by-line analysis of memory consumption for pyth

Fabian Pedregosa 76 Jun 24, 2022
pprofile + matplotlib = Python program profiled as an awesome heatmap!

pyheat Profilers are extremely helpful tools. They help us dig deep into code, find and understand performance bottlenecks. But sometimes we just want

Vishwas B Sharma 721 Jul 8, 2022
Discord bot-CTFD-Thread-Parser - Discord bot CTFD-Thread-Parser

Discord bot CTFD-Thread-Parser Description: This tools is used to create automat

null 15 Mar 22, 2022
🖍️This is a feature-complete clone of the awesome Chalk (JavaScript) library.

Terminal string styling done right This is a feature-complete clone of the awesome Chalk (JavaScript) library. All credits go to Sindre Sorhus. Highli

Fabian Keller 122 Jun 12, 2022
Django query profiler - one profiler to rule them all. Shows queries, detects N+1 and gives recommendations on how to resolve them

Django Query Profiler This is a query profiler for Django applications, for helping developers answer the question "My Django code/page/API is slow, H

Django Query Profiler 105 Jul 12, 2022
Django query profiler - one profiler to rule them all. Shows queries, detects N+1 and gives recommendations on how to resolve them

Django Query Profiler This is a query profiler for Django applications, for helping developers answer the question "My Django code/page/API is slow, H

Django Query Profiler 105 Jul 12, 2022
Coroutine-based concurrency library for Python

gevent Read the documentation online at http://www.gevent.org. Post issues on the bug tracker, discuss and ask open ended questions on the mailing lis

gevent 5.8k Jul 18, 2022
OpenTOTP is yet another time-based, one-time passwords (OTPs) generator/verifier inspired by RFC 6238.

OpenTOTP is yet another time-based, one-time passwords (OTPs) generator/verifier inspired by RFC 6238. It generates and validates OTPs based

null 1 Nov 15, 2021
Transcript-Extractor-Bot - Yet another Telegram Voice Recognition bot but using vosk and supports 20+ languages

transcript extractor Yet another Telegram Voice Recognition bot but using vosk a

null 6 Jun 27, 2022
Yet Another Time Series Model

Yet Another Timeseries Model (YATSM) master v0.6.x-maintenance Build Coverage Docs DOI | About Yet Another Timeseries Model (YATSM) is a Python packag

Chris Holden 60 May 12, 2022
Volt is yet another discord api wrapper for Python. It supports python 3.8 +

Volt Volt is yet another discord api wrapper for Python. It supports python 3.8 + How to install [Currently Not Supported.] pip install volt.py Speed

Minjun Kim (Lapis0875) 11 May 14, 2022
Yet Another Sequence Encoder - Encode sequences to vector of vector in python !

Yase Yet Another Sequence Encoder - encode sequences to vector of vectors in python ! Why Yase ? Yase enable you to encode any sequence which can be r

Pierre PACI 12 Aug 19, 2021
Yet another Python binding for fastText

pyfasttext Warning! pyfasttext is no longer maintained: use the official Python binding from the fastText repository: https://github.com/facebookresea

Vincent Rasneur 230 Jul 14, 2022
Yet another Python binding for fastText

pyfasttext Warning! pyfasttext is no longer maintained: use the official Python binding from the fastText repository: https://github.com/facebookresea

Vincent Rasneur 228 Feb 17, 2021