The Python agent for Apache SkyWalking

Last update: Dec 12, 2022

Overview

SkyWalking Python Agent

SkyWalking-Python: The Python Agent for Apache SkyWalking, which provides the native tracing abilities for Python project.

SkyWalking: an APM(application performance monitor) system, especially designed for microservices, cloud native and container-based (Docker, Kubernetes, Mesos) architectures.

Documentation

Official documentation
Blog about the Python Agent Profiling Feature

Installation Requirements

SkyWalking Python Agent requires SkyWalking 8.0+ and Python 3.6+.

If you would like to try out the latest features that are not released yet, please refer to this guide to build from sources.

Contributing

Before submitting a pull request or pushing a commit, please read our contributing and developer guide.

Contact Us

Submit an GitHub Issue by using [Python] as title prefix.
Mail list: [email protected]. Mail to [email protected], follow the reply to subscribe the mail list.
Join skywalking channel at Apache Slack. If the link is not working, find the latest one at Apache INFRA WIKI.
Twitter, ASFSkyWalking

License

Apache 2.0

Comments

feat: Add support for MeterReportService

Add support for MeterReportService

Behavior

This service runs along with the agent like other services and reports roughly every 20 seconds.

Syntactic sugar

Counter

c = Counter('c2', CounterMode.INCREMENT)
c.build()

# increase Counter c by the time the with-wrapped codes consumed
with c.create_timer():
    # some codes may consume a certain time

c = Counter('c3', CounterMode.INCREMENT)
c.build()

# increase Counter c by num once counter_decorator_test gets called
@Counter.increase(name='c3', num=2)
def counter_decorator_test():
    # some codes

c = Counter('c4', CounterMode.INCREMENT)
c.build()

# increase Counter c by the time counter_decorator_test consumed
@Counter.timer(name='c4')
def counter_decorator_test(s):
    # some codes may consume a certain time

Histogram

h = Histogram('h3', [i / 10 for i in range(10)])
h.build()

# Histogram h will record the time the with-wrapped codes consumed
with h.create_timer():
    # some codes may consume a certain time

h = Histogram('h2', [i / 10 for i in range(10)])
h.build()

# Histogram h will record the time histogram_decorator_test consumed
@Histogram.timer(name='h2')
def histogram_decorator_test(s):
    time.sleep(s)

Simple test

insert following codes below this line agent/__init__.py#L151

from skywalking.meter.gauge import Gauge
import gc

def generator_for_gc_g0():
    while(True):
        yield gc.get_stats()[0]['collected']

def generator_for_gc_g1():
    while(True):
        yield gc.get_stats()[1]['collected']

def generator_for_gc_g2():
    while(True):
        yield gc.get_stats()[2]['collected']


gc_g0_meter = Gauge("gc_g0", generator_for_gc_g0())
gc_g0_meter.build()

gc_g1_meter = Gauge("gc_g1", generator_for_gc_g1())
gc_g1_meter.build()

gc_g2_meter = Gauge("gc_g2", generator_for_gc_g2())
gc_g2_meter.build()

feature core

opened by jiang1997 19

fix: grpc timeout segment data loss

Please review this thoroughly because all I wanted to do was resend segments which failed to send due to a grpc timeout. But unexpectedly the behavior of grpc timeout changed, it doesn't time out anymore, the heartbeat seems to keep it alive now. I don't understand why this behavior changed so that is why I am requesting a thorough looking at this.
enhancement

opened by tom-pytel 18
Add http ignore by method
Added SW_HTTP_IGNORE_METHOD which will allow ignoring http operations by method (GET, POST, HEAD, OPTIONS, etc...)

Set default SW_AGENT_MAX_BUFFER_SIZE back to 10000 from 1000, which turns out to be too small for python (many unnecessarily dropped spans).

feature
opened by tom-pytel 14
feat: report metrics related to pvm

Metrics

CPU

total_cpu_utilization

process_cpu_utilization

cpu utilization of the current process

thread_active_count

number of threads invoked by the current process and its children

Memory

total_mem_utilization

process_mem_utilization

memory utilization of the current process

GC

gc_g0

number of objects of generation 0

gc_g1

number of objects of generation 1

gc_g2

number of objects of generation 2

gc_time

For now, MeterReportService is available with gRPC only. So does the e2e test. I will implement MeterReportService and enable e2e test for Kafka later.

Welcome any advice about naming.
feature core

opened by jiang1997 12
[Fix][Plugin] sw_flask general exceptions handled
sw_flask fix will handle errors like returning the wrong type from a handler or other internal errors.

Updated StackedSpan to track depth, and made depth variable instance instead of class level (this was a bug).

Changed how SpanContext decides when all spans finished to write Segment data, now counts span start / stops which should work better across different async scenarios.

Changed new_exit_span() with span.inject() to work simpler like the NodeJS agent, now plugins inject directly themselves if they need to.

Removed carrier from plugins which didn't actually use it.

bug plugin
opened by tom-pytel 12
Add plugin for bottle
[x] Add a test case for the new plugin

[x] Add a CHANGELOG entry for the new plugin

[x] Add a component id in the main repo

[x] Add a logo in the UI repo

[x] Rebuild the requirements.txt by running tools/env/build_requirements_(linux|windows).sh

[x] Rebuild the Plugins.md documentation by running make doc-gen

plugin
opened by jiang1997 11
Enable HTTP log reporting

This is a work in progress.

Now reporting logs in JSON through HTTP to http://oap/v3/logs does work.

But, I'm not sure if the oap/v3/logs endpoint is for such usage(It seems designed for fluent-bit batch reporting?). Reporting logs one by one through HTTP may not be ideal in terms of performance. I'm not sure whether the Java agent only implements gRPC reporter intentionally out of this reason.

Please advise.

Signed-off-by: YihaoChen [email protected]
feature

opened by Superskyyy 11
Feature: collect and report logs
This is a OSPP Summer 2021 project supervised by @Humbertzhang | apache/skywalking#7118

The feature implements optional log reporter functionalities in alignment with the SkyWalking Java agent.

Intercepts logs from Python logging module.

Reports logs via a new temporary gRPC channel(to be removed in the future).

Supports unformatted/ formatted logs with custom layout.

Supports custom logging level threshold.

Bumps up submodule to support log collection protocols.

Bumps up skywalking-eyes and adds a config entry to ignore .gitignore during license checks.

feature core
opened by Superskyyy 11
Introduce better coding style

This PR migrates existing usage of old-style string manipulation (%, .format and +) to f-strings for better clarity and some performance boost.

Adds stricter coding style.

Optimizes debug messages for performance.

Signed-off-by: Superskyyy [email protected]
enhancement chore

opened by Superskyyy 10
Add E2E coverage for Trace and Logging
Change Fastapi to FastAPI component name (sync with OAP change)

Add E2E coverage for Python agent (Trace and Logging), profiling is not tested for now.

Renamed a build variable to remove possible mix-ups with CLI environment Var.

Enhance plugin test to reuse images.

Combined CI to pass if changes are non-essential to agent and workflows.

Note: There's a flaky test due to a possible {{- contains }} issue in the Infra-e2e verifier, the logs seem to be queried in an unstable order by the CLI where later log often placed first in log list, combining with the contains bug, it becomes flaky.

A time.sleep() workaround makes sure the log arrives in clear order which passes the e2e for now.
test
opened by Superskyyy 9
[Enhancement] Async tasks should work 100%

These changes should address all remaining async-related problems not handled in #88. The spans list has been moved from an instance variable in SpanContext to a global task-local variable which allows "forking" it for new subtasks so that multiple concurrent async subtasks don't interfere with one another. spans is now a top-level module var because Python stipulates that ContextVar should be such and only one context exists at any given time so it is essentially a singleton anyway.
enhancement core

opened by tom-pytel 9
Enhance redis plugin to adapt Virtual Cache

Resolves: #10212
enhancement

opened by Jedore 2

Releases(v0.8.0)

v0.8.0(Jul 10, 2022)
What's Changed

spans correctly reference finished parents by @tom-pytel in https://github.com/apache/skywalking-python/pull/161

Refactor current Python agent docs to serve on SkyWalking official website by @Superskyyy in https://github.com/apache/skywalking-python/pull/162

fix broken url by @JaredTan95 in https://github.com/apache/skywalking-python/pull/163

Remove docs from main README.md - Add website doc links. by @Superskyyy in https://github.com/apache/skywalking-python/pull/164

Refactor SkyWalking Python to use the CLI for CI instead of legacy setup by @Superskyyy in https://github.com/apache/skywalking-python/pull/165

Remove places that mentions Python 3.5 support due to EOL by @Superskyyy in https://github.com/apache/skywalking-python/pull/166

Cleanup and Python 3.10 support by @Superskyyy in https://github.com/apache/skywalking-python/pull/167

filled out rest of psycopg 3 plugin by @tom-pytel in https://github.com/apache/skywalking-python/pull/168

Move flake configs all together by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/169

Enable faster CI by categorical parallelism by @Superskyyy in https://github.com/apache/skywalking-python/pull/170

Introduce better coding style by @Superskyyy in https://github.com/apache/skywalking-python/pull/171

Minimize exclude in lint by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/173

Introduce another set of flake8 extensions by @Superskyyy in https://github.com/apache/skywalking-python/pull/174

fix aiohttp outgoing request url by @tom-pytel in https://github.com/apache/skywalking-python/pull/175

bugfix: flask + nginx got KeyError: 'REMOTE_PORT' in sw_flask.py plugin by @VxCoder in https://github.com/apache/skywalking-python/pull/176

Add missing ending quote by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/177

updated deprecated configuration by @doddi in https://github.com/apache/skywalking-python/pull/179

Add plugin for mysqlclient by @katelei6 in https://github.com/apache/skywalking-python/pull/178

Fix sw-rabbitmq TypeError when there are no headers by @dcryans in https://github.com/apache/skywalking-python/pull/182

Fix agent bootup traceback not shown in sw-python CLI by @Superskyyy in https://github.com/apache/skywalking-python/pull/183

Add plugin for FastAPI by @katelei6 in https://github.com/apache/skywalking-python/pull/181

Support CI container logs by @Superskyyy in https://github.com/apache/skywalking-python/pull/185

Update mySQL plugin to support two different parameter keys. by @katelei6 in https://github.com/apache/skywalking-python/pull/186

add the code > 400 is error by @katelei6 in https://github.com/apache/skywalking-python/pull/187

Doc: add how to use with uwsgi by @arcosx in https://github.com/apache/skywalking-python/pull/188

Fix: right doc link for How To Use With uWSGI by @arcosx in https://github.com/apache/skywalking-python/pull/189

The local log stack depth is not truncated by @wzy960520 in https://github.com/apache/skywalking-python/pull/190

Revert "The local log stack depth is not truncated (#190)" by @Superskyyy in https://github.com/apache/skywalking-python/pull/191

Fix sw_formatter wrongly set cache that impacts user handlers by @Superskyyy in https://github.com/apache/skywalking-python/pull/192

Fix typo that cause failure in loading user sitecustomize.py by @Superskyyy in https://github.com/apache/skywalking-python/pull/193

Drop support for Flask 1.x due to EOL & Jinja2 issue by @Superskyyy in https://github.com/apache/skywalking-python/pull/195

Update agent docs, changelogs and PR template. by @Superskyyy in https://github.com/apache/skywalking-python/pull/197

Fix Python shown as UNKNOWN language in OAP by @Superskyyy in https://github.com/apache/skywalking-python/pull/194

Fix logging level not properly set according to config by @Superskyyy in https://github.com/apache/skywalking-python/pull/196

Fix the properties are not set correctly by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/198

Add E2E coverage for Trace and Logging by @Superskyyy in https://github.com/apache/skywalking-python/pull/199

Support log reporter safe mode by @Superskyyy in https://github.com/apache/skywalking-python/pull/200

Fix scheduled event fails changes CI by @Superskyyy in https://github.com/apache/skywalking-python/pull/201

Fix deadlink and CI on schedule by @Superskyyy in https://github.com/apache/skywalking-python/pull/203

Remove namespace to instance properties and add pid by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/205

Enhance Traceback depth default to 10 by @Superskyyy in https://github.com/apache/skywalking-python/pull/206

Unify the tag name with other agents by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/208

Improved ignore path regex by @tom-pytel in https://github.com/apache/skywalking-python/pull/210

Fix sw_psycopg2 register_type() by @tom-pytel in https://github.com/apache/skywalking-python/pull/211

Fix psycopg2 register_type() second arg default by @tom-pytel in https://github.com/apache/skywalking-python/pull/212

Add plugin doc check in CI by @JarvisG495 in https://github.com/apache/skywalking-python/pull/213

Fix typo in docker/Makefile by @jiang1997 in https://github.com/apache/skywalking-python/pull/216

Update UI repository link in PR template by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/217

Add plugin for bottle by @jiang1997 in https://github.com/apache/skywalking-python/pull/214

Draft release 0.8.0 by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/219

Migrate license checker to license-eye and adjust the release script by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/220

New Contributors

@JaredTan95 made their first contribution in https://github.com/apache/skywalking-python/pull/163

@VxCoder made their first contribution in https://github.com/apache/skywalking-python/pull/176

@doddi made their first contribution in https://github.com/apache/skywalking-python/pull/179

@katelei6 made their first contribution in https://github.com/apache/skywalking-python/pull/178

@dcryans made their first contribution in https://github.com/apache/skywalking-python/pull/182

@arcosx made their first contribution in https://github.com/apache/skywalking-python/pull/188

@wzy960520 made their first contribution in https://github.com/apache/skywalking-python/pull/190

@JarvisG495 made their first contribution in https://github.com/apache/skywalking-python/pull/213

@jiang1997 made their first contribution in https://github.com/apache/skywalking-python/pull/216

Full Changelog: https://github.com/apache/skywalking-python/compare/v0.7.0...v0.8.0
Source code(tar.gz)
Source code(zip)
v0.7.0(Sep 13, 2021)
Feature:

Support collecting and reporting logs to backend (#147)

Support profiling Python method level performance (#127

Add a new sw-python CLI that enables agent non-intrusive integration (#156)

Add exponential reconnection backoff strategy when OAP is down (#157)

Support ignoring traces by http method (#143)

NoopSpan on queue full, propagation downstream (#141)

Support agent namespace. (#126)

Support secure connection option for GRPC and HTTP (#134)

Plugins:

Add Falcon Plugin (#146)

Update sw_pymongo.py to be compatible with cluster mode (#150)

Add Python celery plugin (#125)

Support tornado5+ and tornado6+ (#119)

Fixes:

Remove HTTP basic auth credentials from log, stacktrace, segment (#152)

Fix @trace decorator not work (#136)

Fix grpc disconnect, add SW_AGENT_MAX_BUFFER_SIZE to control buffer queue size (#138)

Others:

Chore: bump up requests version to avoid license issue (#142)

Fix module wrapt as normal install dependency (#123)

Explicit component inheritance (#132)

Provide dockerfile & images for easy integration in containerized scenarios (#159)

Source code(tar.gz)
Source code(zip)
v0.6.0(Mar 31, 2021)
Fixes:

Segment data loss when gRPC timing out. (#116)

sw_tornado plugin async handler status set correctly. (#115)

sw_pymysql error when connection haven't db. (#113)

Source code(tar.gz)
Source code(zip)
v0.5.0(Dec 31, 2020)
New plugins

Pyramid Plugin (#102)

AioHttp Plugin (#101)

Sanic Plugin (#91)

API and enhancements

@trace decorator supports async functions

Supports async task context

Optimized path trace ignore

Moved exception check to Span.__exit__

Moved Method & Url tags before requests

Fixes:

BaseExceptions not recorded as errors

Allow pending data to send before exit

sw_flask general exceptions handled

Make skywalking logging Non-global

Chores and tests

Make tests really run on specified Python version

Deprecate 3.5 as it's EOL

Source code(tar.gz)
Source code(zip)
v0.4.0(Nov 24, 2020)
Feature: Support Kafka reporter protocol (#74)

BugFix: Move generated packages into skywalking namespace to avoid conflicts (#72)

BugFix: Agent cannot reconnect after server is down (#79)

Test: Mitigate unsafe yaml loading (#76)

Source code(tar.gz)
Source code(zip)
v0.3.0(Aug 28, 2020)
New plugins

Urllib3 Plugin (#69)

Elasticsearch Plugin (#64)

PyMongo Plugin (#60)

Rabbitmq Plugin (#53)

Make plugin compatible with Django (#52)

API

Add process propagation (#67)

Add tags to decorators (#65)

Add Check version of packages when install plugins (#63)

Add thread propagation (#62)

Add trace ignore (#59)

Support snapshot context (#56)

Support correlation context (#55)

Chores and tests

Test: run multiple versions of supported libraries (#66)

Chore: add pull request template for plugin (#61)

Chore: add dev doc and reorganize the structure (#58)

Test: update test health check (#57)

Chore: add make goal to package release tar ball (#54)

Source code(tar.gz)
Source code(zip)
v0.2.0(Jul 28, 2020)
New plugins

Kafka Plugin (#50)

Tornado Plugin (#48)

Redis Plugin (#44)

Django Plugin (#37)

PyMsql Plugin (#35)

Flask plugin (#31)

API

Add ignore_suffix Config (#40)

Add missing log method and simplify test codes (#34)

Add content equality of SegmentRef (#30)

Validate carrier before using it (#29)

Chores and tests

Test: print the diff list when validation failed (#46)

Created venv builders for linux/windows and req flashers + use documentation (#38)

Source code(tar.gz)
Source code(zip)
v0.1.0(Jun 28, 2020)
API: agent core APIs, check the APIs and the examples

Plugin: built-in libraries http, urllib.request and third-party library requests are supported.

Test: agent test framework is setup, and the corresponding tests of aforementioned plugins are also added.

Source code(tar.gz)
Source code(zip)

Owner

The Apache Software Foundation

GitHub Repository https://skywalking.apache.org/

Collection of functions for working with interlaced content in VapourSynth.

vsfieldkit Collection of functions for working with interlaced content in VapourSynth. It does not have any hard dependencies outside of VapourSynth.

11 May 27, 2022

Appointment Tracker that allows user to input client information and update if needed.

Appointment-Tracker Appointment Tracker allows an assigned admin to input client information regarding their appointment and their appointment time. T

1 Nov 30, 2021

Ghost source since the developer of the project quit due to reasons

👻 Ghost Selfbot The official code for Ghost which was recently discontinued and released to the public. Feel free to use any of the code found in thi

2 Mar 24, 2022

Custom python interface to xstan (a modified (cmd)stan)

Custom python interface to xstan (a modified (cmd)stan) Use at your own risk, currently everything is very brittle and will probably be changed in the

2 Dec 16, 2021

A Python library to simulate a Zoom H6 recorder remote control

H6 A Python library to emulate a Zoom H6 recorder remote control Introduction This library allows you to control your Zoom H6 recorder from your compu

68 Nov 02, 2022

Fetch data from an excel file and create HTML file

excel-to-html Problem Statement! - Fetch data from excel file and create html file Excel.xlsx file contain the information.in multiple rows that is ne

1 Oct 25, 2021

The Zig programming language, packaged for PyPI

Zig PyPI distribution This repository contains the script used to repackage the releases of the Zig programming language as Python binary wheels. This

100 Nov 04, 2022

⚙️ Compile, Read and update your .conf file in python

2 Aug 15, 2022

MoBioTools A simple yet versatile toolkit to automatically setup quantum mechanics/molecular mechanics

A simple yet versatile toolkit to setup quantum mechanical/molecular mechanical (QM/MM) calculations from molecular dynamics trajectories.

17 Nov 27, 2022

fetchmesh is a tool to simplify working with Atlas anchoring mesh measurements

A Python library for working with the RIPE Atlas anchoring mesh. fetchmesh is a tool to simplify working with Atlas anchoring mesh measurements. It ca

2 Aug 30, 2022

DSG - Source code for Digital Scholarship Grant project.

DSG Source code for Dr. Stephanie Tsang's Digital Scholarship Grant project. Work performed by Mr. Wang Minghao while as her Research Assistant. The s

1 Jan 04, 2022

A webapp for taking fast notes, designed for business, school, and collaboration with groups.

JOTS Journal of the Session A webapp for taking fast notes, designed for business, school, and collaboration with groups.

2 Jun 10, 2022

A tool to determine optimal projects for Gridcoin crunchers. Maximize your magnitude!

FindTheMag FindTheMag helps optimize your BOINC client for Gridcoin mining. You can group BOINC projects into two groups: "preferred" projects and "mi

7 Oct 04, 2022

A module to prevent invites and joins to Matrix rooms by checking the involved server(s)' domain.

Synapse Domain Rule Checker A module to prevent invites and joins to Matrix rooms by checking the involved server(s)' domain. Installation From the vi

4 Oct 24, 2022

A repository of study materials related to Think Python 2nd Edition by Allen B. Downey. More information about the book can be found here: https://greenteapress.com/wp/think-python-2e/

Intro-To-Python This content is based on the book Think Python 2nd Edition by Allen B. Downey. More information about the book can be found here: http

63 Jan 07, 2023

FollowSpot is a comprehensive audition tracking fullstack web application for entertainment industry professionals.

FollowSpot is a comprehensive audition tracking fullstack web application for entertainment industry professionals. This app allows users to store information/media for all of their auditions while a

9 Jul 12, 2022

Users can read others' travel journeys in addition to being able to upload and delete posts detailing their own experiences

Users can read others' travel journeys in addition to being able to upload and delete posts detailing their own experiences! Posts are organized by country and destination within that country.

1 Feb 03, 2022

The Python agent for Apache SkyWalking

Related tags

Overview

SkyWalking Python Agent

Documentation

Installation Requirements

Contributing

Contact Us

License

Comments

Behavior

Syntactic sugar

Counter

Histogram

Simple test

Metrics

CPU

total_cpu_utilization

process_cpu_utilization

thread_active_count

Memory

total_mem_utilization

process_mem_utilization

GC

gc_g0

gc_g1

gc_g2

gc_time

Releases(v0.8.0)

v0.8.0(Jul 10, 2022)

What's Changed

New Contributors

v0.7.0(Sep 13, 2021)

v0.6.0(Mar 31, 2021)

v0.5.0(Dec 31, 2020)

v0.4.0(Nov 24, 2020)

v0.3.0(Aug 28, 2020)

v0.2.0(Jul 28, 2020)

v0.1.0(Jun 28, 2020)