Push Prometheus metrics to VictoriaMetrics or other exporters

Overview

test-all codecov

prometheus-push-client

Push metrics from your periodic long-running jobs to existing Prometheus/VictoriaMetrics monitoring system.

Currently supports pushes directly to VictoriaMetrics:

For pure Prometheus setups, several options are supported:

  • to pushgateway or prom-aggregation-gateway in OpenMetrics format via HTTP. Please read corresponding docs about appropriate use cases and limitations;
  • to StatsD or statsd-exporter in StatsD format via UDP. Prometheus and StatsD metric types are not fully compatible, so currenly all metrics become StatsD gauges, but rate, increase, histogram_quantile and other PromQL functions produce same results as if types never changed.

Install it via pip:

pip install prometheus-push-client

Metrics

This library uses prometheus-client metric implementation, but adds some minor tweaks.

Separate registry

New metric constructors use separate PUSH_REGISTRY as a default, not to interfere with other metrics already defined and monitored in existing projects.

Default labelvalues

With regular prometheus_client, defaults may be defined for either none or all the labels (with labelvalues), but that's not enough. Moreover labelvalues sometimes doesn't work as expected.

We probably want to define some defaults, like hostname, or more importantly, if we use VictoriaMetrics cluster, VictoriaMetrics_AccountID= (try 0 as a default) label must always be set, and metrics without it will be ignored.

Following example shows how to use defaults, and how to override them if necessary.

import prometheus_push_client as ppc


counter1 = ppc.Counter(
    name="c1",
    labelnames=["VictoriaMetrics_AccountID", "host", "event_type"],
    default_labelvalues={
        "VictoriaMetrics_AccountID": 0,
        "host": socket.gethostname(),
    }
)


# regular usage
counter1.labels(event_type="login").inc()

# overriding defaults
counter1.labels(host="non-default", event_type="login").inc()
# same effect as above: defaults are applied in `labelvalues`
# order for "missing" labels in the beginning
counter1.labels("non-default", "login").inc()

Metrics with no labels are initialized at creation time. This can have unpleasant side-effect: if we initialize lots of metrics not used in currently running job, batch clients will have to push their non-changing values in every synchronization session.

To avoid that we'll have to properly isolate each task's metrics, which can be impossible or rather tricky, or we can create metrics with default, non-changing labels (like hostname). Such metrics will be initialized on first use (inc), and we'll be pushing only those we actually utilized.

Clients

Batch clients

Batch clients spawn synchronization jobs "in background" (meaning in a thread or asyncio task) to periodically send all metrics from ppc.PUSH_REGISTRY to the destination.

Clients will attempt to stop gracefully, synchronizing registry "one last time" after job exits or crashes. Sometimes this may mess up sampling, but the worst case I could artifically create looks like this:

graceful push effect

Best way to use them is via decorators / context managers. These clients are intended to be used with long running, but finite tasks, which could be spawned anywhere, therefor not easily accessible by the scraper. If that's not the case -- just use "passive mode" w/ the scraper instead.

def influx_udp_async(host, port, period=15.0):
def influx_udp_thread(host, port, period=15.0):
def statsd_udp_async(host, port, period=15.0):
def statsd_udp_thread(host, port, period=15.0):
def influx_http_async(url, verb="POST", period=15.0):
def influx_http_thread(url, verb="POST", period=15.0):
def openmetrics_http_async(url, verb="POST", period=15.0):
def openmetrics_http_thread(url, verb="POST", period=15.0):

Usage example:

import prometheus_push_client as ppc


req_hist = ppc.Histogram(
    name="external_requests",
    namespace="acme"
    subsystem="job123",
    unit="seconds",
    labelnames=["service"]
)


@ppc.influx_udp_async("victoria.acme.inc.net", 9876, period=15)
async def main(urls):
    # the job ...
    req_hist.labels(gethostname(url)).observe(response.elapsed)

# OR

async def main(urls):
    async with ppc.influx_udp_async("victoria.acme.inc.net", 9876, period=15):
        # the job ...
        req_hist.labels(gethostname(url)).observe(response.elapsed)

Please read about mandatory job tag within url while using pushgateway.

Streaming clients

If for some reason every metric change needs to be synced, UDP streaming clients are implemented in this library.

def influx_udp_aiostream(host, port):
def influx_udp_stream(host, port):
def statsd_udp_aiostream(host, port):
def statsd_udp_stream(host, port):

Usage is completely identical to batch clients' decorators / context managers.

⚠️ Histogram and Summary .time() decorator doesn't work in this mode atm, because it can't be monkey-patched easily.

Transports

Main goal is not to interrupt measured jobs with errors from monitoring code. Therefor all transports will attempt to catch all network errors, logging error info and corresponding tracebacks to stdout.

You might also like...
Google Foobar challenge solutions from my experience and other's on the web.
Google Foobar challenge solutions from my experience and other's on the web.

Google Foobar challenge Google Foobar challenge solutions from my experience and other's on the web. Note: Problems indicated with "Mine" are tested a

Custom component to calculate estimated power consumption of lights and other appliances
Custom component to calculate estimated power consumption of lights and other appliances

Custom component to calculate estimated power consumption of lights and other appliances. Provides easy configuration to get virtual power consumption sensors in Home Assistant for all your devices which don't have a build in power meter.

The tool helps to find hidden parameters that can be vulnerable or can reveal interesting functionality that other hunters miss.
The tool helps to find hidden parameters that can be vulnerable or can reveal interesting functionality that other hunters miss.

The tool helps to find hidden parameters that can be vulnerable or can reveal interesting functionality that other hunters miss. Greater accuracy is achieved thanks to the line-by-line comparison of pages, comparison of response code and reflections.

Emulate and Dissect MSF and *other* attacks
Emulate and Dissect MSF and *other* attacks

Need help in analyzing Windows shellcode or attack coming from Metasploit Framework or Cobalt Strike (or may be also other malicious or obfuscated code)? Do you need to automate tasks with simple scripting? Do you want help to decrypt MSF generated traffic by extracting keys from payloads?

External Network Pentest Automation using Shodan API and other tools.

Chopin External Network Pentest Automation using Shodan API and other tools. Workflow Input a file containing CIDR ranges. Converts CIDR ranges to ind

to learn how to do pull request and do contribution to other's repo
to learn how to do pull request and do contribution to other's repo

Hacktoberfest-2021 - open-source-contribution An Open Source repository to Teach people How to contribute to open sources. 💥 🔥 JOIN PVX PROGRAMMING

python scripts and other files to generate induction encoder PCBs in Kicad
python scripts and other files to generate induction encoder PCBs in Kicad

induction_encoder python scripts and other files to generate induction encoder PCBs in Kicad Targeting the Renesas IPS2200 encoder chips.

scap is a tool for putting code in places and for other purposes

Scap is the deployment script used by Wikimedia Foundation to publish code and configuration on production web servers.

Python script for changing the SSH banner content with other content

Banner-changer-py Python script for changing the SSH banner content with other content. The Script will take the content of a specified file range and

Comments
  • Can I use this to push historical data?

    Can I use this to push historical data?

    I came here from https://stackoverflow.com/a/67562080/7424510.

    Suppose, I've got a temperature sensor, and I've got a list of UNIX time stamps and temperatures, like so:

    [
        (1661863713, 22),
        (1661863714, 21),
        (1661863715, 22)
    ]
    

    Can I push this with this client? How?

    opened by joernschellhaas 1
  • http fixes, pushgateway client

    http fixes, pushgateway client

    • http: ensure data ends with "\n\n"
    • openmetrics: data type metric header
    • added openmetrics http clients for pushgateway / prom-aggregation-gateway
    opened by gistart 1
Releases(0.0.8)
Owner
olegm
olegm
Python 3.9.4 Graphics and Compute Shader Framework and Primitives with no external module dependencies

pyshader Python 3.9.4 Graphics and Compute Shader Framework and Primitives with no external module dependencies Fully programmable shader model (even

Alastair Cota 1 Jan 11, 2022
This collection is to provide an easier way to interact with Juniper

Ansible Collection - cremsburg.apstra Overview The goal of this collection is to provide an easier way to interact with Juniper's Apstra solution. Whi

Calvin Remsburg 1 Jan 18, 2022
Python data loader for Solar Orbiter's (SolO) Energetic Particle Detector (EPD).

Data loader (and downloader) for Solar Orbiter/EPD energetic charged particle sensors EPT, HET, and STEP. Supports level 2 and low latency data provided by ESA's Solar Orbiter Archive.

Jan Gieseler 9 Dec 16, 2022
A command-line utility that creates projects from cookiecutters (project templates), e.g. Python package projects, VueJS projects.

Cookiecutter A command-line utility that creates projects from cookiecutters (project templates), e.g. creating a Python package project from a Python

18.6k Jan 02, 2023
Easytile blender - Simple Blender 2.83 addon for tiling meshes easily

easytile_blender Dead simple, barebones Blender (2.83) addon for placing meshes as tiles. Installation In Blender, go to Edit Preferences Add-ons

Sam Gibson 6 Jul 19, 2022
The best way to learn Python is by practicing examples. The repository contains examples of basic concepts of Python. You are advised to take the references from these examples and try them on your own.

90_Python_Exercises_and_Challenges The best way to learn Python is by practicing examples. This repository contains the examples on basic and advance

Milaan Parmar / Милан пармар / _米兰 帕尔马 205 Jan 06, 2023
Code for ML, domain generation, graph generation of ABC dataset

This is the repository for codes for ML, domain generation, graph generation of Asymmetric Buckling Columns (ABC) dataset in the paper "Learning Mechanically Driven Emergent Behavior with Message Pas

Peerasait Prachaseree (Jeffrey) 0 Jan 28, 2022
Retrying library for Python

Tenacity Tenacity is an Apache 2.0 licensed general-purpose retrying library, written in Python, to simplify the task of adding retry behavior to just

Julien Danjou 4.3k Jan 02, 2023
WordlistPasswordGenerator - Shuhfab Basheer

WordlistPasswordGenerator - Shuhfab Basheer Python wordlist generator MAINTAINER

1 Dec 31, 2021
A simple code for processing images to local binary pattern.

This figure is gotten from this link https://link.springer.com/chapter/10.1007/978-3-030-01449-0_24 LBP-Local-Binary-Pattern A simple code for process

Happy N. Monday 3 Feb 15, 2022
Homed - Light-weight, easily configurable, dockerized homepage

homed GitHub Repo Docker Hub homed is a light-weight customizable portal primari

Matt Walters 12 Dec 15, 2022
A10 cipher - A Hill 2x2 cipher that totally gone wrong

A10_cipher This is a Hill 2x2 cipher that totally gone wrong, it encrypts with H

Caner Çetin 15 Oct 19, 2022
Collection of tools to be more productive in your work environment and to avoid certain repetitive tasks. 💛💙💚

Collection of tools to be more productive in your work environment and to avoid certain repetitive tasks. 💛💙💚

Raja Rakotonirina 2 Jan 10, 2022
OpenSea NFT API App using Python and Streamlit

opensea-nft-api-tutorial OpenSea NFT API App using Python and Streamlit Tutorial Video Walkthrough https://www.youtube.com/watch?v=49SupvcFC1M Instruc

64 Oct 28, 2022
Explores the python bytecode, provides some tools to access it for fun and profit.

Pyasmtools - looking at the python bytecode for fun and profit. The pyasmtools library is made up of two parts A python bytecode disassembler . See Py

Michael Moser 299 Jan 04, 2023
Subscribe, listen and (in the future) download your favorite podcasts, quickly and easily.

Minimal Podcasts Player https://github.com/son-link/minimal-podcasts-player Subscribe, listen and (in the future) download your favorite podcasts, qui

Alfonso Saavedra 14 Nov 11, 2022
The Official Jaseci Code Repository

Jaseci Release Notes Version 1.2.2 Updates Added new built-ins for nodes and edges (context, info, and details) Fixed dot output Added reset command t

136 Dec 20, 2022
A community-driven python bot that aims to be as simple as possible to serve humans with their everyday tasks

JARVIS on Messenger Just A Rather Very Intelligent System, now on Messenger! Messenger is now used by 1.2 billion people every month. With the launch

Swapnil Agarwal 1.3k Jan 07, 2023
Python 3 script for installing kali tools on your linux machine

Python 3 script for installing kali tools on your linux machine

gh0st 2 Apr 20, 2022
Gmvault: Backup and restore your gmail account

Gmvault: Backup and restore your gmail account Gmvault is a tool for backing up your gmail account and never lose email correspondence. Gmvault is ope

Guillaume Aubert 3.5k Jan 01, 2023