Small toolkit for python multiprocessing logging to file

Related tags

Loggingmp-socket-logging
Overview

Small Toolkit for Python Multiprocessing Logging

This is a small toolkit for solving unsafe python mutliprocess logging (file logging and rotation)

Install

Only for Python 3.5+

pip install mp-socket-logging

Features

                            (child process)
                           +---------------+
             +<------------+     client    |
 +-------+-+-+             +---------------+
 |           |              (child process)
 +  Socket   |             +---------------+
 |           +<------------+     client    |
 +---------+-+             +---------------+
       |
       |                 (main process)
       |               +---------------+          +--------------+
       +-------------->+     server    +--------->+   log file   +
                       +---------------+          +--------------+
  • The package uses unix socket for communication between processes.
  • I/O operations will all be done by server in main process.
  • Client Class itself is a logging.Handler which will be added to user's logger inside child process. Client handler will do the formatting and sending the logs to server side through socket.
  • Inside child process, user can do logging as normal with the logger.

Quick Start

  • start a server before in the main process
server_handler = ServerHandler(LOG_FILE, maxBytes=MAX_BYTES)
server_handler.setLevel(logging.INFO)
server = Server(server_handler, batch_size=BATCH_SIZE)
server.start()
  • register the client to your logger inside multiprocessing worker for sending logs to server through socket
  • you can define your own formatter to add to the handler (otherwise a default formatter will be used)
client_logger = logging.getLogger("client_logger")
formatter = logging.Formatter(
    "%(asctime)s - %(name)s - %(filename)s- [%(levelname)s]: %(message)s"
)
register_handler(client_logger, logging_level=logging.INFO, formatter=formatter)
client_logger.setLevel(logging.INFO)
You might also like...
A Python package which supports global logfmt formatted logging.

Python Logfmter A Python package which supports global logfmt formatted logging. Install $ pip install logfmter Usage Before integrating this library,

Joshua Taylor Eppinette 15 Dec 29, 2022
Stand-alone parser for User Access Logging from Server 2012 and newer systems
Stand-alone parser for User Access Logging from Server 2012 and newer systems

KStrike Stand-alone parser for User Access Logging from Server 2012 and newer systems BriMor Labs KStrike This script will parse data from the User Ac

BriMor Labs 69 Nov 1, 2022
Logging system for the TPC software.

tpc_logger Logging system for the TPC software. The TPC Logger class provides a singleton for logging information within C++ code or in the python API

UC Davis Machine Learning 1 Jan 10, 2022
Outlog it's a library to make logging a simple task

outlog Outlog it's a library to make logging a simple task!. I'm a lazy python user, the times that i do logging on my apps it's hard to do, a lot of

ZSendokame 2 Mar 5, 2022
metovlogs is a very simple logging library

metovlogs is a very simple logging library. Setup is one line, then you can use it as a drop-in print replacement. Sane and useful log format out of the box. Best for small or early projects.

Azat Akhmetov 1 Mar 1, 2022
A small utility to pretty-print Python tracebacks. ⛺
A small utility to pretty-print Python tracebacks. ⛺

TBVaccine TBVaccine is a utility that pretty-prints Python tracebacks. It automatically highlights lines you care about and deemphasizes lines you don

Stavros Korokithakis 365 Nov 11, 2022
Track Nano accounts and notify via log file or email

nano-address-notifier Track accounts and notify via log file or email Required python libs

Joohansson (Json) 4 Nov 8, 2021
Pretty-print tabular data in Python, a library and a command-line utility. Repository migrated from bitbucket.org/astanin/python-tabulate.

python-tabulate Pretty-print tabular data in Python, a library and a command-line utility. The main use cases of the library are: printing small table

Sergey Astanin 1.5k Jan 6, 2023
Progressbar 2 - A progress bar for Python 2 and Python 3 - "pip install progressbar2"

Text progress bar library for Python. Travis status: Coverage: Install The package can be installed through pip (this is the recommended method): pip

Rick van Hattem 795 Dec 18, 2022
Releases(v0.1.1)

  • v0.1.1(Nov 10, 2021)

    What's Changed

    • add verbose and update version by @FerdinandZhong in https://github.com/FerdinandZhong/mp-socket-logging/pull/6

    Full Changelog: https://github.com/FerdinandZhong/mp-socket-logging/compare/v0.1.0...v0.1.1

    Source code(tar.gz)
    Source code(zip)

  • v0.1.0(Oct 19, 2021)

    What's Changed

    • Develop the package
    • Add auto-testing workflow
    • Add brief introduction

    New Contributors

    • @FerdinandZhong

    Full Changelog: https://github.com/FerdinandZhong/mp-socket-logging/commits/v0.1.0

    Source code(tar.gz)
    Source code(zip)
Owner
Qishuai
Qishuai
GitHub Repository
This is a DemoCode for parsing through large log files and triggering an email whenever there's an error.

LogFileParserDemoCode This is a DemoCode for parsing through large log files and triggering an email whenever there's an error. There are a total of f

2 Jan 06, 2022
Pretty-print tabular data in Python, a library and a command-line utility. Repository migrated from bitbucket.org/astanin/python-tabulate.

python-tabulate Pretty-print tabular data in Python, a library and a command-line utility. The main use cases of the library are: printing small table

Sergey Astanin 1.5k Jan 06, 2023
Small toolkit for python multiprocessing logging to file

Small Toolkit for Python Multiprocessing Logging This is a small toolkit for solving unsafe python mutliprocess logging (file logging and rotation) In

Qishuai 1 Nov 10, 2021
Python script to scan log files/system for unauthorized access around system

checkLogs Python script to scan log files/system for unauthorized access around Linux systems Table of contents General info Getting started Usage Gen

James Kelly 1 Feb 25, 2022
This open-source python3 script is a builder to the very popular token logger that is on my github that many people use.

Discord-Logger-Builder This open-source python3 script is a builder to the very popular token logger that is on my github that many people use. This i

Local 4 Nov 17, 2021
A Prometheus exporter for monitoring & analyzing Grafana Labs' technical documentation

grafana-docs-exporter A Prometheus exporter for monitoring & analyzing Grafana Labs' technical documentation Here is the public endpoint.

Matt Abrams 5 May 02, 2022
Json Formatter for the standard python logger

This library is provided to allow standard python logging to output log data as json objects. With JSON we can make our logs more readable by machines and we can stop writing custom parsers for syslo

Zakaria Zajac 1.4k Jan 04, 2023
Progressbar 2 - A progress bar for Python 2 and Python 3 - "pip install progressbar2"

Text progress bar library for Python. Travis status: Coverage: Install The package can be installed through pip (this is the recommended method): pip

Rick van Hattem 795 Dec 18, 2022
Outlog it's a library to make logging a simple task

outlog Outlog it's a library to make logging a simple task!. I'm a lazy python user, the times that i do logging on my apps it's hard to do, a lot of

ZSendokame 2 Mar 05, 2022
Logging system for the TPC software.

tpc_logger Logging system for the TPC software. The TPC Logger class provides a singleton for logging information within C++ code or in the python API

UC Davis Machine Learning 1 Jan 10, 2022
Fancy console logger and wise assistant within your python projects

Fancy console logger and wise assistant within your python projects. Made to save tons of hours for common routines.

BoB 5 Apr 01, 2022
GTK and Python based, system performance and usage monitoring tool

System Monitoring Center GTK3 and Python 3 based, system performance and usage monitoring tool. Features: Detailed system performance and usage usage

Hakan Dündar 649 Jan 03, 2023
Simple and versatile logging library for python 3.6 above

Simple and versatile logging library for python 3.6 above

Miguel 1 Nov 23, 2022
Structured Logging for Python

structlog makes logging in Python faster, less painful, and more powerful by adding structure to your log entries. It's up to you whether you want str

Hynek Schlawack 2.3k Jan 05, 2023
The easy way to send notifications

See changelog for recent changes Got an app or service and you want to enable your users to use notifications with their provider of choice? Working o

Or Carmi 2.4k Dec 25, 2022
Integrates a UPS monitored by NUT into OctoPrint

OctoPrint UPS This OctoPrint plugin interfaces with a UPS monitored by NUT (Network UPS Tools). Requirements NUT must be configured by the user. This

Shawn Bruce 11 Jul 05, 2022
Ransomware leak site monitoring

RansomWatch RansomWatch is a ransomware leak site monitoring tool. It will scrape all of the entries on various ransomware leak sites, store the data

Zander Work 278 Dec 31, 2022
A new kind of Progress Bar, with real time throughput, eta and very cool animations!

alive-progress :) A new kind of Progress Bar, with real-time throughput, eta and very cool animations! Ever found yourself in a remote ssh session, do

Rogério Sampaio de Almeida 4k Dec 30, 2022
giving — the reactive logger

giving is a simple, magical library that lets you log or "give" arbitrary data throughout a program and then process it as an event stream.

Olivier Breuleux 0 May 24, 2022
Splunk Add-On to collect audit log events from Github Enterprise Cloud

GitHub Enterprise Audit Log Monitoring Splunk modular input plugin to fetch the enterprise audit log from GitHub Enterprise Support for modular inputs

Splunk GitHub 12 Aug 18, 2022