Docker container log aggregation with Elasticsearch, Kibana & Filebeat

Related tags

Loggingepilog
Overview

Epilog

>> Dead simple container log aggregation with ELK stack <<

python elasticsearch kibana kibana github_actions

Preface

Epilog aims to demonstrate a language-agnostic, non-invasive, and straightforward way to add centralized logging to your stack. Centralized logging can be difficult depending on how much control you need over the log messages, how robust you need the logging system to be, and how you want to display the data to the consumer.

Why?

Invasive logging usually entails you having to build a logging pipeline and integrate that into your application. Adding an extensive logging workflow directly to your application is non-trivial for a few reasons:

  • The workflow becomes language-specific and hard to scale as your application gets decentralized over time and starts to take advantage of multiple languages.

  • The logging pipeline gets tightly coupled with the application code.

  • Extensive logging in a blocking manner can significantly hurt the performance of the application.

  • Doing logging in a non-blocking state is difficult and usually requires a non-trivial amount of application code changes when the logging requirements change.

This repository lays out a dead-simple but extensible centralized logging workflow that collects logs from docker containers in a non-invasive manner. To achieve this, we've used the reliable ELK stack which is at this point, an industry standard.

Features

  • Asynchronous log-aggregation pipeline that's completely decoupled from the app instances generating the logs.

  • Zero effect on performance if the app instances aren't doing expensive synchronous logging operations internally.

  • Horizontal scaling is achievable by adding more nodes to the Elasticsearch cluster.

  • To keep the storage requirements at bay, log messages are automatically deleted after 7 days. This is configurable.

  • Synchronization during container startup to reduce the number of missing logs.

  • All the Log messages can be filtered and queried interactively from a centralized location via the Kibana dashboard.

Architecture

This workflow leverages Filebeat to collect the logs, Elasticsearch to store and query the log messages, and Kibana to visualize the data interactively. The following diagram explains how logs flow from your application containers and becomes queryable in the Kibana dashboards:

epilog_arch

Here, the Application is a dockerized Python module that continuously sends log messages to the standard output.

On a Unix machine, Docker containers save these log messages in the /var/lib/docker/containers/*/*.log directory. In this directory, Filebeat listens for new log messages and sends them to Elasticsearch in batches. This makes the entire logging workflow asynchronous as Filebeat isn't coupled with the application and is lightweight enough to be deployed with every instance of your application.

The log consumer can make query requests via the Kibana dashboards and interactively search and filter the relevant log messages. The Caddy reverse proxy server is helpful during local development as you won't have to memorize the ports to access Elasticsearch and Kibana. You can also choose to use Caddy instead of Ngnix as a reverse proxy and load balancer in your production orchestration.

Installation

  • Make sure you have Docker, Docker compose V2 installed on your system.

  • Clone the repo.

  • Go to the root directory and run:

    make up
    

    This will spin up 2 Elasticsearch nodes, 1 Filebeat instance, 1 log emitting app instance, and the reverse proxy server.

  • To shut down everything gracefully, run:

    make down
    
  • To kill the container processes and clean up all the volumes, run:

    make kill && make clean
    

Exploration

Once you've run the make up command:

  • To access the Kibana dashboard, go to https://kibana.localhost. Since our reverse proxy adds SSL to the localhost, your browser will complain about the site being unsafe. Just ignore it and move past.

  • When prompted for credentials, use elastic as username and debian as password. You can configure this in the .env file.

  • Once you're inside the Kibana dashboard, head over to the Logs panel under the Observability section on the left panel.

    kibana_1

  • You can filter the logs by container name. Once you start typing container.name literally, Kibana will give you suggestions based on the names of the containers running on your machine.

    kibana_2 )

  • Another filter you might want to explore is filtering by hostname. To do so, type host.name and it'll show the available host identifiers in a dropdown. In this case, all the containers live in the same host. So there's only one available host to filter by. These filters are defined in the processors segment of the filebeat.yml file. You can find a comprehensive list of processors here.

    kibana_3

Maintenance & Extensibility

  • If you need log transformation, adding Logstash to this stack is quite easy. All you'll have to do is add a Logstash instance to the docker-compose.yml file and point Filebeat to send the logs to Logstash instead of Elasticsearch. Logstash will then transform the logs and save them in the Elasticsearch search cluster.

  • To scale up the Elasticsearch cluster, you can follow the configuration of es02 node in the docker-compose file. More nodes can be added similarly to achieve horizontal scaling.

  • In a production setup, your app will most likely live in separate hosts than the Elasticsearch clusters. In that case, a Filebeat instance should live with every instance of the log generating app and these will send the logs to a centralized location—directly to Elasticsearch or first to Logstash and then to Elasticsearch clusters—depending on your need.

Disclaimer

  • This pipleline was tested in a Unix-like system, mainly Ubuntu and macOS. Also, the bash scripts might not work out of the box on Windows.

  • This setup only employs a rudimentary password-based authentication system. You should add TLS encryption to your production ELK stack. Here's an example of how you might be able to do so.

  • For demonstration purposes, this repository has .env file in the root directory. In your production application, you should never add the .env files to your version control system.

Resources

🍰
Owner
Redowan Delowar
Skeptical Empiricist. Indefatigable Walker. Software Artisan. Opinions are an amalgamation of diverse multifaceted factors.
Redowan Delowar
This is a key logger based in python which when executed records all the keystrokes of the system it has been executed on .

This is a key logger based in python which when executed records all the keystrokes of the system it has been executed on

Purbayan Majumder 0 Mar 28, 2022
GTK and Python based, system performance and usage monitoring tool

System Monitoring Center GTK3 and Python 3 based, system performance and usage monitoring tool. Features: Detailed system performance and usage usage

Hakan Dündar 649 Jan 03, 2023
Ransomware leak site monitoring

RansomWatch RansomWatch is a ransomware leak site monitoring tool. It will scrape all of the entries on various ransomware leak sites, store the data

Zander Work 278 Dec 31, 2022
A watchdog and logger to Discord for hosting ScPrime servers.

ScpDog A watchdog and logger to Discord for hosting ScPrime servers. Designed to work on Linux servers. This is only capable of sending the logs from

Keagan Landfried 3 Jan 10, 2022
Debugging-friendly exceptions for Python

Better tracebacks This is a more helpful version of Python's built-in exception message: It shows more code context and the current values of nearby v

Clemens Korndörfer 1.2k Dec 28, 2022
Key Logger - Key Logger using Python

Key_Logger Key Logger using Python This is the basic Keylogger that i have made

Mudit Sinha 2 Jan 15, 2022
A python library used to interact with webots robocup game web logs

A python library used to interact with webots robocup game web logs

Hamburg Bit-Bots 2 Nov 05, 2021
Python logging package for easy reproducible experimenting in research

smilelogging Python logging package for easy reproducible experimenting in research. Why you may need this package This project is meant to provide an

Huan Wang 20 Dec 23, 2022
Log processor for nginx or apache that extracts user and user sessions and calculates other types of useful data for bot detection or traffic analysis

Log processor for nginx or apache that extracts user and user sessions and calculates other types of useful data for bot detection or traffic analysis

David Puerta Martín 1 Nov 11, 2021
Python bindings for g3log

g3logPython Python bindings for g3log This library provides python3 bindings for g3log + g3sinks (currently logrotate, syslog, and a color-terminal ou

4 May 21, 2021
Lazy Profiler is a simple utility to collect CPU, GPU, RAM and GPU Memory stats while the program is running.

lazyprofiler Lazy Profiler is a simple utility to collect CPU, GPU, RAM and GPU Memory stats while the program is running. Installation Use the packag

Shankar Rao Pandala 28 Dec 09, 2022
Simple and versatile logging library for python 3.6 above

Simple and versatile logging library for python 3.6 above

Miguel 1 Nov 23, 2022
A cool logging replacement for Python.

Welcome to Logbook Travis AppVeyor Supported Versions Latest Version Test Coverage Logbook is a nice logging replacement. It should be easy to setup,

1.4k Nov 11, 2022
Small toolkit for python multiprocessing logging to file

Small Toolkit for Python Multiprocessing Logging This is a small toolkit for solving unsafe python mutliprocess logging (file logging and rotation) In

Qishuai 1 Nov 10, 2021
Progressbar 2 - A progress bar for Python 2 and Python 3 - "pip install progressbar2"

Text progress bar library for Python. Travis status: Coverage: Install The package can be installed through pip (this is the recommended method): pip

Rick van Hattem 795 Dec 18, 2022
Scout: an open-source version of the monitoring tool

Badger Scout Scout is an open-source version of the monitoring tool used by Badg

Badger Finance 2 Jan 13, 2022
Beautifully colored, quick and simple Python logging

Python Quick Logging | QLogging Beautifully colored, quick and simple Python logging. This logger is based on Python logging package Screenshots: Term

45 Sep 25, 2022
Track Nano accounts and notify via log file or email

nano-address-notifier Track accounts and notify via log file or email Required python libs

Joohansson (Json) 4 Nov 08, 2021
This open-source python3 script is a builder to the very popular token logger that is on my github that many people use.

Discord-Logger-Builder This open-source python3 script is a builder to the very popular token logger that is on my github that many people use. This i

Local 4 Nov 17, 2021
Colored terminal output for Python's logging module

coloredlogs: Colored terminal output for Python's logging module The coloredlogs package enables colored terminal output for Python's logging module.

Peter Odding 496 Dec 30, 2022