蓝鲸日志平台(BK-LOG)是为解决分布式架构下日志收集、查询困难的一款日志产品,基于业界主流的全文检索引擎

Related tags

Loggingbk-log
Overview

license GitHub release (latest by date) PRs Welcome codecov Web Test

English | 简体中文

重要提示: master 分支在开发过程中可能处于 不稳定或者不可用状态 。 请通过 releases 而非 master 去获取稳定的二进制文件。

蓝鲸日志平台(BK-LOG)是为解决分布式架构下日志收集、查询困难的一款日志产品,基于业界主流的全文检索引擎,通过蓝鲸智云的专属 Agent 进行日志采集,提供多种场景化的采集、查询功能。

Overview

Features

  • 简单易用的日志采集
  • 可视化的日志字段提取
  • 功能强大的日志查询
  • 实时日志和日志上下文
  • 日志关键字/汇聚告警
  • 支持第三方 ES 接入
  • 分布式跟踪支持
  • 仪表盘能力
  • 在线日志文件提取

Getting Started

  • 安装好MySQL 5.7Python3.6,若同时开发多个项目,请创建Python虚拟环境

  • 创建数据库 CREATE DATABASE bk_log DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;

  • 在项目config新建local_settings.py文件,文件内容为数据库配置,如

       DATABASES = {
          'default': {
              'ENGINE': 'django.db.backends.mysql',
              'NAME': 'bk_log',
              'USER': 'root',
              'PASSWORD': '',
              'HOST': '127.0.0.1',
              'PORT': '3306',
          },
      }
  • 编译前端

    cd web
    npm install
    npm run build
  • 配置环境变量

    APP_ID=${APP_ID}
    BK_IAM_V3_INNER_HOST=${BK_IAM_V3_INNER_HOST}
    BK_PAAS_HOST=${BK_PAAS_HOST}
    APP_TOKEN=${APP_TOKEN}
    # BKAPP_REDIS_PASSWORD=${BKAPP_REDIS_PASSWORD}  # 缓存和Celery会使用到redis,如果本地redis有密码需要增加这个环境变量
  • 启动工程 python manage.py runserver 8000

  • 启动celery celery -A worker -l info -c 8

Support

BlueKing Community

  • BK-CMDB:蓝鲸配置平台(蓝鲸 CMDB)是一个面向资产及应用的企业级配置管理平台。
  • BK-CI:蓝鲸持续集成平台是一个开源的持续集成和持续交付系统,可以轻松将你的研发流程呈现到你面前。
  • BK-BCS:蓝鲸容器管理平台是以容器技术为基础,为微服务业务提供编排管理的基础服务平台。
  • BK-BCS-SaaS:蓝鲸容器管理平台 SaaS 基于原生 Kubernetes 和 Mesos 自研的两种模式,提供给用户高度可扩展、灵活易用的容器产品服务。
  • BK-PaaS:蓝鲸 PaaS 平台是一个开放式的开发平台,让开发者可以方便快捷地创建、开发、部署和管理 SaaS 应用。
  • BK-SOPS:标准运维(SOPS)是通过可视化的图形界面进行任务流程编排和执行的系统,是蓝鲸体系中一款轻量级的调度编排类 SaaS 产品。

Contributing

如果你有好的意见或建议,欢迎给我们提 Issues 或 Pull Requests,为蓝鲸开源社区贡献力量。关于 bk-log 分支管理、Issue 以及 PR 规范, 请阅读 Contributing Guide

腾讯开源激励计划 鼓励开发者的参与和贡献,期待你的加入。

License

项目基于 MIT 协议, 详细请参考 LICENSE

You might also like...
This is a DemoCode for parsing through large log files and triggering an email whenever there's an error.

LogFileParserDemoCode This is a DemoCode for parsing through large log files and triggering an email whenever there's an error. There are a total of f

Python script to scan log files/system for unauthorized access around system

checkLogs Python script to scan log files/system for unauthorized access around Linux systems Table of contents General info Getting started Usage Gen

LightLog is an open source deep learning based lightweight log analysis tool for log anomaly detection.

LightLog Introduction LightLog is an open source deep learning based lightweight log analysis tool for log anomaly detection. Function description [BG

A simple CLI to convert snapshots into EAVT log, and EAVT log into SCD.

EAVT helper CLI Simple CLI to convert snapshots into eavt log, and eavt log into slowly changing dimensions Usage Installation Snapshot to EAVT log EA

GoAccess is a real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser.
GoAccess is a real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser.

GoAccess What is it? GoAccess is an open source real-time web log analyzer and interactive viewer that runs in a terminal on *nix systems or through y

Yet another Django audit log app, hopefully the simplest one.

django-easy-audit Yet another Django audit log app, hopefully the easiest one. This app allows you to keep track of every action taken by your users.

The goal of pandas-log is to provide feedback about basic pandas operations. It provides simple wrapper functions for the most common functions that add additional logs

pandas-log The goal of pandas-log is to provide feedback about basic pandas operations. It provides simple wrapper functions for the most common funct

Log and View requests made on Django
Log and View requests made on Django

Django Request Viewer Log and view requests made on your Django App Introduction Recently, @ichtrojan and @toniastro released horus, a request logger

With Django Hijack, admins can log in and work on behalf of other users without having to know their credentials.
With Django Hijack, admins can log in and work on behalf of other users without having to know their credentials.

Django Hijack With Django Hijack, admins can log in and work on behalf of other users without having to know their credentials. Docs See http://django

"Log in as user" for the Django admin.

django-loginas About "Login as user" for the Django admin. loginas supports Python 3 only, as of version 0.4. If you're on 2, use 0.3.6. Installing dj

Django app that enables staff to log in as other users using their own credentials.
Django app that enables staff to log in as other users using their own credentials.

Impostor Impostor is a Django application which allows staff members to login as a different user by using their own username and password. Login Logg

Inject an ID into every log message from a Django request. ASGI compatible, integrates with Sentry, and works with Celery

Django GUID Now with ASGI support! Django GUID attaches a unique correlation ID/request ID to all your log outputs for every request. In other words,

Paper list of log-based anomaly detection

Paper list of log-based anomaly detection

A python application to log QSOs directly to QRZ.com from the command line
A python application to log QSOs directly to QRZ.com from the command line

qrzlogger This script is a QRZ.com command line QSO logger. It does the following: asks the user for a call sign displays available call sign info pul

[Singing Log] Let your program learn to sing!

[Singing Log] Let your program learn to sing! You must have thought this was changelog when you saw the English title, but it's not, it's chànggēlog. What it does is allow your program to print logs and sing at the same time!

Track Nano accounts and notify via log file or email

nano-address-notifier Track accounts and notify via log file or email Required python libs

A simple Python script using Telethon to log all (or some) messages a user or bot account can see on Telegram.

telegram-logger A simple Python script using Telethon to log all (or some) messages a user or bot account can see on Telegram. Requirements Python 3.6

Summarize LSF job properties by parsing log files.
Summarize LSF job properties by parsing log files.

Summarize LSF job properties by parsing log files of workflows executed by Snakemake.

Comments
  • 分隔符过滤匹配带有空格数据无法采集到数据

    分隔符过滤匹配带有空格数据无法采集到数据

    数据接入配置日志内容过滤规则,采用分隔符过滤,匹配字段前后带有空格数据无法采集到日志数据:

    • 采集接入配置:采用","分割,匹配第二列包含“ after callback”(字符串前面带有空格),编辑查看详情已将空格自动删除; image image

    • 问题原因

    1. django 序列化器默认将空格删除(默认trim_whitespace=True); image

    2. 下发的采集配置文件已将空格取消,但采集器匹配内容的时候分割日志后是带空格去匹配关键字的,导致匹配不到内容; image image

    • 解决方案
    1. 方案一:修改序列器,增加trim_whitespace=False;
    2. 方案二:修改采集器匹配逻辑,分割完去除前后空格去匹配关键字(个人建议采用该方案,因为空格内容对用户无意义); image
    kind/bug 
    opened by Eli-ZhangLu 3
  • 采集项数据量过大,索引集管理新增采集类型索引集报502错误

    采集项数据量过大,索引集管理新增采集类型索引集报502错误

    1. 问题描述 当业务下“数据接入”建立的采集项过多时,在“索引集管理”-“新建”-“采集接入”-“新增索引” 报502错误;

    2. 版本信息 4.2.653

    3. 报错截图 image

    4. 问题原因 metadata_get_result_table_storage 接口设计不合理,为get请求,参数result_table_list支持传list,当达到一定大小后接口返回502错误

    kind/bug 
    opened by Eli-ZhangLu 1
  • 字段提取功能问题

    字段提取功能问题

    版本信息

    • saas&后台版本:4.2.653

    json提取方式问题

    1. 问题描述:json清洗后带特殊字符的字段重命名后前端校验不通过;
    2. 理想结果:json清洗后带特殊字符的字段重命名,只需要校验重命名字段格式,而不需要校验原始字段;
    3. 问题截图: image
    4. 日志原文:
    {
        "@timestamp": "2021-11-05T22:10:00.000Z",
        "@version": "1",
        "program": "/usr/sbin/cron",
        "host": "127.0.0.1",
        "timestamp": "Nov  6 06:10:00",
        "message": "(root) CMD (   /usr/libexec/atrun)",
        "priority": 78,
        "pid": "39172",
        "facility_label": "clock",
        "logsource": "ELI-PC",
        "severity_label": "Informational",
        "severity": 6,
        "facility": 9
    }
    

    时间字段清洗问题

    1. 问题描述:“2021-11-05T22:10:00.000Z” 时间数据用“YYYY-MM-DDTHH:mm:ss.SSSZ”格式清洗失败;
    2. 理想结果:时间字段和数据格式匹配应该清洗成功;
    3. 问题截图: image
    4. 日志原文:
    {
        "@timestamp": "2021-11-05T22:10:00.000Z",
        "@version": "1",
        "program": "/usr/sbin/cron",
        "host": "127.0.0.1",
        "timestamp": "Nov  6 06:10:00",
        "message": "(root) CMD (   /usr/libexec/atrun)",
        "priority": 78,
        "pid": "39172",
        "facility_label": "clock",
        "logsource": "ELI-PC",
        "severity_label": "Informational",
        "severity": 6,
        "facility": 9
    }
    
    kind/bug uat 
    opened by Eli-ZhangLu 1
  • Trace-Log-Metric关联方案

    Trace-Log-Metric关联方案

    指标关联Trace

    exemplar机制

    prometheus

    prometheus主要是采用 exemplars 的机制在 metrics 中带上额外的信息。通过metrics的接口可以同事暴露exemplar https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#exemplars-1

    # 后面的内容就是exemplar
    # lable 采样值 采样时间
    foo_bucket{le="0.1"} 8 # {} 0.054
    foo_bucket{le="1"} 11 # {trace_id="KOO5S4vxi0o"} 0.67
    foo_bucket{le="10"} 17 # {trace_id="oHg5SJYRHA0"} 9.8 1520879607.789
    

    注入方式

    c := GetPlayURLTotal.WithLabelValues(
                strconv.FormatInt(int64(callerType), 10),
                strconv.FormatInt(int64(device.GetOs()), 10),
                strconv.FormatInt(int64(device.GetNetwork()), 10),
                videoFormat,
    )
    sp := trace.SpanFromContext(ctx).SpanContext()
    if sp.IsSampled() { // 可以继续增加其他条件使得exemplar样本更加典型
        c.(prometheus.ExemplarAdder).AddWithExemplar(1, prometheus.Labels{
              "traceID": sp.TraceID().String(),
        }) // 如果是histogram类型的则类型断言为prometheus.ExemplarObserver
    } else {
        c.Inc()
    }
    

    otlp

    otlp在协议中有Exemplar字段 可以在指标上报时将被采样的span跟指标关联.otlp-SDK是自动进行注入的,因为trace-log-metric 三者共享同样的otlp-context,所以可以不必要进行手工关联

    // A representation of an exemplar, which is a sample input measurement.
    
    // Exemplars also hold information about the environment when the measurement
    
    // was recorded, for example the span and trace ID of the active span when the
    
    // exemplar was recorded.
    
    message Exemplar {
    
    // The set of key/value pairs that were filtered out by the aggregator, but
    
    // recorded alongside the original measurement. Only key/value pairs that were
    
    // filtered out by the aggregator should be included
    
    repeated opentelemetry.proto.common.v1.KeyValue filtered_attributes = 7;
    
    // Labels is deprecated and will be removed soon.
    
    // 1. Old senders and receivers that are not aware of this change will
    
    // continue using the `filtered_labels` field.
    
    // 2. New senders, which are aware of this change MUST send only
    
    // `filtered_attributes`.
    
    // 3. New receivers, which are aware of this change MUST convert this into
    
    // `filtered_labels` by simply converting all int64 values into float.
    
    //
    
    // This field will be removed in ~3 months, on July 1, 2021.
    
    repeated opentelemetry.proto.common.v1.StringKeyValue filtered_labels = 1 [deprecated = true];
    
    // time_unix_nano is the exact time when this exemplar was recorded
    
    //
    
    // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January
    
    // 1970.
    
    fixed64 time_unix_nano = 2;
    
    // The value of the measurement that was recorded. An exemplar is
    
    // considered invalid when one of the recognized value fields is not present
    
    // inside this oneof.
    
    oneof value {
    
    double as_double = 3;
    
    sfixed64 as_int = 6;
    
    }
    
    // (Optional) Span ID of the exemplar trace.
    
    // span_id may be missing if the measurement is not recorded inside a trace
    
    // or if the trace is not sampled.
    
    bytes span_id = 4;
    
    // (Optional) Trace ID of the exemplar trace.
    
    // trace_id may be missing if the measurement is not recorded inside a trace
    
    // or if the trace is not sampled.
    
    bytes trace_id = 5;
    
    }
    

    prometheus存储方式(tjg使用该方式)

    https://github.com/prometheus/prometheus/pull/6635/files prometheus 实现了一种环形连续内存的结构来存储 exemplar,并实现了对应的查询接口

    $ curl -g 'http://localhost:9090/api/v1/query_exemplars?query=test_exemplar_metric_total&start=2020-09-14T15:22:25.479Z&end=020-09-14T15:23:25.479Z'
    {
        "status": "success",
        "data": [
            {
                "seriesLabels": {
                    "__name__": "test_exemplar_metric_total",
                    "instance": "localhost:8090",
                    "job": "prometheus",
                    "service": "bar"
                },
                "exemplars": [
                    {
                        "labels": {
                            "traceID": "EpTxMJ40fUus7aGY"
                        },
                        "value": "6",
                        "timestamp": 1600096945.479,
                    }
                ]
            },
            {
                "seriesLabels": {
                    "__name__": "test_exemplar_metric_total",
                    "instance": "localhost:8090",
                    "job": "prometheus",
                    "service": "foo"
                },
                "exemplars": [
                    {
                        "labels": {
                            "traceID": "Olp9XHlq763ccsfa"
                        },
                        "value": "19",
                        "timestamp": 1600096955.479,
                    },
                    {
                        "labels": {
                            "traceID": "hCtjygkIHwAN9vs4"
                        },
                        "value": "20",
                        "timestamp": 1600096965.489,
                    },
                ]
            }
        ]
    }
    

    image

    日志关联Trace

    日志关联Trace 比较简单 只要在打印日志的时候获取到链路的TraceId和spanId 就可以关联Trace和单条日志了

    Log
    timestamp= TraceId=xxxx SpanId=xxxxx
    Json
    {"trace_id": "xxx", "span_id": "xxx", "log": "xxxx"}
    

    最终清洗入库并标记trace_id和span_id即可实现联动 image

    otlp-SDK 最终可以实现默认关联因为共享Context

    监控存储exemplar

    由于influxdb目前不支持exemplar入库,所以基于现有存储结构监控可以使用ES进行exemplar存储,避免高基线问题 修改如下

    • 相关prometheus的数据解析需要支持exemplar类型的解析并上报
    • transfer需要支持exemplar数据入库到ES
    • saas支持exemplar数据的查询
    Technical solution 
    opened by zzhutianyu 0
Releases(V4.3.5)
  • V4.3.5(Oct 4, 2022)

    What's Changed

    新增的功能如下:

    • 新增支持蓝鲸BCS容器日志采集
    • 新增关联跳转蓝鲸监控metric、log、trace

    本更新还修复了如下的内容:

    • 修复若干问题
    Source code(tar.gz)
    Source code(zip)
  • V4.3.4-334(Aug 25, 2022)

    What's Changed

    新增的功能如下:

    • 变更日志采集下发流程
    • 集群可见范围多样化
    • 新增healthz自身健康检测
    • SLI指标上报

    本更新还修复了如下的内容:

    • 为了解决跨天分裂索引上下文无法定位到的问题
    • 修复grafana 添加variable值为主机时缺少innerip的bug
    • 第三方es中time_field获取失败
    • 修复若干问题

    Full Changelog: https://github.com/TencentBlueKing/bk-log/compare/V4.3.4-rc300...V4.3.4-334

    Source code(tar.gz)
    Source code(zip)
  • V4.3.4-rc300(Jul 1, 2022)

    -【新增】变更日志采集下发流程 -【新增】集群可见范围多样化 -【新增】新增healthz自身健康检测 -【新增】SLI指标上报 -【修复】若干问题

    Source code(tar.gz)
    Source code(zip)
  • V4.3.1-299(Jun 14, 2022)

    What's Changed

    • minor: 第三方es中time_field获取失败 by @liuwenping in https://github.com/TencentBlueKing/bk-log/pull/1090
    • merge: merge from master 为了解决跨天分裂索引上下文无法定位到的问题 by @EvildoerXiaoyy in https://github.com/TencentBlueKing/bk-log/pull/1088
    • bugfix: 修复grafana 添加variable值为主机时缺少innerip的bug by @kiritoscs in https://github.com/TencentBlueKing/bk-log/pull/1098
    • minor: 添加版本文档 by @kiritoscs in https://github.com/TencentBlueKing/bk-log/pull/1104

    Full Changelog: https://github.com/TencentBlueKing/bk-log/compare/V4.3.1-275...V4.3.1-299

    Source code(tar.gz)
    Source code(zip)
  • V4.3.1-275(Apr 29, 2022)

  • V4.3.3-272(Apr 19, 2022)

    修复

    • 修复title不符合2.0的规范问题
    • 修复问题反馈的跳转链接为空白页
    • 修复登录态续期小窗不消失的问题
    Source code(tar.gz)
    Source code(zip)
  • V4.3.1-273(Apr 19, 2022)

  • V4.3.1-267(Apr 15, 2022)

  • V4.3.1-257(Apr 12, 2022)

    修复

    • 修复日志提取成功后没有进行国际化转换的问题
    • 修复仪表盘首页国际化暂未适配的问题
    • 修复跳转为正确官网文档地址的问题
    Source code(tar.gz)
    Source code(zip)
  • V4.3.3-rc240(Mar 23, 2022)

  • V4.3.1-238(Mar 17, 2022)

  • V4.3.1-230(Mar 9, 2022)

  • V4.3.1-227(Mar 9, 2022)

  • V4.3.1.132(Nov 4, 2021)

dash-manufacture-spc-dashboard is a dashboard for monitoring read-time process quality along manufacture production line

In our solution based on plotly, dash and influxdb, the user will firstly generate the specifications for different robots, and then a wide range of interactive visualizations for different machines

Dequn Teng 1 Feb 13, 2022
Progressbar 2 - A progress bar for Python 2 and Python 3 - "pip install progressbar2"

Text progress bar library for Python. Travis status: Coverage: Install The package can be installed through pip (this is the recommended method): pip

Rick van Hattem 795 Dec 18, 2022
Structured Logging for Python

structlog makes logging in Python faster, less painful, and more powerful by adding structure to your log entries. It's up to you whether you want str

Hynek Schlawack 2.3k Jan 05, 2023
pyEventLogger - a simple Python Library for making customized Logs of certain events that occur in a program

pyEventLogger is a simple Python Library for making customized Logs of certain events that occur in a program. The logs can be fully customized and can be printed in colored format or can be stored i

Siddhesh Chavan 2 Nov 03, 2022
metovlogs is a very simple logging library

metovlogs is a very simple logging library. Setup is one line, then you can use it as a drop-in print replacement. Sane and useful log format out of the box. Best for small or early projects.

Azat Akhmetov 1 Mar 01, 2022
Yaml - Loggers are like print() statements

Upgrade your print statements Loggers are like print() statements except they also include loads of other metadata: timestamp msg (same as print!) arg

isaac peterson 38 Jul 20, 2022
A simple, transparent, open-source key logger, written in Python, for tracking your own key-usage statistics.

A simple, transparent, open-source key logger, written in Python, for tracking your own key-usage statistics, originally intended for keyboard layout optimization.

Ga68 56 Jan 03, 2023
Simple and versatile logging library for python 3.6 above

Simple and versatile logging library for python 3.6 above

Miguel 1 Nov 23, 2022
Ultimate Logger - A Discord bot that logs lots of events in a channel written in python

Ultimate Logger - A Discord bot that logs lots of events in a channel written in python

Luca 2 Mar 27, 2022
Monitoring plugin to check disk io with Icinga, Nagios and other compatible monitoring solutions

check_disk_io - Monitor disk io This is a monitoring plugin for Icinga, Nagios and other compatible monitoring solutions to check the disk io. It uses

DinoTools 3 Nov 15, 2022
A colored formatter for the python logging module

Log formatting with colors! colorlog.ColoredFormatter is a formatter for use with Python's logging module that outputs records using terminal colors.

Sam Clements 778 Dec 26, 2022
Debugging-friendly exceptions for Python

Better tracebacks This is a more helpful version of Python's built-in exception message: It shows more code context and the current values of nearby v

Clemens Korndörfer 1.2k Dec 28, 2022
A simple package that allows you to save inputs & outputs as .log files

wolf_dot_log A simple package that allows you to save inputs & outputs as .log files pip install wolf_dot_log pip3 install wolf_dot_log |Instructions|

Alpwuf 1 Nov 16, 2021
This is a wonderful simple python tool used to store the keyboard log.

Keylogger This is a wonderful simple python tool used to store the keyboard log. Record your keys. It will capture passwords and credentials in a comp

Rithin Lehan 2 Nov 25, 2021
APT-Hunter is Threat Hunting tool for windows event logs

APT-Hunter is Threat Hunting tool for windows event logs which made by purple team mindset to provide detect APT movements hidden in the sea of windows event logs to decrease the time to uncover susp

824 Jan 08, 2023
ClusterMonitor - a very simple python script which monitors and records the CPU and RAM consumption of submitted cluster jobs

ClusterMonitor A very simple python script which monitors and records the CPU and RAM consumption of submitted cluster jobs. Usage To start recording

23 Oct 04, 2021
loghandler allows you to easily log messages to multiple endpoints.

loghandler loghandler allows you to easily log messages to multiple endpoints. Using Install loghandler via pip pip install loghandler In your code im

Mathias V. Nielsen 2 Dec 04, 2021
changedetection.io - The best and simplest self-hosted website change detection monitoring service

changedetection.io - The best and simplest self-hosted website change detection monitoring service. An alternative to Visualping, Watchtower etc. Designed for simplicity - the main goal is to simply

7.3k Jan 01, 2023
Monitor and log Network and Disks statistics in MegaBytes per second.

iometrics Monitor and log Network and Disks statistics in MegaBytes per second. Install pip install iometrics Usage Pytorch-lightning integration from

Leo Gallucci 17 May 03, 2022
Outlog it's a library to make logging a simple task

outlog Outlog it's a library to make logging a simple task!. I'm a lazy python user, the times that i do logging on my apps it's hard to do, a lot of

ZSendokame 2 Mar 05, 2022