A rule-based log analyzer & filter

Overview

Flog

一个根据规则集来处理文本日志的工具。

前言

在日常开发过程中,由于缺乏必要的日志规范,导致很多人乱打一通,一个日志文件夹解压缩后往往有几十万行。

日志泛滥会导致信息密度骤减,给排查问题带来了不小的麻烦。

以前都是用grep之类的工具先挑选出有用的,再逐条进行排查,费时费力。在忍无可忍之后决定写这个工具,根据规则自动分析日志、剔除垃圾信息。

使用方法

安装

python setup.py install

基础用法

flog -r rules.yaml /path/to/1.log /path/to/2.log /path/to/3.log -o /path/to/filtered.log

其中:

  • rules.yaml是规则文件
  • /path/to/x.log是原始的日志文件,支持一次输入多个日志文件。
  • /path/to/filtered.log是过滤后的日志文件,如果不指定文件名(直接一个-o),会自动生成一个。

如果不需要过滤日志内容,只需显示分析结果,可以直接:

flog -r rules.yaml /path/to/your.log

规则语法

基础

name: Rule Name #规则集名称
patterns: #规则列表
  # 单行模式,如果匹配到 ^Hello,就输出 Match Hello
  - match: "^Hello"
    message: "Match Hello"
    action: bypass #保留此条日志(会输出到-o指定的文件中)
    
  # 多行模式,以^Hello开头,以^End结束,输出 Match Hello to End,并丢弃此条日志
  - start: "^Hello"
    end: "^End"
    message: "Match Hello to End"
    action: drop

  - start: "Start"
    start_message: "Match Start" #匹配开始时显示的信息
    end: "End"
    end_messagee: "Match End" #结束时显示的信息

纯过滤模式

name: Rule Name
patterns:
  - match: "^Hello" #删除日志中以Hello开头的行
  - start: "^Hello" #多行模式,删除从Hello到End中间的所有内容
    end: "^End"

过滤日志内容,并输出信息

name: Rule Name
patterns:
  - match: "^Hello" #删除日志中以Hello开头的行
    message: "Match Hello"
    action: drop #删除此行日志

规则嵌套

仅多行模式支持规则嵌套。

name: Rule
patterns:
  - start: "^Response.*{$"
    end: "^}"
    patterns:
      - match: "username = (.*)"
        message: "Current user: {{ capture[0] }}"

输入:

Login Response {
  username = zorro
  userid = 123456
}

输出:

Current user: zorro

action

action字段主要用于控制是否过滤此条日志,仅在指定 -o 参数后生效。 取值范围:【dropbypass】。

为了简化纯过滤类型规则的书写,action默认值的规则如下:

  • 如果规则中包含messagestart_messageend_message字段,action默认为bypass,即输出到文件中。
  • 如果规则中不包含message相关字段,action默认为drop,变成一条纯过滤规则。

message

message 字段用于在标准输出显示信息,并且支持 Jinja 模版语法来自定义输出信息内容,通过它可以实现一些简单的日志分析功能。

目前支持的参数有:

  • lines: (多行模式下)匹配到的所有行
  • content: 匹配到的日志内容
  • captures: 正则表达式(match/start/end)捕获的内容

例如:

name: Rule Name
patterns:
  - match: "^Hello (.*)"
    message: "Match {{captures[0]}}"

如果遇到:"Hello lilei",则会在终端输出"Match lilei"

context

可以把日志中频繁出现的正则提炼出来,放到context字段下,避免复制粘贴多次,例如:

name: Rule Name

context:
  timestamp: "\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}.\\d{3}"
patterns:
  - match: "hello ([^:]*):"
    message: "{{ timestamp }} - {{ captures[0] }}"

输入:2022-04-08 16:52:37.152 hello world: this is a test message
输出:2022-04-08 16:52:37.152 - world

高亮

内置了一些 Jinjafilter,可以在终端高亮输出结果,目前包含:

black, red, green, yellow, blue, purple, cyan, white, bold, light, italic, underline, blink, reverse, strike

例如:

patterns:
  - match: "Error: (.*)"
    message: "{{ captures[0] | red }}"

输入:Error: file not found
输出:file not found

include

支持引入其它规则文件,例如:

name: Rule
include: base #引入同级目录下的 base.yaml 或 base.yml

include支持引入一个或多个文件,例如:

name: Rule
include:
  - base
  - ../base
  - base.yaml
  - base/base1
  - base/base2.yaml
  - ../base.yaml
  - /usr/etc/rules/base.yml

contextpatterns会按照引用顺序依次合并,如果有同名的context,后面的会替换之前的。

License

MIT

Owner
上山打老虎
专业造工具
上山打老虎
YouRefIt: Embodied Reference Understanding with Language and Gesture

YouRefIt: Embodied Reference Understanding with Language and Gesture YouRefIt: Embodied Reference Understanding with Language and Gesture by Yixin Che

16 Jul 11, 2022
SAS: Self-Augmentation Strategy for Language Model Pre-training

SAS: Self-Augmentation Strategy for Language Model Pre-training This repository

Alibaba 5 Nov 02, 2022
A setup script to generate ITK Python Wheels

ITK Python Package This project provides a setup.py script to build ITK Python binary packages and infrastructure to build ITK external module Python

Insight Software Consortium 59 Dec 14, 2022
RTSeg: Real-time Semantic Segmentation Comparative Study

Real-time Semantic Segmentation Comparative Study The repository contains the official TensorFlow code used in our papers: RTSEG: REAL-TIME SEMANTIC S

Mennatullah Siam 592 Nov 18, 2022
The 2nd Version Of Slothybot

SlothyBot Go to this website: "https://bitly.com/SlothyBot" The 2nd Version Of Slothybot. The Bot Has Many Features, Such As: Moderation Commands; Kic

Slothy 0 Jun 01, 2022
Code for the ICML 2021 paper: "ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision"

ViLT Code for the paper: "ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision" Install pip install -r requirements.txt pip

Wonjae Kim 922 Jan 01, 2023
Differentiable Surface Triangulation

Differentiable Surface Triangulation This is our implementation of the paper Differentiable Surface Triangulation that enables optimization for any pe

61 Dec 07, 2022
Lucid Sonic Dreams syncs GAN-generated visuals to music.

Lucid Sonic Dreams Lucid Sonic Dreams syncs GAN-generated visuals to music. By default, it uses NVLabs StyleGAN2, with pre-trained models lifted from

731 Jan 02, 2023
This computer program provides a reference implementation of Lagrangian Monte Carlo in metric induced by the Monge patch

This computer program provides a reference implementation of Lagrangian Monte Carlo in metric induced by the Monge patch. The code was prepared to the final version of the accepted manuscript in AIST

Marcelo Hartmann 2 May 06, 2022
Equipped customers with insights about their EVs Hourly energy consumption and helped predict future charging behavior using LSTM model

Equipped customers with insights about their EVs Hourly energy consumption and helped predict future charging behavior using LSTM model. Designed sample dashboard with insights and recommendation for

Yash 2 Apr 07, 2022
Code for Mesh Convolution Using a Learned Kernel Basis

Mesh Convolution This repository contains the implementation (in PyTorch) of the paper FULLY CONVOLUTIONAL MESH AUTOENCODER USING EFFICIENT SPATIALLY

Yi_Zhou 35 Jan 03, 2023
Deep Q-Learning Network in pytorch (not actively maintained)

pytoch-dqn This project is pytorch implementation of Human-level control through deep reinforcement learning and I also plan to implement the followin

Hung-Tu Chen 342 Jan 01, 2023
PyTorch for Semantic Segmentation

PyTorch for Semantic Segmentation This repository contains some models for semantic segmentation and the pipeline of training and testing models, impl

Zijun Deng 1.7k Jan 06, 2023
This repository implements and evaluates convolutional networks on the Möbius strip as toy model instantiations of Coordinate Independent Convolutional Networks.

Orientation independent Möbius CNNs This repository implements and evaluates convolutional networks on the Möbius strip as toy model instantiations of

Maurice Weiler 59 Dec 09, 2022
A stock generator that assess a list of stocks and returns the best stocks for investing and money allocations based on users choices of volatility, duration and number of stocks

Stock-Generator Please visit "Stock Generator.ipynb" for a clearer view and "Stock Generator.py" for scripts. The stock generator is designed to allow

jmengnyay 1 Aug 02, 2022
Pytorch implementations of the paper Value Functions Factorization with Latent State Information Sharing in Decentralized Multi-Agent Policy Gradients

LSF-SAC Pytorch implementations of the paper Value Functions Factorization with Latent State Information Sharing in Decentralized Multi-Agent Policy G

Hanhan 2 Aug 14, 2022
Quantile Regression DQN a Minimal Working Example, Distributional Reinforcement Learning with Quantile Regression

Quantile Regression DQN Quantile Regression DQN a Minimal Working Example, Distributional Reinforcement Learning with Quantile Regression (https://arx

Arsenii Senya Ashukha 80 Sep 17, 2022
Unofficial PyTorch implementation of MobileViT based on paper "MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer".

MobileViT RegNet Unofficial PyTorch implementation of MobileViT based on paper MOBILEVIT: LIGHT-WEIGHT, GENERAL-PURPOSE, AND MOBILE-FRIENDLY VISION TR

Hong-Jia Chen 91 Dec 02, 2022
Mmrotate - OpenMMLab Rotated Object Detection Benchmark

OpenMMLab website HOT OpenMMLab platform TRY IT OUT 📘 Documentation | 🛠️ Insta

OpenMMLab 1.2k Jan 04, 2023
Lightweight library to build and train neural networks in Theano

Lasagne Lasagne is a lightweight library to build and train neural networks in Theano. Its main features are: Supports feed-forward networks such as C

Lasagne 3.8k Dec 29, 2022