Hydra: an Extensible Fuzzing Framework for Finding Semantic Bugs in File Systems

Related tags

Deep Learninghydra
Overview

Hydra: An Extensible Fuzzing Framework for Finding Semantic Bugs in File Systems

Paper

Overview

Hydra is a state-of-the-art fuzzing framework for file systems. It provides building blocks for file system fuzzing, including multi-dimensional input mutators, feedback engines, a libOS-based executor, and a bug reproducer with test case minimizer. Developers only need to focus on writing (or bringing in) a checker which defines the core logic for finding the types of bugs of their own interests. Along with the framework, this repository includes our in-house developed crash consistency checker (SymC3), with which 11 new crash consistency bugs were revealed from ext4, Btrfs, F2FS, and from two verified file systems: FSCQ and Yxv6.

Contents

  • General code base

    • src/combined: Hydra input mutator
    • src/lkl/tools/lkl/{FS}-combined-consistency: Hydra LibOS-based Executor (will be downloaded and compiled during setup)
  • Checkers

    • src/emulator: Hydra's in-house crash consistency checker, SymC3

Setup

1. All setup should be done under src

$ cd src

2. Install dependencies

./dep.sh

3. Compile for each file system

$ make build-btrfs-imgwrp
  • We can do the same for other file systems:
$ make build-ext4-imgwrp
$ make build-f2fs-imgwrp
$ make build-xfs-imgwrp
  • (Skip if you want to test the latest kernel) To reproduce bugs presented in the SOSP'19 paper, do the following to back-port LKL to kernel 4.16.
$ cd lkl (pwd: proj_root/src/lkl) # assuming that you are in the src directory
$ make mrproper
$ git pull
$ git checkout v4.16-backport
$ ./compile -t btrfs
$ cd .. (pwd: proj_root/src)

4. Set up environments

$ sudo ./prepare_fuzzing.sh
$ ./prepare_env.sh

5. Run fuzzing (single / multiple instance)

  • Single instance
$ ./run.py -t [fstype] -c [cpu_id] -l [tmpfs_id] -g [fuzz_group]

-t: choose from btrfs, f2fs, ext4, xfs
-c: cpu id to run this fuzzer instance
-l: tmpfs id to store logs (choose one from /tmp/mosbench/tmpfs-separate/)
-g: specify group id for parallel fuzzing, default: 0

e.g., ./run.py -t btrfs -c 4 -l 10 -g 1
Runs btrfs fuzzer, and pins the instance to Core #4.
Logs will be accumulated under /tmp/mosbench/tmpfs-separate/10/log/ .
  • You can also run multiple fuzzers in parallel by doing:
[Terminal 1] ./run.py -t btrfs -c 1 -l 10 -g 1
[Terminal 2] ./run.py -t btrfs -c 2 -l 10 -g 1
[Terminal 3] ./run.py -t btrfs -c 3 -l 10 -g 1
[Terminal 4] ./run.py -t btrfs -c 4 -l 10 -g 1
// all btrfs bug logs will be under /tmp/mosbench/tmpfs-separate/10/log/

[Terminal 5] ./run.py -t f2fs -c 5 -l 11 -g 2
[Terminal 6] ./run.py -t f2fs -c 6 -l 11 -g 2
[Terminal 7] ./run.py -t f2fs -c 7 -l 11 -g 2
[Terminal 8] ./run.py -t f2fs -c 8 -l 11 -g 2
// all f2fs bug logs will be under /tmp/mosbench/tmpfs-separate/11/log/

6. Important note

It is highly encouraged that you use separate input, output, log directories for each file system, unless you are running fuzzers in parallel. If you reuse the same directories from previous testings of other file systems, it won't work properly.

7. Experiments

Please refer to EXPERIMENTS.md for detailed experiment information.

Contacts

Owner
gts3.org ([email protected])
https://gts3.org
gts3.org (<a href=[email protected])">
Aircraft design optimization made fast through modern automatic differentiation

Aircraft design optimization made fast through modern automatic differentiation. Plug-and-play analysis tools for aerodynamics, propulsion, structures, trajectory design, and much more.

Peter Sharpe 394 Dec 23, 2022
TGRNet: A Table Graph Reconstruction Network for Table Structure Recognition

TGRNet: A Table Graph Reconstruction Network for Table Structure Recognition Xue, Wenyuan, et al. "TGRNet: A Table Graph Reconstruction Network for Ta

Wenyuan 68 Jan 04, 2023
Some tentative models that incorporate label propagation to graph neural networks for graph representation learning in nodes, links or graphs.

Some tentative models that incorporate label propagation to graph neural networks for graph representation learning in nodes, links or graphs.

zshicode 1 Nov 18, 2021
GLODISMO: Gradient-Based Learning of Discrete Structured Measurement Operators for Signal Recovery

GLODISMO: Gradient-Based Learning of Discrete Structured Measurement Operators for Signal Recovery This is the code to the paper: Gradient-Based Learn

3 Feb 15, 2022
Clustering is a popular approach to detect patterns in unlabeled data

Visual Clustering Clustering is a popular approach to detect patterns in unlabeled data. Existing clustering methods typically treat samples in a data

Tarek Naous 24 Nov 11, 2022
Hard cater examples from Hopper ICLR paper

CATER-h Honglu Zhou*, Asim Kadav, Farley Lai, Alexandru Niculescu-Mizil, Martin Renqiang Min, Mubbasir Kapadia, Hans Peter Graf (*Contact: honglu.zhou

NECLA ML Group 6 May 11, 2021
PyTorch implementation of DD3D: Is Pseudo-Lidar needed for Monocular 3D Object detection?

PyTorch implementation of DD3D: Is Pseudo-Lidar needed for Monocular 3D Object detection? (ICCV 2021), Dennis Park*, Rares Ambrus*, Vitor Guizilini, Jie Li, and Adrien Gaidon.

Toyota Research Institute - Machine Learning 364 Dec 27, 2022
Official source code of Fast Point Transformer, CVPR 2022

Fast Point Transformer Project Page | Paper This repository contains the official source code and data for our paper: Fast Point Transformer Chunghyun

182 Dec 23, 2022
✨风纪委员会自动投票脚本,利用Github Action帮你进行裁决操作(为了让其他风纪委员有案件可判,本程序从中午12点才开始运行,有需要请自己修改运行时间)

风纪委员会自动投票 本脚本通过使用Github Action来实现B站风纪委员的自动投票功能,喜欢请给我点个STAR吧! 如果你不是风纪委员,在符合风纪委员申请条件的情况下,本脚本会自动帮你申请 投票时间是早上八点,如果有需要请自行修改.github/workflows/Judge.yml中的时间,

Pesy Wu 25 Feb 17, 2021
Implementation of Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning

advantage-weighted-regression Implementation of Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning, by Peng et al. (

Omar D. Domingues 1 Dec 02, 2021
A forwarding MPI implementation that can use any other MPI implementation via an MPI ABI

MPItrampoline MPI wrapper library: MPI trampoline library: MPI integration tests: MPI is the de-facto standard for inter-node communication on HPC sys

Erik Schnetter 31 Dec 22, 2022
Simple tools for logging and visualizing, loading and training

TNT TNT is a library providing powerful dataloading, logging and visualization utilities for Python. It is closely integrated with PyTorch and is desi

1.5k Jan 02, 2023
Defocus Map Estimation and Deblurring from a Single Dual-Pixel Image

Defocus Map Estimation and Deblurring from a Single Dual-Pixel Image This repository is an implementation of the method described in the following pap

21 Dec 15, 2022
CONetV2: Efficient Auto-Channel Size Optimization for CNNs

CONetV2: Efficient Auto-Channel Size Optimization for CNNs Exciting News! CONetV2: Efficient Auto-Channel Size Optimization for CNNs has been accepted

Mahdi S. Hosseini 3 Dec 13, 2021
Data-depth-inference - Data depth inference with python

Welcome! This readme will guide you through the use of the code in this reposito

Marco 3 Feb 08, 2022
Accompanying code for the paper "A Kernel Test for Causal Association via Noise Contrastive Backdoor Adjustment".

#backdoor-HSIC (bd_HSIC) Accompanying code for the paper "A Kernel Test for Causal Association via Noise Contrastive Backdoor Adjustment". To generate

Robert Hu 0 Nov 25, 2021
LBBA-boosted WSOD

LBBA-boosted WSOD Summary Our code is based on ruotianluo/pytorch-faster-rcnn and WSCDN Sincerely thanks for your resources. Newer version of our code

Martin Dong 20 Sep 19, 2022
Convnet transfer - Code for paper How transferable are features in deep neural networks?

How transferable are features in deep neural networks? This repository contains source code necessary to reproduce the results presented in the follow

Jason Yosinski 143 Sep 13, 2022
OpenMMLab 3D Human Parametric Model Toolbox and Benchmark

Introduction English | 简体中文 MMHuman3D is an open source PyTorch-based codebase for the use of 3D human parametric models in computer vision and comput

OpenMMLab 782 Jan 04, 2023
This toolkit provides codes to download and pre-process the SLUE datasets, train the baseline models, and evaluate SLUE tasks.

slue-toolkit We introduce Spoken Language Understanding Evaluation (SLUE) benchmark. This toolkit provides codes to download and pre-process the SLUE

ASAPP Research 39 Sep 21, 2022