FFPuppet is a Python module that automates browser process related tasks to aid in fuzzing

Overview

FFPuppet

Task Status codecov Matrix PyPI

FFPuppet is a Python module that automates browser process related tasks to aid in fuzzing. Happy bug hunting!

Are you fuzzing the browser? Grizzly can help.

Installation

To install the latest version from PyPI
pip install ffpuppet
Xvfb on Linux

On Linux xvfb can be used in order to run headless (this is not the same as Firefox's -headless mode).

To install xvfb on Ubuntu run:

apt-get install xvfb
Installing minidump_stackwalk

minidump_stackwalk is used to extract a crash report when the browser crashes without a debugger (GDB/Valgrind) or instrumentation (ASan). If desired, minidump_stackwalk should be installed in the users path after obtaining it from FirefoxCI. Choose the appropriate tool for your platform (linux64-minidump-stackwalk, macosx64-minidump-stackwalk, win32-minidump-stackwalk, etc.), choose latest, then extract the executable from the downloaded tar archive.

Browser Builds

If you are looking for builds to use with FFPuppet there are a few options.

Downloading builds

fuzzfetch is the recommended method for obtaining builds and is also very helpful in automation.

Taskcluster has a collection of many different build types for multiple platforms and branches. An index of the latest mozilla-central builds can be found here.

Build your own

If you would like to compile your own, build instructions can be found here.

Usage

Once installed FFPuppet can be run using the following command:

python -m ffpuppet
usage: ffpuppet [-h] [-d] [--log-level LOG_LEVEL] [-e EXTENSION] [-p PREFS]
                [-P PROFILE] [-u URL] [--xvfb] [-a ABORT_TOKEN]
                [--launch-timeout LAUNCH_TIMEOUT] [-l LOGS]
                [--log-limit LOG_LIMIT] [-m MEMORY]
                [--poll-interval POLL_INTERVAL] [--save-all]
                [--gdb | --pernosco | --rr | --valgrind]
                binary

FFPuppet - Firefox process launcher and log collector. Happy bug hunting!

positional arguments:
  binary                Firefox binary to launch

optional arguments:
  -h, --help            show this help message and exit
  -d, --display-logs    Display summary of browser logs on process exit.
  --log-level LOG_LEVEL
                        Configure console logging. Options: DEBUG, INFO, WARN,
                        ERROR (default: INFO)

Browser Configuration:
  -e EXTENSION, --extension EXTENSION
                        Install extensions. Specify the path to the xpi or the
                        directory containing the unpacked extension.
  -p PREFS, --prefs PREFS
                        Custom prefs.js file to use (default: profile default)
  -P PROFILE, --profile PROFILE
                        Profile to use. This is non-destructive. A copy of the
                        target profile will be used. (default: temporary
                        profile)
  -u URL, --url URL     Server URL or path to local file to load.
  --xvfb                Use Xvfb. (Linux only)

Issue Detection & Reporting:
  -a ABORT_TOKEN, --abort-token ABORT_TOKEN
                        Scan the browser logs for the given value and close
                        browser if detected. For example '-a ###!!!
                        ASSERTION:' would be used to detect soft assertions.
  --launch-timeout LAUNCH_TIMEOUT
                        Number of seconds to wait for the browser to become
                        responsive after launching. (default: 300)
  -l LOGS, --logs LOGS  Location to save browser logs. A sub-directory
                        containing the browser logs will be created.
  --log-limit LOG_LIMIT
                        Browser log file size limit in MBs (default: 0, no
                        limit)
  -m MEMORY, --memory MEMORY
                        Browser memory limit in MBs (default: 0, no limit)
  --poll-interval POLL_INTERVAL
                        Delay between checks for results (default: 0.5)
  --save-all            Always save logs. By default logs are saved only when
                        an issue is detected.

Available Debuggers:
  --gdb                 Use GDB. (Linux only)
  --pernosco            Use rr. Trace intended to be submitted to Pernosco. (Linux only)
  --rr                  Use rr. (Linux only)
  --valgrind            Use Valgrind. (Linux only)

Replaying a test case
python -m ffpuppet 
   
     -p 
    
      -d -u 
     

     
    
   

This will open the provided test case file in Firefox using the provided prefs.js file and any log data (stderr, stdout, ASan logs... etc) will be dumped to the console when the browser process terminates. Grizzly Replay is recommended for replaying test cases.

Prefs.js files

prefs.js files that can be used for fuzzing or other automated testing can be generated with PrefPicker.

Comments
  • ffpuppet does not work with older version of Firefox

    ffpuppet does not work with older version of Firefox

    I have build Firefox 57.0 and Firefox 63.0.3 (with ASAN) by myself. Running python -m ffpuppet ../../firefox-57.0/objdir-ff-asan/dist/bin/firefox -p $CODE/browsers/prefs.js -d -u /home/ug16zy2/test.html -v --xvfb --log ./out/ gives:

    I ffpuppet [2020-01-01 19:55:44] Launching Firefox... D ffpuppet [2020-01-01 19:55:44] requested location: '/home/ug16zy2/test.html' D ffpuppet [2020-01-01 19:55:44] launch timeout: 300 D ffpuppet [2020-01-01 19:55:44] profile directory: '/tmp/ffprof_VspYuS' D ffpuppet [2020-01-01 19:55:44] using prefs.js: '/home/ug16zy2/fuzz/browsers/prefs.js' D ffpuppet [2020-01-01 19:55:44] launch command: '/home/ug16zy2/firefox-57.0/objdir-ff-asan/dist/bin/firefox -no-remote -profile /tmp/ffprof_VspYuS http://127.0.0.1:46027' D ffpuppet [2020-01-01 19:55:44] launched firefox with pid: 21507 D ffpuppet [2020-01-01 19:55:48] crash report found I ffpuppet [2020-01-01 19:55:48] Shutting down... D ffpuppet [2020-01-01 19:55:48] close(force_close=False) called D ffpuppet [2020-01-01 19:55:48] browser pid: 21507 D ffpuppet [2020-01-01 19:55:48] 1 crash report(s) are available D ffpuppet [2020-01-01 19:55:53] reviewing 0 check(s) D ffpuppet [2020-01-01 19:55:53] scan_path '/tmp/ffprof_VspYuS/minidumps' does not exist D ffpuppet [2020-01-01 19:55:53] exit reason code 'ALERT' I ffpuppet [2020-01-01 19:55:53] Firefox process is closed. (Reason: 'ALERT') I ffpuppet [2020-01-01 19:55:53] Saving logs to '/home/ug16zy2/fuzz/ffpuppet/out' D ffpuppet [2020-01-01 19:55:53] save_logs() called, dest='./out/', logs_only=False, meta=False I ffpuppet [2020-01-01 19:55:53] Displaying logs...

    and the ASAN log is:

    === Dumping 'log_ffp_asan_21493.log.21507.txt' (1.83KB) ==21507==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7f9c17e13646 bp 0x7fffd3f6b340 sp 0x7fffd3f6aab8 T0) ==21507==The signal is caused by a READ memory access. ==21507==Hint: address points to the zero page. #0 0x7f9c17e13645 (/lib/x86_64-linux-gnu/libc.so.6+0xb1645) #1 0x7f9c18ef7146 (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x42146) #2 0x7f9c11652660 (/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x39660) #3 0x7f9c11652806 in g_dgettext (/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x39806) #4 0x7f9c144f13ae in gtk_get_option_group (/usr/lib/x86_64-linux-gnu/libgtk-3.so.0+0x22f3ae) #5 0x7f9c144f15fb in gtk_parse_args (/usr/lib/x86_64-linux-gnu/libgtk-3.so.0+0x22f5fb) #6 0x7f9bfb69192f in XREMain::XRE_mainStartup(bool*) /home/ug16zy2/firefox-57.0/toolkit/xre/nsAppRunner.cpp:3911 #7 0x7f9bfb698931 in XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) /home/ug16zy2/firefox-57.0/toolkit/xre/nsAppRunner.cpp:4852 #8 0x7f9bfb699030 in XRE_main(int, char**, mozilla::BootstrapConfig const&) /home/ug16zy2/firefox-57.0/toolkit/xre/nsAppRunner.cpp:4962 #9 0x7f9bfb6c3d6d in mozilla::BootstrapImpl::XRE_main(int, char**, mozilla::BootstrapConfig const&) /home/ug16zy2/firefox-57.0/toolkit/xre/Bootstrap.cpp:45 #10 0x55fa5cd415df in do_main /home/ug16zy2/firefox-57.0/browser/app/nsBrowserApp.cpp:236 #11 0x55fa5cd41c87 in main /home/ug16zy2/firefox-57.0/browser/app/nsBrowserApp.cpp:309 #12 0x7f9c17d83b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96) #13 0x55fa5cd408d9 in _start (/home/ug16zy2/firefox-57.0/objdir-ff-asan/dist/bin/firefox+0x3e8d9) AddressSanitizer can not provide additional info. SUMMARY: AddressSanitizer: SEGV (/lib/x86_64-linux-gnu/libc.so.6+0xb1645) ==21507==ABORTING

    When I run FFPuppet with /usr/bin/firefox, it works fine and Firefox instance pops up with the input HTML file.

    Does FFPuppet support older version of Firefox?

    opened by ZihanYe 6
  • Added the ability to specify a timeout in order to terminate during hangups.

    Added the ability to specify a timeout in order to terminate during hangups.

    This adds a timeout parameter (-w) by implementing a check called CheckRunningTimeout. The timeout lets someone specify a number of seconds before terminating the browser. Pretty basic really.

    enhancement wontfix 
    opened by arizvisa 6
  • Finish typing support.

    Finish typing support.

    This finishes the work by @nth10sd in #115 . Much of the nicer syntax depends on __future__ annotations which has other caveats, so I've backported this to 3.6 for now.

    This also enables mypy in pre-commit, enables Python 3.10 CI, and enables lint across all Python versions in CI.

    Supersedes #115.

    opened by jschwartzentruber 4
  • Run minidump_stackwalk on any minidumps in the profile directory …

    Run minidump_stackwalk on any minidumps in the profile directory …

    …and append to the log.

    • only works if minidump_stackwalk is in path, otherwise a warning is printed if a minidump is found
    • remove obsolete and incomplete breakpad symbolization
    • symbolize argument to save_log() and clone_log() has been removed
    opened by jschwartzentruber 4
  • Add mypy type information and bump minimum Python version to 3.7

    Add mypy type information and bump minimum Python version to 3.7

    This PR adds mypy type information for ffpuppet, which helps the PR for FuzzManager type info as well. I also deleted type information if present in the comments, since some were already inaccurate IIRC.

    It bumps the minimum Python version to 3.7 since we import annotations from __future__.

    GitHub Actions CI is here.

    To-do:

    • [x] Push a py.typed file on completion
    opened by nth10sd 3
  • Fix #18: --url localfile doesn't work on Windows.

    Fix #18: --url localfile doesn't work on Windows.

    After this change yields:

    windows:
    D ffpuppet [2017-09-07 14:55:24] sending response with redirect url: 'file:///C:/Users/user/Desktop/tc.html'
    linux:
    D ffpuppet [2017-09-07 15:01:45] sending response with redirect url: 'file:/home/user/Desktop/tc.html'
    

    Both redirects work properly.

    opened by jschwartzentruber 3
  • Add support for arbitrary/multiple extensions.

    Add support for arbitrary/multiple extensions.

    Extensions are not going away, and we should support installing them for testing even if we stop using them regularly (which is not planned). This supports arbitrary add-ons by looking up the extension ID in the manifest (may be fragile for legacy add-ons because of XML namespaces).

    Fixes #4.

    enhancement 
    opened by jschwartzentruber 2
  • Only set max_allocation_size_mb and soft_rss_limit_mb for ASan and TSan

    Only set max_allocation_size_mb and soft_rss_limit_mb for ASan and TSan

    Setting these values in UBSAN_OPTIONS as well is a foot gun. It can can lead to confusion if set for multiple sanitizers. It can still be set manually in the environment.

    opened by tysmith 1
  • Reduce the shutdown wait after a crash

    Reduce the shutdown wait after a crash

    This will reduce the wait period for the browser processes to close after a crash log is detected.

    The browser should shutdown quickly after a crash report is written.

    opened by tysmith 1
  • Consider randomizing the resolution passed to xvfb

    Consider randomizing the resolution passed to xvfb

    It might be worth considering setting a random resolution to xvfb as this might affect layout. Similar concept to the TTF Fuzzer outlined here:

    https://media.blackhat.com/bh-eu-12/Lee/bh-eu-12-Lee-GDI_Font_Fuzzing-WP.pdf

    opened by pyoor 1
Releases(0.9.2)
Owner
Mozilla Fuzzing Security
Fuzzing projects at the Mozilla Corporation
Mozilla Fuzzing Security
自动化爬取并自动测试所有swagger-ui.html显示的接口

swagger-hack 在测试中偶尔会碰到swagger泄露 常见的泄露如图: 有的泄露接口特别多,每一个都手动去试根本试不过来 于是用python写了个脚本自动爬取所有接口,配置好传参发包访问 原理是首先抓取http://url/swagger-resources 获取到有哪些标准及对应的文档地

jayus 534 Dec 29, 2022
Minimal example of getting Django + PyTest running on GitHub Actions

Minimal Django + Pytest + GitHub Actions example This minimal example shows you how you can runs pytest on your Django app on every commit using GitHu

Matt Segal 5 Sep 19, 2022
A library for generating fake data and populating database tables.

Knockoff Factory A library for generating mock data and creating database fixtures that can be used for unit testing. Table of content Installation Ch

Nike Inc. 30 Sep 23, 2022
1st Solution to QQ Browser 2021 AIAC Track 2

1st Solution to QQ Browser 2021 AIAC Track 2 This repository is the winning solution to QQ Browser 2021 AI Algorithm Competition Track 2 Automated Hyp

DAIR Lab 24 Sep 10, 2022
Simple frontend TypeScript testing utility

TSFTest Simple frontend TypeScript testing utility. Installation Install webpack in your project directory: npm install --save-dev webpack webpack-cli

2 Nov 09, 2021
Python Rest Testing

pyresttest Table of Contents What Is It? Status Installation Sample Test Examples Installation How Do I Use It? Running A Simple Test Using JSON Valid

Sam Van Oort 1.1k Dec 28, 2022
The Social-Engineer Toolkit (SET) repository from TrustedSec - All new versions of SET will be deployed here.

💼 The Social-Engineer Toolkit (SET) 💼 Copyright 2020 The Social-Engineer Toolkit (SET) Written by: David Kennedy (ReL1K) @HackingDave Company: Trust

trustedsec 8.4k Dec 31, 2022
FakeDataGen is a Full Valid Fake Data Generator.

FakeDataGen is a Full Valid Fake Data Generator. This tool helps you to create fake accounts (in Spanish format) with fully valid data. Within this in

Joel GM 64 Dec 12, 2022
Coverage plugin for pytest.

Overview docs tests package This plugin produces coverage reports. Compared to just using coverage run this plugin does some extras: Subprocess suppor

pytest-dev 1.4k Dec 29, 2022
🎓 Stepik Academy Автоматизация тестирования на Python

🎓 Stepik Academy Автоматизация тестирования на Python Запуск тестов выполняется в командной строке: pytest -v --tb=line --language=en --alluredir=all

Sergey 1 Dec 03, 2021
An improbable web debugger through WebSockets

wdb - Web Debugger Description wdb is a full featured web debugger based on a client-server architecture. The wdb server which is responsible of manag

Kozea 1.6k Dec 09, 2022
A collection of testing examples using pytest and many other libreris

Effective testing with Python This project was created for PyConEs 2021 Check out the test samples at tests Check out the slides at slides (markdown o

Héctor Canto 10 Oct 23, 2022
A python bot using the Selenium library to auto-buy specified sneakers on the nike.com website.

Sneaker-Bot-UK A python bot using the Selenium library to auto-buy specified sneakers on the nike.com website. This bot is still in development and is

Daniel Hinds 4 Dec 14, 2022
A mocking library for requests

httmock A mocking library for requests for Python 2.7 and 3.4+. Installation pip install httmock Or, if you are a Gentoo user: emerge dev-python/httm

Patryk Zawadzki 452 Dec 28, 2022
Compiles python selenium script to be a Window's executable

Problem Statement Setting up a Python project can be frustrating for non-developers. From downloading the right version of python, setting up virtual

Jerry Ng 8 Jan 09, 2023
Spam the buzzer and upgrade automatically - Selenium

CookieClicker Usage: Let's check your chrome navigator version : Consequently, you have to : download the right chromedriver in the follow link : http

Iliam Amara 1 Nov 22, 2021
A browser automation framework and ecosystem.

Selenium Selenium is an umbrella project encapsulating a variety of tools and libraries enabling web browser automation. Selenium specifically provide

Selenium 25.5k Jan 01, 2023
A wrapper for webdriver that is a jumping off point for web automation.

Webdriver Automation Plus ===================================== Description: Tests the user can save messages then find them in search and Saved items

1 Nov 08, 2021
Descriptor Vector Exchange

Descriptor Vector Exchange This repo provides code for learning dense landmarks without supervision. Our approach is described in the ICCV 2019 paper

James Thewlis 74 Nov 29, 2022