FFPuppet is a Python module that automates browser process related tasks to aid in fuzzing

Overview

FFPuppet

Task Status codecov Matrix PyPI

FFPuppet is a Python module that automates browser process related tasks to aid in fuzzing. Happy bug hunting!

Are you fuzzing the browser? Grizzly can help.

Installation

To install the latest version from PyPI
pip install ffpuppet
Xvfb on Linux

On Linux xvfb can be used in order to run headless (this is not the same as Firefox's -headless mode).

To install xvfb on Ubuntu run:

apt-get install xvfb
Installing minidump_stackwalk

minidump_stackwalk is used to extract a crash report when the browser crashes without a debugger (GDB/Valgrind) or instrumentation (ASan). If desired, minidump_stackwalk should be installed in the users path after obtaining it from FirefoxCI. Choose the appropriate tool for your platform (linux64-minidump-stackwalk, macosx64-minidump-stackwalk, win32-minidump-stackwalk, etc.), choose latest, then extract the executable from the downloaded tar archive.

Browser Builds

If you are looking for builds to use with FFPuppet there are a few options.

Downloading builds

fuzzfetch is the recommended method for obtaining builds and is also very helpful in automation.

Taskcluster has a collection of many different build types for multiple platforms and branches. An index of the latest mozilla-central builds can be found here.

Build your own

If you would like to compile your own, build instructions can be found here.

Usage

Once installed FFPuppet can be run using the following command:

python -m ffpuppet
usage: ffpuppet [-h] [-d] [--log-level LOG_LEVEL] [-e EXTENSION] [-p PREFS]
                [-P PROFILE] [-u URL] [--xvfb] [-a ABORT_TOKEN]
                [--launch-timeout LAUNCH_TIMEOUT] [-l LOGS]
                [--log-limit LOG_LIMIT] [-m MEMORY]
                [--poll-interval POLL_INTERVAL] [--save-all]
                [--gdb | --pernosco | --rr | --valgrind]
                binary

FFPuppet - Firefox process launcher and log collector. Happy bug hunting!

positional arguments:
  binary                Firefox binary to launch

optional arguments:
  -h, --help            show this help message and exit
  -d, --display-logs    Display summary of browser logs on process exit.
  --log-level LOG_LEVEL
                        Configure console logging. Options: DEBUG, INFO, WARN,
                        ERROR (default: INFO)

Browser Configuration:
  -e EXTENSION, --extension EXTENSION
                        Install extensions. Specify the path to the xpi or the
                        directory containing the unpacked extension.
  -p PREFS, --prefs PREFS
                        Custom prefs.js file to use (default: profile default)
  -P PROFILE, --profile PROFILE
                        Profile to use. This is non-destructive. A copy of the
                        target profile will be used. (default: temporary
                        profile)
  -u URL, --url URL     Server URL or path to local file to load.
  --xvfb                Use Xvfb. (Linux only)

Issue Detection & Reporting:
  -a ABORT_TOKEN, --abort-token ABORT_TOKEN
                        Scan the browser logs for the given value and close
                        browser if detected. For example '-a ###!!!
                        ASSERTION:' would be used to detect soft assertions.
  --launch-timeout LAUNCH_TIMEOUT
                        Number of seconds to wait for the browser to become
                        responsive after launching. (default: 300)
  -l LOGS, --logs LOGS  Location to save browser logs. A sub-directory
                        containing the browser logs will be created.
  --log-limit LOG_LIMIT
                        Browser log file size limit in MBs (default: 0, no
                        limit)
  -m MEMORY, --memory MEMORY
                        Browser memory limit in MBs (default: 0, no limit)
  --poll-interval POLL_INTERVAL
                        Delay between checks for results (default: 0.5)
  --save-all            Always save logs. By default logs are saved only when
                        an issue is detected.

Available Debuggers:
  --gdb                 Use GDB. (Linux only)
  --pernosco            Use rr. Trace intended to be submitted to Pernosco. (Linux only)
  --rr                  Use rr. (Linux only)
  --valgrind            Use Valgrind. (Linux only)

Replaying a test case
python -m ffpuppet 
   
     -p 
    
      -d -u 
     

     
    
   

This will open the provided test case file in Firefox using the provided prefs.js file and any log data (stderr, stdout, ASan logs... etc) will be dumped to the console when the browser process terminates. Grizzly Replay is recommended for replaying test cases.

Prefs.js files

prefs.js files that can be used for fuzzing or other automated testing can be generated with PrefPicker.

Comments
  • ffpuppet does not work with older version of Firefox

    ffpuppet does not work with older version of Firefox

    I have build Firefox 57.0 and Firefox 63.0.3 (with ASAN) by myself. Running python -m ffpuppet ../../firefox-57.0/objdir-ff-asan/dist/bin/firefox -p $CODE/browsers/prefs.js -d -u /home/ug16zy2/test.html -v --xvfb --log ./out/ gives:

    I ffpuppet [2020-01-01 19:55:44] Launching Firefox... D ffpuppet [2020-01-01 19:55:44] requested location: '/home/ug16zy2/test.html' D ffpuppet [2020-01-01 19:55:44] launch timeout: 300 D ffpuppet [2020-01-01 19:55:44] profile directory: '/tmp/ffprof_VspYuS' D ffpuppet [2020-01-01 19:55:44] using prefs.js: '/home/ug16zy2/fuzz/browsers/prefs.js' D ffpuppet [2020-01-01 19:55:44] launch command: '/home/ug16zy2/firefox-57.0/objdir-ff-asan/dist/bin/firefox -no-remote -profile /tmp/ffprof_VspYuS http://127.0.0.1:46027' D ffpuppet [2020-01-01 19:55:44] launched firefox with pid: 21507 D ffpuppet [2020-01-01 19:55:48] crash report found I ffpuppet [2020-01-01 19:55:48] Shutting down... D ffpuppet [2020-01-01 19:55:48] close(force_close=False) called D ffpuppet [2020-01-01 19:55:48] browser pid: 21507 D ffpuppet [2020-01-01 19:55:48] 1 crash report(s) are available D ffpuppet [2020-01-01 19:55:53] reviewing 0 check(s) D ffpuppet [2020-01-01 19:55:53] scan_path '/tmp/ffprof_VspYuS/minidumps' does not exist D ffpuppet [2020-01-01 19:55:53] exit reason code 'ALERT' I ffpuppet [2020-01-01 19:55:53] Firefox process is closed. (Reason: 'ALERT') I ffpuppet [2020-01-01 19:55:53] Saving logs to '/home/ug16zy2/fuzz/ffpuppet/out' D ffpuppet [2020-01-01 19:55:53] save_logs() called, dest='./out/', logs_only=False, meta=False I ffpuppet [2020-01-01 19:55:53] Displaying logs...

    and the ASAN log is:

    === Dumping 'log_ffp_asan_21493.log.21507.txt' (1.83KB) ==21507==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7f9c17e13646 bp 0x7fffd3f6b340 sp 0x7fffd3f6aab8 T0) ==21507==The signal is caused by a READ memory access. ==21507==Hint: address points to the zero page. #0 0x7f9c17e13645 (/lib/x86_64-linux-gnu/libc.so.6+0xb1645) #1 0x7f9c18ef7146 (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x42146) #2 0x7f9c11652660 (/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x39660) #3 0x7f9c11652806 in g_dgettext (/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x39806) #4 0x7f9c144f13ae in gtk_get_option_group (/usr/lib/x86_64-linux-gnu/libgtk-3.so.0+0x22f3ae) #5 0x7f9c144f15fb in gtk_parse_args (/usr/lib/x86_64-linux-gnu/libgtk-3.so.0+0x22f5fb) #6 0x7f9bfb69192f in XREMain::XRE_mainStartup(bool*) /home/ug16zy2/firefox-57.0/toolkit/xre/nsAppRunner.cpp:3911 #7 0x7f9bfb698931 in XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) /home/ug16zy2/firefox-57.0/toolkit/xre/nsAppRunner.cpp:4852 #8 0x7f9bfb699030 in XRE_main(int, char**, mozilla::BootstrapConfig const&) /home/ug16zy2/firefox-57.0/toolkit/xre/nsAppRunner.cpp:4962 #9 0x7f9bfb6c3d6d in mozilla::BootstrapImpl::XRE_main(int, char**, mozilla::BootstrapConfig const&) /home/ug16zy2/firefox-57.0/toolkit/xre/Bootstrap.cpp:45 #10 0x55fa5cd415df in do_main /home/ug16zy2/firefox-57.0/browser/app/nsBrowserApp.cpp:236 #11 0x55fa5cd41c87 in main /home/ug16zy2/firefox-57.0/browser/app/nsBrowserApp.cpp:309 #12 0x7f9c17d83b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96) #13 0x55fa5cd408d9 in _start (/home/ug16zy2/firefox-57.0/objdir-ff-asan/dist/bin/firefox+0x3e8d9) AddressSanitizer can not provide additional info. SUMMARY: AddressSanitizer: SEGV (/lib/x86_64-linux-gnu/libc.so.6+0xb1645) ==21507==ABORTING

    When I run FFPuppet with /usr/bin/firefox, it works fine and Firefox instance pops up with the input HTML file.

    Does FFPuppet support older version of Firefox?

    opened by ZihanYe 6
  • Added the ability to specify a timeout in order to terminate during hangups.

    Added the ability to specify a timeout in order to terminate during hangups.

    This adds a timeout parameter (-w) by implementing a check called CheckRunningTimeout. The timeout lets someone specify a number of seconds before terminating the browser. Pretty basic really.

    enhancement wontfix 
    opened by arizvisa 6
  • Finish typing support.

    Finish typing support.

    This finishes the work by @nth10sd in #115 . Much of the nicer syntax depends on __future__ annotations which has other caveats, so I've backported this to 3.6 for now.

    This also enables mypy in pre-commit, enables Python 3.10 CI, and enables lint across all Python versions in CI.

    Supersedes #115.

    opened by jschwartzentruber 4
  • Run minidump_stackwalk on any minidumps in the profile directory …

    Run minidump_stackwalk on any minidumps in the profile directory …

    …and append to the log.

    • only works if minidump_stackwalk is in path, otherwise a warning is printed if a minidump is found
    • remove obsolete and incomplete breakpad symbolization
    • symbolize argument to save_log() and clone_log() has been removed
    opened by jschwartzentruber 4
  • Add mypy type information and bump minimum Python version to 3.7

    Add mypy type information and bump minimum Python version to 3.7

    This PR adds mypy type information for ffpuppet, which helps the PR for FuzzManager type info as well. I also deleted type information if present in the comments, since some were already inaccurate IIRC.

    It bumps the minimum Python version to 3.7 since we import annotations from __future__.

    GitHub Actions CI is here.

    To-do:

    • [x] Push a py.typed file on completion
    opened by nth10sd 3
  • Fix #18: --url localfile doesn't work on Windows.

    Fix #18: --url localfile doesn't work on Windows.

    After this change yields:

    windows:
    D ffpuppet [2017-09-07 14:55:24] sending response with redirect url: 'file:///C:/Users/user/Desktop/tc.html'
    linux:
    D ffpuppet [2017-09-07 15:01:45] sending response with redirect url: 'file:/home/user/Desktop/tc.html'
    

    Both redirects work properly.

    opened by jschwartzentruber 3
  • Add support for arbitrary/multiple extensions.

    Add support for arbitrary/multiple extensions.

    Extensions are not going away, and we should support installing them for testing even if we stop using them regularly (which is not planned). This supports arbitrary add-ons by looking up the extension ID in the manifest (may be fragile for legacy add-ons because of XML namespaces).

    Fixes #4.

    enhancement 
    opened by jschwartzentruber 2
  • Only set max_allocation_size_mb and soft_rss_limit_mb for ASan and TSan

    Only set max_allocation_size_mb and soft_rss_limit_mb for ASan and TSan

    Setting these values in UBSAN_OPTIONS as well is a foot gun. It can can lead to confusion if set for multiple sanitizers. It can still be set manually in the environment.

    opened by tysmith 1
  • Reduce the shutdown wait after a crash

    Reduce the shutdown wait after a crash

    This will reduce the wait period for the browser processes to close after a crash log is detected.

    The browser should shutdown quickly after a crash report is written.

    opened by tysmith 1
  • Consider randomizing the resolution passed to xvfb

    Consider randomizing the resolution passed to xvfb

    It might be worth considering setting a random resolution to xvfb as this might affect layout. Similar concept to the TTF Fuzzer outlined here:

    https://media.blackhat.com/bh-eu-12/Lee/bh-eu-12-Lee-GDI_Font_Fuzzing-WP.pdf

    opened by pyoor 1
Releases(0.9.2)
Owner
Mozilla Fuzzing Security
Fuzzing projects at the Mozilla Corporation
Mozilla Fuzzing Security
reCaptchaBypasser For Bypass Any reCaptcha For Selenium Python

reCaptchaBypasser ' Usage : from selenium import webdriver from reCaptchaBypasser import reCaptchaScraper import time driver = webdriver.chrome(execu

Dr.Linux 8 Dec 17, 2022
Mock smart contracts for writing Ethereum test suites

Mock smart contracts for writing Ethereum test suites This package contains comm

Trading Strategy 222 Jan 04, 2023
A small automated test structure using python to test *.cpp codes

Get Started Insert C++ Codes Add Test Code Run Test Samples Check Coverages Insert C++ Codes you can easily add c++ files in /inputs directory there i

Alireza Zahiri 2 Aug 03, 2022
Pymox - open source mock object framework for Python

Pymox is an open source mock object framework for Python. First Steps Installation Tutorial Documentation http://pymox.readthedocs.io/en/latest/index.

Ivan Rocha 7 Feb 02, 2022
Fidelipy - Semi-automated trading on fidelity.com

fidelipy fidelipy is a simple Python 3.7+ library for semi-automated trading on fidelity.com. The scope is limited to the Trade Stocks/ETFs simplified

Darik Harter 8 May 10, 2022
Python scripts for a generic performance testing infrastructure using Locust.

TODOs Reference to published paper or online version of it loadtest_plotter.py: Cleanup and reading data from files ARS_simulation.py: Cleanup, docume

Juri Tomak 3 Dec 15, 2022
A Simple Unit Test Matcher Library for Python 3

pychoir - Python Test Matchers for humans Super duper low cognitive overhead matching for Python developers reading or writing tests. Implemented in p

Antti Kajander 15 Sep 14, 2022
Show coverage stats online via coveralls.io

Coveralls for Python Test Status: Version Info: Compatibility: Misc: coveralls.io is a service for publishing your coverage stats online. This package

Kevin James 499 Dec 28, 2022
A small faсade for the standard python mocker library to make it user-friendly

unittest-mocker Inspired by the pytest-mock, but written from scratch for using with unittest and convenient tool - patch_class Installation pip insta

Vertliba V.V. 6 Jun 10, 2022
A simple tool to test internet stability.

pingtest Description A personal project for testing internet stability, intended for use in Linux and Windows.

chris 0 Oct 17, 2021
Given some test cases, this program automatically queries the oracle and tests your Cshanty compiler!

The Diviner A complement to The Oracle for compilers class. Given some test cases, this program automatically queries the oracle and tests your compil

Grant Holmes 2 Jan 29, 2022
Mixer -- Is a fixtures replacement. Supported Django, Flask, SqlAlchemy and custom python objects.

The Mixer is a helper to generate instances of Django or SQLAlchemy models. It's useful for testing and fixture replacement. Fast and convenient test-

Kirill Klenov 871 Dec 25, 2022
Lightweight, scriptable browser as a service with an HTTP API

Splash - A javascript rendering service Splash is a javascript rendering service with an HTTP API. It's a lightweight browser with an HTTP API, implem

Scrapinghub 3.8k Jan 03, 2023
API Test Automation with Requests and Pytest

api-testing-requests-pytest Install Make sure you have Python 3 installed on your machine. Then: 1.Install pipenv sudo apt-get install pipenv 2.Go to

Sulaiman Haque 2 Nov 21, 2021
Percy visual testing for Python Selenium

percy-selenium-python Percy visual testing for Python Selenium. Installation npm install @percy/cli: $ npm install --save-dev @percy/cli pip install P

Percy 9 Mar 24, 2022
FaceBot is a script to automatically create a facebook account using the selenium and chromedriver modules.

FaceBot is a script to automatically create a facebook account using the selenium and chromedriver modules. That way, we don't need to input full name, email and password and date of birth. All will

Fadjrir Herlambang 2 Jun 17, 2022
bulk upload files to libgen.lc (Selenium script)

LibgenBulkUpload bulk upload files to http://libgen.lc/librarian.php (Selenium script) Usage ./upload.py to_upload uploaded rejects So title and autho

8 Jul 07, 2022
API mocking with Python.

apyr apyr (all lowercase) is a simple & easy to use mock API server. It's great for front-end development when your API is not ready, or when you are

Umut Seven 55 Nov 25, 2022
Test utility for validating OpenAPI documentation

DRF OpenAPI Tester This is a test utility to validate DRF Test Responses against OpenAPI 2 and 3 schema. It has built-in support for: OpenAPI 2/3 yaml

snok 103 Dec 21, 2022
Django-google-optimize is a Django application designed to make running server side Google Optimize A/B tests easy.

Django-google-optimize Django-google-optimize is a Django application designed to make running Google Optimize A/B tests easy. Here is a tutorial on t

Adin Hodovic 39 Oct 25, 2022