Web Inventory tool, takes screenshots of webpages using Pyppeteer (headless Chrome/Chromium) and provides some extra bells & whistles to make life easier.

Overview

WitnessMe

WitnessMe

WitnessMe is primarily a Web Inventory tool inspired by Eyewitness, its also written to be extensible allowing you to create custom functionality that can take advantage of the headless browser it drives in the back-end.

WitnessMe uses the Pyppeteer library to drive Headless Chromium.

Sponsors

Table of Contents

Motivation

Are there are a bunch of other tools that do this? Absolutely. See the following projects for alternatives (I'm sure there are more, these are the ones I've personally tried):

The reason why I wrote WitnessMe was that none of these projects had all of the features I wanted/needed in order for them to work well within my workflow. Additionally, some of them are prone to a decent amount of installation/dependency hell.

Here are some of the main features that make WitnessMe "stand out":

  • Written in Python 3.7+
  • Ability to parse extremely large Nessus and NMap XML files
  • Docker compatible
  • No installation/dependency hell
  • Full test suite! Everything is less prone to bugs
  • CSV & HTML reporting
  • HTTP Proxy Support
  • Provides a RESTful API! Scan stuff remotely!
  • CLI interface to view and search scan results without having to view the reports.
  • Signature scanning (Signatures use YAML files)
  • Preview screenshots directly in the terminal (On MacOSX/ITerm2 and some Nix terminals)
  • Extensibly written, allowing you to add functionality that can take advantage of headless chromium.
  • Built to be deployed to the Clouds (e.g. GCP Cloud Run , AWS ElasticBeanstalk etc...)

Official Discord Channel

Come hang out on Discord!

Porchetta Industries

Installation

Docker

Running WitnessMe from a Docker container is fully supported and is the easiest/recommended way of using the tool.

Note: it is highly recommended to give the Docker container at least 4GB of RAM during large scans as Chromium can be a resource hog. If you keep running into "Page Crash" errors, it's because your container does not have enough memory. On Mac/Windows you can change this by clicking the Docker Task Bar Icon -> Preferences -> Resources. For Linux, refer to Docker's documentation

Pull the image from Docker Hub:

docker pull byt3bl33d3r/witnessme

You can then spin up a docker container, run it like the main witnessme script and pass it the same arguments:

docker run --rm -ti $IMAGE_ID screenshot https://google.com 192.168.0.1/24

Alternatively, you can drop into a shell within the container and run the tools that way. This also allows you to execute the wmdb and wmapi scripts.

docker run --rm -ti --entrypoint=/bin/sh $IMAGE_ID

Python Package

WitnessMe is also available as a Python package (Python 3.7 or above is required). If you do install it this way it is extremely recommended to use pipx as it takes care of installing everything in isolated environments for you in a seamless manner.

Run the following commands:

python3 -m pip install --user pipx
pipx install witnessme

All of the WitnessMe scripts should now be in your PATH and ready to go.

Development Install

You really should only install WitnessMe this way if you intend to hack on the source code. You're going to Python 3.7+ and Poetry: please refer to the Poetry installation documentation in order to install it.

git clone https://github.com/byt3bl33d3r/WitnessMe && cd WitnessMe
poetry install

Quick Starts

Finding F5 Load Balancers Vulnerable to CVE-2020-5902

Install WitnessMe using Docker:

docker pull byt3bl33d3r/witnessme

Get the $IMAGE_ID from the docker images command output, then run the following command to drop into a shell inside the container. Additionally, specify the -v flag to mount the current directory inside the container at the path /transfer in order to copy the scan results back to your host machine (if so desired):

docker run -it --entrypoint=/bin/sh -v $(pwd):/transfer $IMAGE_ID

Scan your network using WitnessMe, it can accept multiple .Nessus files, Nmap XMLs, IP ranges/CIDRs. Example:

witnessme screenshot 10.0.1.0/24 192.168.0.1-20 ~/my_nessus_scan.nessus ~/my_nmap_scan.xml

After the scan is finished, a folder will have been created in the current directory with the results. Access the results using the wmdb command line utility:

wmdb scan_2020_$TIME/

To quickly identify F5 load balancers, first perform a signature scan using the scan command. Then search for "BIG-IP" or "F5" using the servers command (this will search for the "BIG-IP" and "F5" string in the signature name, page title and server header):

image

Additionally, you can generate an HTML or CSV report using the following commands:

WMDB ≫ generate_report html
WMDB ≫ generate_report csv

You can then copy the entire scan folder which will contain all of the reports and results to your host machine by copying it to the /transfer folder.

Scraping Javascript Heavy Webpages

As of v1.5.0, WitnessMe has a grab command which allows you to quickly scrape Javascript heavy webpages by rendering the page first with Headless Chromium and then parsing the resulting HTML using the specified XPath (see here for an XPath cheatsheet).

Below are a few examples to get your started.

This grabs a list of all advertised domains on the 144.161.160.0/23 subnet from Hurricane Electric's BGP Toolkit:

witnessme -d grab -x '//div[@id="dns"]/table//tr/td[2]/a/text()' https://bgp.he.net/net/144.161.160.0/23#_dns

RESTful API

As of version 1.0, WitnessMe has a RESTful API which allows you to interact with the tool remotely.

Note: Currently, the API does not implement any authentication mechanisms. Make sure to allow/deny access at the transport level

To start the RESTful API for testing/development purposes run :

wmapi

The API documentation will then be available at http://127.0.0.1:8000/docs

Uvicorn should be used to enable SSL and run the API in production. See this dockerfile for an example.

Deploying to the Cloud (™)

Since WitnessMe has a RESTful API now, you can deploy it to the magical cloud and perform scanning from there. This would have a number of benefits, including giving you a fresh external IP on every scan (More OPSEC safe when assessing attack surface on Red Teams).

There are a number of ways of doing this, you can obviously do it the traditional way (e.g. spin up a machine, install docker etc..).

Recently cloud service providers started offering ways of running Docker containers directly in a fully managed environment. Think of it as serverless functions (e.g. AWS Lambdas) only with Docker containers.

This would technically allow you to really quickly deploy and run WitnessMe (or really anything in a Docker container) without having to worry about underlying infrastructure and removes a lot of the security concerns that come with that.

Below are some of the ones I've tried along with the steps necessary to get it going and any issues I encountered.

GCP Cloud Run

Unfortunately, it seems like Cloud Run doesn't allow outbound internet access to containers, if anybody knows of a way to get around this please get in touch

Cloud Run is by far the easiest of these services to work with.

This repository includes the cloudbuild.yaml file necessary to get this setup and running.

From the repositories root folder (after you authenticated and setup a project), these two commands will automatically build the Docker image, publish it to the Gcloud Container Registry and deploy a working container to Cloud Run:

gcloud builds submit --config cloudbuild.yaml
gcloud run deploy --image gcr.io/$PROJECT_ID/witnessme --platform managed

The output will give you a HTTPS url to invoke the WitnessMe RESTful API from :)

When you're done:

gcloud run services delete witnessme
gcloud container images delete gcr.io/$PROJECT_ID/witnessme

AWS ElasticBeanstalk

TO DO

Usage

There are 3 main utilities:

  • witnessme: is the main CLI interface.
  • wmdb: allows you to browse the database (created on each scan) to view results and generate reports.
  • wmapi: provides a RESTful API to schedule, start, stop and monitor scans.

Modes of Operations

As of v1.5.0 there are two main modes (commands) that the witnessme utility Supports:

  • The screenshot command, you guessed it, screenshots webpages. This is the main functionality.
  • The grab command allows you to scrape pages and quickly grab server headers.
usage: witnessme [-h] [--threads THREADS] [--timeout TIMEOUT] [-d] [-v] {screenshot,grab} ...

WitnessMe!

positional arguments:
  {screenshot,grab}

optional arguments:
  -h, --help         show this help message and exit
  --threads THREADS  Number of concurrent browser tab(s) to open
                     [WARNING: This can cause huge RAM consumption if set to high values] (default: 15)
  --timeout TIMEOUT  Timeout for each connection attempt in seconds (default: 15)
  -d, --debug        Enable debug output (default: False)
  -v, --version      show program's version number and exit

Screenshot Mode

$ witnessme screenshot --help
usage: witnessme screenshot [-h] [-p PORTS [PORTS ...]] target [target ...]

positional arguments:
  target                The target IP(s), range(s), CIDR(s) or hostname(s), NMap XML file(s), .Nessus file(s)

optional arguments:
  -h, --help            show this help message and exit
  -p PORTS [PORTS ...], --ports PORTS [PORTS ...]
                        Ports to scan if IP Range/CIDR is provided

Can accept a mix of .Nessus file(s), Nmap XML file(s), files containing URLs and/or IPs, IP addresses/ranges/CIDRs and URLs or alternatively read from stdin.

Note: WitnessMe detects .Nessus and NMap files by their extension so make sure Nessus files have a .nessus extension and NMap scans have a .xml extension

Long story short, should be able to handle anything you throw at it:

witnessme screenshot 192.168.1.0/24 192.168.1.10-20 https://bing.com ~/my_nessus_scan.nessus ~/my_nmap_scan.xml ~/myfilewithURLSandIPs
$ cat my_domain_list.txt | witnessme screenshot -

If an IP address/range/CIDR is specified as a target, WitnessMe will attempt to screenshot HTTP & HTTPS pages on ports 80, 8080, 443, 8443 by default. This is customizable with the --port argument.

Once a scan is completed, a folder with all the screenshots and a database will be in the current directory, point wmdb to the folder in order to see the results.

wmdb scan_2019_11_05_021237/

Grab Mode

$ witnessme grab --help
usage: witnessme grab [-h] [-x XPATH | -l] target [target ...]

positional arguments:
  target                The target IP(s), range(s), CIDR(s) or hostname(s), NMap XML file(s), .Nessus file(s)

optional arguments:
  -h, --help            show this help message and exit
  -x XPATH, --xpath XPATH
                        XPath to use
  -l, --links           Get all links

The grab subcommand allows you to render Javascript heavy webpages and scrape their content using XPaths. See this section for some examples.

Interacting with the Scan Database

Once a scan is completed (using the screenshot mode), a folder with all the screenshots and a database will be in the current directory, point wmdb to the folder in order to see the results.

wmdb scan_2019_11_05_021237/

This will drop you into the WMDB CLI menu.

Pressing tab will show you the available commands and a help menu:

Tab

The servers and hosts commands in the wmdb CLI accept 1 argument. WMCLI is smart enough to know what you're trying to do with that argument

Server Command

No arguments will show all discovered servers. Passing it an argument will search the title and server columns for that pattern (it's case insensitive).

For example if you wanted to search for all discovered Apache Tomcat servers:

  • servers tomcat or servers 'apache tomcat'

Similarly if you wanted to find servers with a 'login' in the title:

  • servers login

Hosts Command

No arguments will show all discovered hosts. Passing it an argument will search the IP and Hostname columns for that pattern (it's case insensitive). If the value corresponds to a Host ID it will show you the host information and all of the servers discovered on that host which is extremely useful for reporting purposes and/or when targeting specific hosts.

Signature Scan

You can perform a signature scan on all discovered services using the scan command.

Generating Reports

You can use the generate_report command in the wmdb cli to generate reports in HTML or CSV format. To generate a HTML report simply run generate_report without any arguments. Here's an example of what it'll look like:

image

To generate a CSV report:

WMDB ≫ generate_report csv

The reports will then be available in the scan folder.

Proxying

As of v1.5 WitnessMe supports proxying all of its traffic through an HTTP proxy. Specify a HTTP_PROXY environment variable to force the underlying headless browser to proxy its traffic through the desired host:

HTTP_PROXY=http://127.0.0.1:8080 witnessme screenshot ~/my_targets.txt
HTTP_PROXY=http://127.0.0.1:8080 witnessme grab https://www.google.com

Preview Screenshots Directly in the Terminal

Note: this feature will only work if you're on MacOSX and using ITerm2

You can preview screenshots directly in the terminal using the show command:

ScreenPreview

Writing Signatures

If you run into a new webapp write a signature for it! It's beyond simple and they're all in YAML!

Don't believe me? Here's the AirOS signature (you can find them all in the signatures directory):

credentials:
- password: ubnt
  username: ubnt
name: AirOS
signatures:
- airos_logo.png
- form enctype="multipart/form-data" id="loginform" method="post"
- align="center" class="loginsubtable"
- function onLangChange()
# AirOS ubnt/ubnt

Yup that's it. Just plop it in the signatures folder and POW! Done.

Owner
byt3bl33d3r
T H I C C M A L W A R E
byt3bl33d3r
REST API with FastAPI and SQLite3.

REST API with FastAPI and SQLite3

Luis Quiñones Requelme 2 Mar 14, 2022
API & Webapp to answer questions about COVID-19. Using NLP (Question Answering) and trusted data sources.

This open source project serves two purposes. Collection and evaluation of a Question Answering dataset to improve existing QA/search methods - COVID-

deepset 329 Nov 10, 2022
Auth for use with FastAPI

FastAPI Auth Pluggable auth for use with FastAPI Supports OAuth2 Password Flow Uses JWT access and refresh tokens 100% mypy and test coverage Supports

David Montague 95 Jan 02, 2023
A rate limiter for Starlette and FastAPI

SlowApi A rate limiting library for Starlette and FastAPI adapted from flask-limiter. Note: this is alpha quality code still, the API may change, and

Laurent Savaete 565 Jan 02, 2023
基于Pytorch的脚手架项目,Celery+FastAPI+Gunicorn+Nginx+Supervisor实现服务部署,支持Docker发布

cookiecutter-pytorch-fastapi 基于Pytorch的 脚手架项目 按规范添加推理函数即可实现Celery+FastAPI+Gunicorn+Nginx+Supervisor+Docker的快速部署 Requirements Python = 3.6 with pip in

17 Dec 23, 2022
The template for building scalable web APIs based on FastAPI, Tortoise ORM and other.

FastAPI and Tortoise ORM. Powerful but simple template for web APIs w/ FastAPI (as web framework) and Tortoise-ORM (for working via database without h

prostomarkeloff 95 Jan 08, 2023
row level security for FastAPI framework

Row Level Permissions for FastAPI While trying out the excellent FastApi framework there was one peace missing for me: an easy, declarative way to def

Holger Frey 315 Dec 25, 2022
Adds integration of the Jinja template language to FastAPI.

fastapi-jinja Adds integration of the Jinja template language to FastAPI. This is inspired and based off fastapi-chamelon by Mike Kennedy. Check that

Marc Brooks 58 Nov 29, 2022
Social Distancing Detector using deep learning and capable to run on edge AI devices such as NVIDIA Jetson, Google Coral, and more.

Smart Social Distancing Smart Social Distancing Introduction Getting Started Prerequisites Usage Processor Optional Parameters Configuring AWS credent

Neuralet 129 Dec 12, 2022
asgi-server-timing-middleware

ASGI Server-Timing middleware An ASGI middleware that wraps the excellent yappi profiler to let you measure the execution time of any function or coro

33 Dec 15, 2022
Basic FastAPI starter with GraphQL, Docker, and MongoDB configurations.

FastAPI + GraphQL Starter A python starter project using FastAPI and GraphQL. This project leverages docker for containerization and provides the scri

Cloud Bytes Collection 1 Nov 24, 2022
Code for my JWT auth for FastAPI tutorial

FastAPI tutorial Code for my video tutorial FastAPI tutorial What is FastAPI? FastAPI is a high-performant REST API framework for Python. It's built o

José Haro Peralta 8 Dec 16, 2022
Admin Panel for GinoORM - ready to up & run (just add your models)

Gino-Admin Docs (state: in process): Gino-Admin docs Play with Demo (current master 0.2.3) Gino-Admin demo (login: admin, pass: 1234) Admin

Iuliia Volkova 46 Nov 02, 2022
FastAPI Learning Example,对应中文视频学习教程:https://space.bilibili.com/396891097

视频教学地址 中文学习教程 1、本教程每一个案例都可以独立跑,前提是安装好依赖包。 2、本教程并未按照官方教程顺序,而是按照实际使用顺序编排。 Video Teaching Address FastAPI Learning Example 1.Each case in this tutorial c

381 Dec 11, 2022
fastapi-crud-sync

Developing and Testing an API with FastAPI and Pytest Syncronous Example Want to use this project? Build the images and run the containers: $ docker-c

59 Dec 11, 2022
Fastapi practice project

todo-list-fastapi practice project How to run Install dependencies npm, yarn: standard-version, husky make: script for lint, test pipenv: virtualenv +

Deo Kim 10 Nov 30, 2022
FastAPI Admin Dashboard based on FastAPI and Tortoise ORM.

FastAPI ADMIN 中文文档 Introduction FastAPI-Admin is a admin dashboard based on fastapi and tortoise-orm. FastAPI-Admin provide crud feature out-of-the-bo

long2ice 1.6k Dec 31, 2022
volunteer-database

This is the official CSM (Crowd source medical) database The What Now? We created this in light of the COVID-19 pandemic to allow volunteers to work t

32 Jun 21, 2022
signal-cli-rest-api is a wrapper around signal-cli and allows you to interact with it through http requests

signal-cli-rest-api signal-cli-rest-api is a wrapper around signal-cli and allows you to interact with it through http requests. Features register/ver

Sebastian Noel Lübke 31 Dec 09, 2022
MLServer

MLServer An open source inference server to serve your machine learning models. ⚠️ This is a Work in Progress. Overview MLServer aims to provide an ea

Seldon 341 Jan 03, 2023