Import, visualize, and analyze SpiderFoot OSINT data in Neo4j, a graph database

Overview

SpiderFoot Neo4j Tools

Import, visualize, and analyze SpiderFoot OSINT data in Neo4j, a graph database

A big graph

Step 1: Installation

NOTE: This installs the sfgraph command-line utility

$ pip install spiderfoot-neo4j

Step 2: Start Neo4j

NOTE: Docker must first be installed

$ docker run --rm --name sfgraph -v "$(pwd)/neo4j_database:/data" -e 'NEO4J_AUTH=neo4j/CHANGETHISIFYOURENOTZUCK' -e 'NEO4JLABS_PLUGINS=["apoc", "graph-data-science"]' -e 'NEO4J_dbms_security_procedures_unrestricted=apoc.*,gds.*' -p "7474:7474" -p "7687:7687" neo4j

Step 3: Import Scans

Spiderfoot scan ID in web browser

$ sfgraph path_to/spiderfoot.db -s   ...

Step 4: Browse Spiderfoot Data in Neo4j

Visit http://127.0.0.1:7474 and log in with neo4j/CHANGETHISIFYOURENOTZUCK Spiderfoot data in Neo4j

Step 5 (Optional): Use cool algorithms to find new targets

The --suggest option will rank nodes based on their connectedness in the graph. This is perfect for finding closely-related affiliates (child companies, etc.) to scan and add to the graph. By default, Harmonic Centrality is used, but others such as PageRank can be specified with --closeness-algorithm

$ sfgraph --suggest DOMAIN_NAME

Closeness scores

Example CYPHER Queries

() RETURN p # shortest path to all INTERNET_NAMEs from seed domain MATCH p=shortestPath((d:DOMAIN_NAME {data:"evilcorp.com"})-[*]-(n:INTERNET_NAME)) RETURN p # match only primary targets (non-affiliates) MATCH (n {scanned: true}) return n # match only affiliates MATCH (n {affiliate: true}) return n ">
# match all INTERNET_NAMEs
MATCH (n:INTERNET_NAME) RETURN n

# match multiple event types
MATCH (n) WHERE n:INTERNET_NAME OR n:DOMAIN_NAME OR n:EMAILADDR RETURN n

# match by attribute
MATCH (n {data: "evilcorp.com"}) RETURN n

# match by spiderfoot module (relationship)
MATCH p=()-[r:WHOIS]->() RETURN p

# shortest path to all INTERNET_NAMEs from seed domain
MATCH p=shortestPath((d:DOMAIN_NAME {data:"evilcorp.com"})-[*]-(n:INTERNET_NAME)) RETURN p

# match only primary targets (non-affiliates)
MATCH (n {scanned: true}) return n

# match only affiliates
MATCH (n {affiliate: true}) return n

CLI Help

sfgraph [-h] [-db SQLITEDB] [-s SCANS [SCANS ...]] [--uri URI] [-u USERNAME] [-p PASSWORD] [--clear] [--suggest SUGGEST]
               [--closeness-algorithm {pageRank,articleRank,closenessCentrality,harmonicCentrality,betweennessCentrality,eigenvectorCentrality}] [-v]

optional arguments:
  -h, --help            show this help message and exit
  -db SQLITEDB, --sqlitedb SQLITEDB
                        Spiderfoot sqlite database
  -s SCANS [SCANS ...], --scans SCANS [SCANS ...]
                        scan IDs to import
  --uri URI             Neo4j database URI (default: bolt://127.0.0.1:7687)
  -u USERNAME, --username USERNAME
                        Neo4j username (default: neo4j)
  -p PASSWORD, --password PASSWORD
                        Neo4j password
  --clear               Wipe the Neo4j database
  --suggest SUGGEST     Suggest targets of this type (e.g. DOMAIN_NAME) based on their connectedness in the graph
  --closeness-algorithm {pageRank,articleRank,closenessCentrality,harmonicCentrality,betweennessCentrality,eigenvectorCentrality}
                        Algorithm to use when suggesting targets
  -v, -d, --debug       Verbose / debug
Owner
Black Lantern Security
Security Organization
Black Lantern Security
Learning Convolutional Neural Networks with Interactive Visualization.

CNN Explainer An interactive visualization system designed to help non-experts learn about Convolutional Neural Networks (CNNs) For more information,

Polo Club of Data Science 6.3k Jan 01, 2023
Main repository for Vispy

VisPy: interactive scientific visualization in Python Main website: http://vispy.org VisPy is a high-performance interactive 2D/3D data visualization

vispy 3k Jan 03, 2023
paintable GitHub contribute table

githeart paintable github contribute table how to use: Functions key color select 1,2,3,4,5 clear c drawing mode mode on turn off e print paint matrix

Bahadır Araz 27 Nov 24, 2022
A gui application to visualize various sorting algorithms using pure python.

Sorting Algorithm Visualizer A gui application to visualize various sorting algorithms using pure python. Language : Python 3 Libraries required Tkint

Rajarshi Banerjee 19 Nov 30, 2022
Custom Plotly Dash components based on Mantine React Components library

Dash Mantine Components Dash Mantine Components is a Dash component library based on Mantine React Components Library. It makes it easier to create go

Snehil Vijay 239 Jan 08, 2023
📊 Extensions for Matplotlib

📊 Extensions for Matplotlib

Nico Schlömer 519 Dec 30, 2022
Gaphas is the diagramming widget library for Python.

Gaphas Gaphas is the diagramming widget library for Python. Gaphas is a library that provides the user interface component (widget) for drawing diagra

Gaphor 144 Dec 14, 2022
Simulation du problème de Monty Hall avec Python et matplotlib

Le problème de Monty Hall C'est un jeu télévisé où il y a trois portes sur le plateau de jeu. Seule une de ces portes cache un trésor. Il n'y a rien d

ETCHART YANG 1 Jan 06, 2022
By default, networkx has problems with drawing self-loops in graphs.

By default, networkx has problems with drawing self-loops in graphs. It makes it hard to draw a graph with self-loops or to make a nicely looking chord diagram. This repository provides some code to

Vladimir Shitov 5 Jan 06, 2022
A python package for animating plots build on matplotlib.

animatplot A python package for making interactive as well as animated plots with matplotlib. Requires Python = 3.5 Matplotlib = 2.2 (because slider

Tyler Makaro 394 Dec 18, 2022
This is Pygrr PolyArt, a program used for drawing custom Polygon models for your Pygrr project!

This is Pygrr PolyArt, a program used for drawing custom Polygon models for your Pygrr project!

Isaac 4 Dec 14, 2021
An interactive GUI for WhiteboxTools in a Jupyter-based environment

whiteboxgui An interactive GUI for WhiteboxTools in a Jupyter-based environment GitHub repo: https://github.com/giswqs/whiteboxgui Documentation: http

Qiusheng Wu 105 Dec 15, 2022
A way of looking at COVID-19 data that I haven't seen before.

Visualizing Omicron: COVID-19 Deaths vs. Cases Click here for other countries. Data is from Our World in Data/Johns Hopkins University. About this pro

1 Jan 10, 2022
Automatically generate GitHub activity!

Commit Bot Automatically generate GitHub activity! We've all wanted to be the developer that commits every day, but that requires a lot of work. Let's

Ricky 4 Jun 07, 2022
termplotlib is a Python library for all your terminal plotting needs.

termplotlib termplotlib is a Python library for all your terminal plotting needs. It aims to work like matplotlib. Line plots For line plots, termplot

Nico Schlömer 553 Dec 30, 2022
An interactive dashboard built with python that enables you to visualise how rent prices differ across Sweden.

sweden-rent-dashboard An interactive dashboard built with python that enables you to visualise how rent prices differ across Sweden. The dashboard/web

Rory Crean 5 Dec 19, 2021
Fast data visualization and GUI tools for scientific / engineering applications

PyQtGraph A pure-Python graphics library for PyQt5/PyQt6/PySide2/PySide6 Copyright 2020 Luke Campagnola, University of North Carolina at Chapel Hill h

pyqtgraph 3.1k Jan 08, 2023
Statistics and Visualization of acceptance rate, main keyword of CVPR 2021 accepted papers for the main Computer Vision conference (CVPR)

Statistics and Visualization of acceptance rate, main keyword of CVPR 2021 accepted papers for the main Computer Vision conference (CVPR)

Hoseong Lee 78 Aug 23, 2022
matplotlib: plotting with Python

Matplotlib is a comprehensive library for creating static, animated, and interactive visualizations in Python. Check out our home page for more inform

Matplotlib Developers 16.7k Jan 08, 2023
IPython/Jupyter notebook module for Vega and Vega-Lite

IPython Vega IPython/Jupyter notebook module for Vega 5, and Vega-Lite 4. Notebooks with embedded visualizations can be viewed on GitHub and nbviewer.

Vega 335 Nov 29, 2022