Automated network configuration backups using Github actions and git-scraping

Overview

Network Config Scraper

This repository demonstrates the use of Github Actions and git-scraping to build an automated backup solution for network configuration files. Git already provides an efficient way to track and manage changes to textual data, and Github Actions provide automation that we can use to fetch and process configuration backups without reliance on any additional infrastructure. The solution in this repository uses both to retrieve configurations from network devices on a defined schedule and commits the detected changes back to the repository.

This approach is heavily inspired by Git scraping: track changes over time by scraping to a Git repository by Simon Willison. His post provides an excellent overview of git-scraping data from various sources on the Internet.

config scraper

Could I use this?

Because of the textual nature of network configurations, git-scraping offers a simple yet effective way to backup and version configuration data. It runs entirely on GitHub Actions, so there's no complex infrastructure or orchestrations to manage.

Because the data we retrieve data from devices in a private lab, the repository uses a self-hosted runner that makes a connection to Github and has access to the lab. You'd have to decide if this is an acceptable model for your environment.

However, the solution works wonderfully and can be extended to fit into your existing automation. For example, using the webhooks in Github can extend this solution by allowing you to build or set up integrations to external automation systems which can subscribe to certain events on the repository. The configured events can trigger an HTTP POST to a system like Ansible Tower to provide additional automation.

How does it work?

The configuration scraper is configured and scheduled in the .github/workflows/scrape.yml action workflow. It's a short and simple workflow that defines all the triggers and steps to run our automation.

trigger

The workflow can be triggered in 3 different ways: on a push event to the repo, manually using the workflow_dispatch, or, most importantly, on a cron schedule. For example, in the snippet below, you can see that we have the workflow triggered every 30 minutes.

on:
  push:
    branches:
      - main
  workflow_dispatch:
  schedule:
    - cron:  '*/30 * * * *'

self-hosted runner(s)

Because our use case is slightly different from fetching data from flat files, we need to account for how GitHub would access the devices that we are fetching data from. For example, the devices in the inventory.yml are in a lab that sits behind a firewall, so a public Github action runner would not have the required access. For that reason, the repo is configured to use a self-hosted Github action runner that has access to the lab environment. It took about 15 minutes to provision and to configure a runner in this environment, so it is a relatively easy and painless process. Once configured, the self-hosted runner creates a connection to Github and listens for job requests to execute your actions.

Within the action workflow, the only things to specify are the fact that we're using a self-hosted runner and we also provide the tags to identify which runner to use for the action. We use the runs-on directive in the action to do just that, as shown below.

jobs:
  scheduled:
    runs-on: [self-hosted, atc-runners]

data

For data gathering, the retrieve_configs.py script is used to retrieve and save configuration data from each of the devices listed in the inventory.yml file. The script uses an async SSH connection transport from the scrapli library to handle parallel sessions to devices. Once the files are saved, the action workflow uses git to stage any files in which changes have been detected. A commit is created with a timestamp before the changes are pushed into the repository.

  - name: Fetch latest configs
    run: |-
      python retrieve_configs.py
    env:
      SSH_AUTH_USERNAME: ${{ secrets.SSH_AUTH_USERNAME }}
      SSH_AUTH_PASSWORD: ${{ secrets.SSH_AUTH_PASSWORD }}

  - name: Commit and push if it changed
    run: |-
      git config user.name "Automated"
      git config user.email "[email protected]"
      git add -A
      timestamp=$(date -u)
      git commit -m "Latest data: ${timestamp}" || exit 0
      git push

How do I track changes?

The neat thing about using Git to manage your configuration backups is that you get this commit log showing the history of commits that have been made to your configs.

Author(s)

Owner
WWT
World Wide Technology, Inc.
WWT
PoC code for stealing the WiFi password of a network with a Lovebox IOT device connected

LoveBoxer PoC code for stealing the WiFi password of a network with a Lovebox IOT device connected. This PoC was is what I used in this blogpost Usage

Graham Helton 10 May 24, 2022
Tool written on Python that locate all up host on your subnet

HOSTSCAN Easy to use command line network host scanner. From noob to noobs. Dependencies Nmap 7.92 or superior Python 3.9 or superior All requirements

NexCreep 4 Feb 27, 2022
BLE parser for passive BLE advertisements

This pypi package is parsing BLE advertisements to readable data for several sensors and can be used for device tracking, as long as the MAC address is static. The parser was originally developed as

Ernst Klamer 19 Dec 26, 2022
A simple software which can use to make a server in local network

home-nas it is simple software which can use to make a server in local network, it has a web site on it which can use by multipale system, i use nginx

R ansh joseph 1 Nov 10, 2021
Readable, simple and fast asynchronous non-blocking network apps

Fast and readable async non-blocking network apps Netius is a Python network library that can be used for the rapid creation of asynchronous non-block

Hive Solutions 120 Nov 20, 2022
Implementing Cisco Support APIs into NetBox

NetBox Cisco Support API Plugin NetBox plugin using Cisco Support APIs to gather EoX and Contract coverage information for Cisco devices. Compatibilit

Timo Reimann 23 Dec 21, 2022
Out-of-box Python RPC framework

typed-jsonrpc Out-of-box Python RPC framework. WIP. Make LSP easy for everyone. The conception of final usage: from typed_jsonrpc import * ls = Langu

Taine Zhao 4 Dec 28, 2021
Ip-Tracker: a script written in python for tracking Someone using targets ip-Tracker address

πŸ”° 𝕀𝕑-π•‹π•£π•’π•”π•œπ•–π•£ πŸ”° Ip-Tracker is a script written in python for tracking Someone using targets ip-Tracker address It was made by Spider Anongre

Spider Anongreyhat 15 Dec 02, 2022
Library containing the core modules for the kingdom-python-server.

🏰 Kingdom Core Library containing the core modules for the kingdom-python-server. Installation Use the package manager pip to install kingdom-core. p

T10 4 Dec 27, 2021
simple subdomain finder

Subdomain-finder Simple SubDomain finder using python which is easy to use just download and run it Wordlist you can use your own wordlist but here i

AsjadOwO 5 Sep 24, 2021
This repository contain sample code of gRPC Communication between Python and GoLang

This repository contain sample code of gRPC Communication between Python and GoLang, the Server is running on GoLang while Python is running the client

Abdullahi Muhammad 2 Nov 29, 2021
Multi-path load balancing is a method used by most of the real-time network to split the packets into different paths rather than transferring it through a single path

Multipath-Load-Balancing Method of managing incoming traffic by distributing and sharing load fairly among multiple routes from source to destination

Dharshan Kumar 6 Dec 10, 2022
Tool for quickly gathering information from Shodan.io about the number of IPs which satisfy large number of different queries

TriOp Tool for quickly gathering information from Shodan.io about the number of IPs which satisfy large number of different queries For furt

Jan Kopriva 27 Nov 03, 2022
Repo used to maintain all notes and scripts developed during my DevNet Expert studies

DevNet Expert Studies Exam Date: TBD (Waiting for registration to open) This repository will be used to track my progress and maintain all notes/scrip

Dan 32 Dec 11, 2022
A gRPC-Web implementation for Python

Sonora Sonora is a Python-first implementation of gRPC-Web built on top of standard Python APIs like WSGI and ASGI for easy integration. Why? Regular

Alex Stapleton 216 Dec 30, 2022
pureSxS - A tool to export Component Based Servicing packages from a full Windows installation

pureSxS A tool to export Component Based Servicing packages from a full Windows installation. Usage pureSxS.py source_mum destination pureSxS wor

Gamers Against Weed 3 Oct 03, 2022
An opensource library to use SNMP get/bulk/set/walk in Python

SNMP-UTILS An opensource library to use SNMP get/bulk/set/walk in Python Features Work with OIDS json list [Find Here](#OIDS List) GET command SET com

Alexandre Gossard 3 Aug 03, 2022
Very simple and tiny file sharing service written in python

Simple File sharing tool Sharing client usage (You will need to use python3 for linux) main.py --send -f file/path Optionnal flags : pwd : set a passw

2 Jul 26, 2022
Modern Denial-of-service ToolKit for python

πŸ’£ Impulse Modern Denial-of-service ToolKit πŸ’» Main window πŸ“‘ Methods: Method Target Description SMS PHONE Sends a massive amount of SMS messages and

1 Nov 29, 2021
A great python/java dynamic DNS service for NameSilo, with log, email reminder...

English NameSilo DDNS is a DDNS service for NameSilo domain names for home broadband , it can automatically detect IP changes in home broadband

云牧青 77 Dec 28, 2022