Automated network configuration backups using Github actions and git-scraping

Overview

Network Config Scraper

This repository demonstrates the use of Github Actions and git-scraping to build an automated backup solution for network configuration files. Git already provides an efficient way to track and manage changes to textual data, and Github Actions provide automation that we can use to fetch and process configuration backups without reliance on any additional infrastructure. The solution in this repository uses both to retrieve configurations from network devices on a defined schedule and commits the detected changes back to the repository.

This approach is heavily inspired by Git scraping: track changes over time by scraping to a Git repository by Simon Willison. His post provides an excellent overview of git-scraping data from various sources on the Internet.

config scraper

Could I use this?

Because of the textual nature of network configurations, git-scraping offers a simple yet effective way to backup and version configuration data. It runs entirely on GitHub Actions, so there's no complex infrastructure or orchestrations to manage.

Because the data we retrieve data from devices in a private lab, the repository uses a self-hosted runner that makes a connection to Github and has access to the lab. You'd have to decide if this is an acceptable model for your environment.

However, the solution works wonderfully and can be extended to fit into your existing automation. For example, using the webhooks in Github can extend this solution by allowing you to build or set up integrations to external automation systems which can subscribe to certain events on the repository. The configured events can trigger an HTTP POST to a system like Ansible Tower to provide additional automation.

How does it work?

The configuration scraper is configured and scheduled in the .github/workflows/scrape.yml action workflow. It's a short and simple workflow that defines all the triggers and steps to run our automation.

trigger

The workflow can be triggered in 3 different ways: on a push event to the repo, manually using the workflow_dispatch, or, most importantly, on a cron schedule. For example, in the snippet below, you can see that we have the workflow triggered every 30 minutes.

on:
  push:
    branches:
      - main
  workflow_dispatch:
  schedule:
    - cron:  '*/30 * * * *'

self-hosted runner(s)

Because our use case is slightly different from fetching data from flat files, we need to account for how GitHub would access the devices that we are fetching data from. For example, the devices in the inventory.yml are in a lab that sits behind a firewall, so a public Github action runner would not have the required access. For that reason, the repo is configured to use a self-hosted Github action runner that has access to the lab environment. It took about 15 minutes to provision and to configure a runner in this environment, so it is a relatively easy and painless process. Once configured, the self-hosted runner creates a connection to Github and listens for job requests to execute your actions.

Within the action workflow, the only things to specify are the fact that we're using a self-hosted runner and we also provide the tags to identify which runner to use for the action. We use the runs-on directive in the action to do just that, as shown below.

jobs:
  scheduled:
    runs-on: [self-hosted, atc-runners]

data

For data gathering, the retrieve_configs.py script is used to retrieve and save configuration data from each of the devices listed in the inventory.yml file. The script uses an async SSH connection transport from the scrapli library to handle parallel sessions to devices. Once the files are saved, the action workflow uses git to stage any files in which changes have been detected. A commit is created with a timestamp before the changes are pushed into the repository.

  - name: Fetch latest configs
    run: |-
      python retrieve_configs.py
    env:
      SSH_AUTH_USERNAME: ${{ secrets.SSH_AUTH_USERNAME }}
      SSH_AUTH_PASSWORD: ${{ secrets.SSH_AUTH_PASSWORD }}

  - name: Commit and push if it changed
    run: |-
      git config user.name "Automated"
      git config user.email "[email protected]"
      git add -A
      timestamp=$(date -u)
      git commit -m "Latest data: ${timestamp}" || exit 0
      git push

How do I track changes?

The neat thing about using Git to manage your configuration backups is that you get this commit log showing the history of commits that have been made to your configs.

Author(s)

Owner
WWT
World Wide Technology, Inc.
WWT
Multiple-requests-poster - A tool to send multiple requests to a particular website written in Python

Multiple-requests-poster - A tool to send multiple requests to a particular website written in Python

RLX 2 Feb 14, 2022
A Project to resolve hostname and receive IP

hostname-resolver A Project to resolve hostname and receive IP Installation git clone https://github.com/ihapiw/hostname-resolver.git Head into the ho

iHapiW 5 Sep 12, 2022
An automatic reaction network generator for reactive molecular dynamics simulation.

ReacNetGenerator An automatic reaction network generator for reactive molecular dynamics simulation. ReacNetGenerator: an automatic reaction network g

Tong Zhu Research Group 35 Dec 14, 2022
API for concurrency connections

Multi-connection-server-API API for concurrency connections difference between this server and the echo server is the call to lsock.setblocking(False)

Muziwandile Nkomo 1 Jan 04, 2022
A powerful framework for decentralized federated learning with user-defined communication topology

Scatterbrained Decentralized Federated Learning Scatterbrained makes it easy to build federated learning systems. In addition to traditional federated

Johns Hopkins Applied Physics Laboratory 7 Sep 26, 2022
Tool to get the top 100 of the fastest nodes in the Tor network. Based on Kirzahk tool.

Tor Network Top 100 IPs Tool to get the top 100 of the fastest nodes in the Tor network. Based on Kirzahk tool. Just execute top100ipstor.py to get th

Juan Manuel 0 Jan 23, 2022
snappi-trex is a snappi plugin that allows executing scripts written using snappi with Cisco's TRex Traffic Generator

snappi-trex snappi-trex is a snappi plugin that allows executing scripts written using snappi with Cisco's TRex Traffic Generator Design snappi-trex c

Open Traffic Generator 14 Sep 07, 2022
ANalyse is a vehicle network analysis and attack tool.

CANalyse is a tool built to analyze the log files to find out unique datasets automatically and able to connect to simple user interfaces suc

0xh3nry 87 Dec 18, 2022
OpenNeoMC:an Open-source Tool for Particle Transport Optimization that Combining OpenMC with NEORL

OpenNeoMC:an Open-source Tool for Particle Transport Optimization that Combining OpenMC with NEORL OpenMC is a community-developed Monte Carlo neutron

7 Aug 17, 2022
This is the code repository for Mastering Python for Networking and Security – Second Edition

Mastering Python for Networking and Security – Second Edition This is the code repository for Mastering Python for Networking and Security – Second Ed

Frank Gottinger 1 Feb 09, 2022
Blockchain-Enabled IoT Sensor Framework that uses Augmented Reality and Artificial Intelligence.

Arduino + Raspberry Pi + Unity3D + Cloud + Hyperledger Our Mission: Keep it simple, leave no one behind. Blockchain-Enabled Smart Sensor Framework usi

DappAR 23 Dec 05, 2021
RabbitMQ asynchronous connector library for Python with built in RPC support

About RabbitMQ connector library for Python that is fully integrated with the aio-pika framework. Introduction BunnyStorm is here to simplify working

22 Sep 11, 2022
Take a list of domains and probe for working HTTP and HTTPS servers

httprobe Take a list of domains and probe for working http and https servers. Install ▶ go get -u github.com/tomnomnom/httprobe Basic Usage httprobe

Tom Hudson 2.3k Dec 28, 2022
These scripts send notifications to a Webex space when a new IP is banned by Expressway, and allow to request more info or change the ban status

Spam Call and Toll Fraud Mitigation Cisco Expressway release X14 is able to mitigate spam calls and toll fraud attempts by jailing the spam IP address

Luca Pellegrini 6 Aug 05, 2022
Medusa is a cross-platform agent compatible with both Python 3.8 and Python 2.7.

Medusa Medusa is a cross-platform agent compatible with both Python 3.8 and Python 2.7. Installation To install Medusa, you'll need Mythic installed o

Mythic Agents 123 Nov 09, 2022
NetworkX is a Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks.

NetworkX is a Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks.

NetworkX 12k Jan 02, 2023
With the use of this tool, you can change your MAC address

Akshat0404/MAC_CHANGER This tool has to be used on linux kernel. Now o

1 Jan 25, 2022
Simple Python Script to Parse Apache Log, Get all Unique IPs and Urls visited by that IP

Parse_Apache_Log Simple Python Script to Parse Apache Log, Get all Unique IPs and Urls visited by that IP. It will create 3 different files. allIP.txt

Kathan Patel 2 Mar 29, 2022
A non-custodial oracle and escrow system for the lightning network. Make LN contracts more expressive.

Hodl contracts A non-custodial oracle and escrow system for the lightning network. Make LN contracts more expressive. If you fire it up, be aware: (1)

31 Nov 30, 2022
AdaFruit Funhouse publishing Temperature, Humidity and Pressure to MQTT / Apache Pulsar

pulsar-adafruit-funhouse AdaFruit Funhouse publishing Temperature, Humidity and Pressure to MQTT / Apache Pulsar Device Get your own from adafruit Ada

Timothy Spann 1 Dec 30, 2021