This repository has automation content to test Arista devices.

Overview

GitHub test workflow

Network tests automation

About this repository

This repository has automation content to test Arista devices.

To run tests programatically on your EOS devices, once you are done with the requirements described below, you simply need to indicate your devices name in a text file and to indicate the tests you would like to run in a YAML file.

This repository has also content to collect commands output on devices, to clear counters on devices, and to test the devices reachability.

This repository uses Python scripts and eAPI (EOS API). You can find examples of EOS automation with eAPI in this repository.

Requirements

Requirements on your laptop

Python 3 (at least 3.3) and some packages that are not part of the standard Python library.

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.4 LTS
Release:        18.04
Codename:       bionic
python -V
Python 3.9.6

The required packages are in the requirements.txt file. Clone the repository and install the requirements:

pip install -r requirements.txt

Verify

pip list | grep 'jsonrpclib-pelix\|PyYAML\|colorama\|prettytable'
colorama                   0.3.7
jsonrpclib-pelix           0.4.2
prettytable                2.5.0
PyYAML                     5.4.1

Requirements on the switches

  • Enable eAPI:
switch1#configure
switch1(config)#management api http-commands
switch1(config-mgmt-api-http-cmds)#protocol https port 443
switch1(config-mgmt-api-http-cmds)#no shutdown
switch1(config-mgmt-api-http-cmds)#vrf MGMT
switch1(config-mgmt-api-http-cmds-vrf-MGMT)#no shutdown
switch1(config-mgmt-api-http-cmds-vrf-MGMT)#exit
switch1(config-mgmt-api-http-cmds)#exit
switch1(config)#
switch1#sh run int ma1
interface Management1
   description oob_management
   vrf MGMT
   ip address 10.73.1.105/24
switch1#

Now the swicth accepts HTTPS requests on port 443 in the MGMT VRF containing a list of CLI commands.

  • Verify:
switch1(config)#sh management http-server
SSL Profile:        none
FIPS Mode:          No
QoS DSCP:           0
Log Level:          none
CSP Frame Ancestor: None
TLS Protocols:      1.0 1.1 1.2
   VRF        Server Status      Enabled Services
---------- --------------------- ----------------
   MGMT       HTTPS: port 443    http-commands

switch1>show management api http-commands
Enabled:            Yes
HTTPS server:       running, set to use port 443
HTTP server:        shutdown, set to use port 80
Local HTTP server:  shutdown, no authentication, set to use port 8080
Unix Socket server: shutdown, no authentication
VRFs:               MGMT
Hits:               83
Last hit:           2631 seconds ago
Bytes in:           11348
Bytes out:          335951
Requests:           53
Commands:           64
Duration:           9.242 seconds
SSL Profile:        none
FIPS Mode:          No
QoS DSCP:           0
Log Level:          none
CSP Frame Ancestor: None
TLS Protocols:      1.0 1.1 1.2
   User          Requests       Bytes in       Bytes out    Last hit
------------- -------------- -------------- --------------- ------------------
   arista        2              305            1235         639908 seconds ago
   ansible       51             11043          334716       2631 seconds ago

URLs
-------------------------------------
Management1 : https://10.73.1.105:443

switch1>

Quick test

Run this python script to validate the requirements and the device reachability. Use your device credentials and IP address.

from jsonrpclib import Server
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
username = "arista"
password = "arista"
ip = "10.100.164.145"
url = "https://" + username + ":" + password + "@" + ip + "/command-api"
switch = Server(url)
result=switch.runCmds(1,['show version'], 'text')
print(result[0]['output'])

List of available tests

The tests are defined in the python module functions.py in the python package tests_eos. Each function returns True or False (or None when it can not run properly).

The documentation directory has the tests documentation:

We indicate the tests we would like to run in a YAML file. Some tests require an input. In that case we provide it using the same YAML file. Here's an example tests.yaml.

Repository usage

  • Clone this repository.
  • Install the requirements (see above)

To test devices reachability

vi devices.txt
./check-devices-reachability.py --help
./check-devices-reachability.py -i devices.txt -u username

To run tests on devices

  • Update the devices inventory devices.txt with the devices IP address or hostnames.
  • Update the file tests.yaml to indicate the tests you would like to run. Some tests require an input. In that case, provide it using the same YAML file.
  • Run the python script check-devices.py.
  • Check the result in the output file.
vi devices.txt
vi tests.yaml
./check-devices.py --help
./check-devices.py -i devices.txt -t tests.yaml -o output.txt -u username
cat output.txt

To collect commands output from EOS devices

  • Update the devices inventory devices.txt with your devices IP address or hostnames.
  • Update the EOS commands list eos-commands.yaml you would like to collect from the devices in text or JSON format.
  • Run the python script collect-eos-commands.py.
  • Check the output in the output directory.
vi devices-list.text
vi eos-commands.yaml
./collect-eos-commands.py --help
./collect-eos-commands.py -i devices.txt -c eos-commands.yaml -o outdir -u username
ls outdir

To clear counters on EOS devices

vi devices-list.text
./clear_counters.py --help
./clear_counters.py -i devices.txt -u username

Demo

$ ./check-devices.py -u ansible -i devices.txt -o output.txt -t tests.yaml
Device password:
Enable password (if any):
Can not connect to device 2.2.2.2
Running tests on device 10.73.1.101 ...
Running tests on device 10.73.1.102 ...
Running tests on device 10.73.1.106 ...
Test results are saved on output.txt
$ cat output.txt
Fri Jan 28 09:48:54 2022
devices inventory file was devices.txt
devices username was ansible
list of unreachable devices is
2.2.2.2
tests file was tests.yaml

***** Results *****

+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|    devices    |  01.01  |  01.02  |  01.03  |  01.04  |  02.01  |  02.02  |  02.03  |  02.04  |  02.05  |  02.06  |  02.07  |  02.08  |  03.01  |
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|  10.73.1.101  |   Fail  |   Pass  |   Pass  |   Pass  |   Pass  |   Skip  |   Pass  |   Fail  |   Pass  |   Pass  |   Pass  |   Pass  |   Skip  |
|  10.73.1.102  |   Fail  |   Pass  |   Pass  |   Pass  |   Pass  |   Skip  |   Pass  |   Pass  |   Pass  |   Pass  |   Pass  |   Pass  |   Skip  |
|  10.73.1.106  |   Fail  |   Pass  |   Pass  |   Pass  |   Pass  |   Skip  |   Pass  |   Pass  |   Fail  |   Pass  |   Pass  |   Pass  |   Skip  |
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|    devices    |  03.02  |  03.03  |  03.04  |  03.05  |  04.01  |  04.02  |  05.01  |  05.02  |  06.01  |  07.01  |  08.01  |  08.02  |  08.03  |
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|  10.73.1.101  |   Fail  |   Pass  |   Fail  |   Skip  |   Pass  |   Fail  |   Skip  |   Skip  |   Skip  |   Pass  |   Pass  |   Pass  |   Pass  |
|  10.73.1.102  |   Fail  |   Pass  |   Fail  |   Skip  |   Pass  |   Fail  |   Skip  |   Skip  |   Skip  |   Pass  |   Pass  |   Pass  |   Pass  |
|  10.73.1.106  |   Fail  |   Pass  |   Fail  |   Skip  |   Pass  |   Fail  |   Skip  |   Skip  |   Skip  |   Pass  |   Pass  |   Pass  |   Pass  |
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|    devices    |  08.04  |  08.05  |  09.01  |  09.02  |  09.03  |  09.04  |  10.01  |  11.01  |  11.02  |  12.01  |  13.01  |  14.01  |  14.02  |
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|  10.73.1.101  |   Pass  |   Pass  |   Skip  |   Skip  |   Skip  |   Skip  |   Fail  |   Fail  |   Skip  |   Pass  |   Pass  |   Pass  |   Pass  |
|  10.73.1.102  |   Pass  |   Pass  |   Skip  |   Skip  |   Skip  |   Skip  |   Fail  |   Fail  |   Skip  |   Pass  |   Pass  |   Pass  |   Pass  |
|  10.73.1.106  |   Pass  |   Pass  |   Fail  |   Pass  |   Fail  |   Fail  |   Fail  |   Pass  |   Fail  |   Pass  |   Pass  |   Pass  |   Pass  |
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|    devices    |  15.01  |  16.01  |  16.02  |  16.03  |  16.04  |  16.05  |  16.06  |  16.07  |  17.01  |  18.01  |  18.02  |
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|  10.73.1.101  |   Pass  |   Fail  |   Pass  |   Fail  |   Skip  |   Fail  |   Skip  |   Fail  |   Fail  |   Pass  |   Fail  |
|  10.73.1.102  |   Pass  |   Fail  |   Pass  |   Fail  |   Skip  |   Fail  |   Skip  |   Fail  |   Fail  |   Skip  |   Pass  |
|  10.73.1.106  |   Pass  |   Fail  |   Pass  |   Pass  |   Skip  |   Pass  |   Skip  |   Fail  |   Fail  |   Skip  |   Fail  |
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+

***** Tests *****
01.01    {"name": "verify_eos_version", "versions": ["4.25.4M", "4.26.1F"]}
01.02    {"name": "verify_terminattr_version", "versions": ["v1.13.6", "v1.8.0"]}
01.03    "verify_eos_extensions"
01.04    "verify_field_notice_44_resolution"
02.01    {"name": "verify_uptime", "min": 86400}
02.02    "verify_reload_cause"
02.03    "verify_coredump"
02.04    "verify_agent_logs"
02.05    "verify_syslog"
02.06    "verify_cpu_utilization"
02.07    "verify_memory_utilization"
02.08    "verify_filesystem_utilization"
03.01    {"name": "verify_transceivers_manufacturers", "manufacturers": ["Not Present", "Arista Networks", "Arastra, Inc."]}
03.02    "verify_system_temperature"
03.03    "verify_transceiver_temperature"
03.04    "verify_environment_cooling"
03.05    "verify_environment_power"
04.01    "verify_zerotouch"
04.02    "verify_running_config_diffs"
05.01    {"name": "verify_unified_forwarding_table_mode", "mode": 3}
05.02    {"name": "verify_tcam_profile", "profile": "vxlan-routing"}
06.01    "verify_adverse_drops"
07.01    "verify_ntp"
08.01    "verify_interface_utilization"
08.02    "verify_interface_errors"
08.03    "verify_interface_discards"
08.04    "verify_interface_errdisabled"
08.05    {"name": "verify_interfaces_status", "minimum": 4}
09.01    "verify_portchannels"
09.02    "verify_illegal_lacp"
09.03    "verify_mlag_status"
09.04    "verify_mlag_interfaces"
10.01    {"name": "verify_loopback_count", "number": 3}
11.01    "verify_vxlan"
11.02    "verify_vxlan_config_sanity"
12.01    "verify_svi"
13.01    "verify_spanning_tree_blocked_ports"
14.01    {"name": "verify_routing_protocol_model", "model": "multi-agent"}
14.02    {"name": "verify_routing_table_size", "min": 2, "max": 20}
15.01    "verify_bfd"
16.01    "verify_bgp_ipv4_unicast_state"
16.02    "verify_bgp_ipv6_unicast_state"
16.03    "verify_bgp_evpn_state"
16.04    "verify_bgp_rtc_state"
16.05    {"name": "verify_bgp_evpn_count", "number": 2}
16.06    {"name": "verify_bgp_rtc_count", "number": 2}
16.07    {"name": "verify_bgp_ipv4_unicast_count", "number": 2}
17.01    {"name": "verify_ospf", "number": 3}
18.01    {"name": "verify_igmp_snooping_vlans", "configuration": "disabled", "vlans": [10, 12]}
18.02    {"name": "verify_igmp_snooping_global", "configuration": "enabled"}

Repository structure

devices.txt file

The file devices.txt is the devices inventory.

The devices inventory is a text file with the devices IP address or hostnames (if resolvable with DNS or your hosts file). This file has one device per ligne.

eos-commands.yaml file

The file eos-commands.yaml is a YAML file used to indicated the list of commands output we would like to collect from EOS devices in text or json format.

tests.yaml file

The file tests.yaml is a YAML file used to indicated the tests we would like to run. It is also used to indicated the parameters used by the tests. Each test has an identifier which is then used in the tests report. The tests are defined in the directory tests_eos.

tests_eos directory

The directory tests_eos is a python package.

The python functions to test EOS devices are defined the python module functions.py in the python package tests_eos.

generate_functions_documentation.py file

The script generate_functions_documentation.py is used to generate the functions documentation in markdown format. It requires the installation of the package lazydocs that is indicated in the file requirements-dev.txt

The functions to test EOS devices are coded in the python module functions.py in the python package tests_eos. These functions have docstrings. The docstrings are used by the script generate_functions_documentation.py to generate the functions documentation in markdown format in the directory documentation.

documentation directory

The documentation directory has the tests documentation in markdown format:

check-devices-reachability.py file

The python script check-devices-reachability.py is used to test devices reachability with eAPI.

The python script check-devices-reachability.py takes as input a text file with the devices IP address or hostnames (when resolvable), and tests devices reachability with eAPI and prints the unreachable devices on the console.

clear_counters.py file

The python script clear_counters.py is used to clear counters on EOS devices.

It takes as input a text file with the devices IP address or hostnames (when resolvable) and clears counters on these devices.

collect-eos-commands.py file

The python script collect-eos-commands.py is used to collect commands output from EOS devices.

The python script collect-eos-commands.py:

  • Takes as input:
    • A text file with the devices IP address or hostnames (when resolvable).
    • A YAML file with the list of EOS commands we would like to collect in text or JSON format.
  • Collects the EOS commands from the devices, and saves the result in files.

check-devices.py file

The python script check-devices.py is used to run tests on devices.

The python script check-devices.py:

  • Imports the python functions defined in the directory tests_eos.
    • These functions defined the tests.
  • Takes as input:
    • A text file with the devices IP address or hostnames (when resolvable).
    • A YAML file with the list of the tests we would like to use and their parameters.
  • Runs the tests, and prints the result on the console and saves the result in a file.

unit_test.py file

The python script unit_test.py is used to test the functions defined in the directory test_eos without using actual EOS devices. It requires the installation of the package pytest that is indicated in the file requirements-dev.txt

mock_data directory

The mock_data directory has data used by the python script unit_test.py to test the functions defined in the directory test_eos without using actual EOS devices.

Contribution guide

Contributions are welcome. The contribution guide is CONTRIBUTING.md

Credits

Thank you to Paul Lavelle, Colin MacGiollaEain, Matthieu Tache, Angélique Phillipps, Thomas Grimonet, Onur Gashi and Khelil Sator for their contributions and guidances.

Comments
  • feat(anta.inventory): Add support for DNS names, eAPI port and device alias names

    feat(anta.inventory): Add support for DNS names, eAPI port and device alias names

    When using a containerlab topology on macOS, we need to use port forwarding to localhost to access eAPI endpoints. The current inventory only supports IP addresses. The PR is to support hostnames (DNS names instead of IP), ports and alias names in the inventory.

    The following inventory works with this PR:

    anta_inventory:
      hosts:
      - name: spine1
        host: localhost
        port: 44301
      - name: spine2
        host: localhost
        port: 44302
      - name: leaf1
        host: localhost
        port: 44311
      - name: leaf1
        host: localhost
        port: 44312
      - name: leaf3
        host: localhost
        port: 44313
      - name: leaf4
        host: localhost
        port: 44314
    
    opened by mtache 3
  • fix(check-devices): Update --timeout management + add logging management

    fix(check-devices): Update --timeout management + add logging management

    • Fix an issue where timeout was not correctly set.
    • Fix timeout defined as an str where a float was expected
    • Add -log or --logging to set log level for inventory and result_manager

    Default mode

    check-devices.py -i .personal/avd-lab.yml -c .personal/ceos-catalog.yml --tags arista --username admin --password admin123 --table --timeout 1
    ╭──────────────────────── Settings ─────────────────────────╮
    │ Running check-devices with:                               │
    │               - Inventory: .personal/avd-lab.yml          │
    │               - Tests catalog: .personal/ceos-catalog.yml │
    ╰───────────────────────────────────────────────────────────╯
                     All tests results                  
    ┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
    ┃ Device IP ┃ Test Name ┃ Test Status ┃ Message(s) ┃
    ┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
    └───────────┴───────────┴─────────────┴────────────┘
    

    With logging enabled

    ❯ check-devices.py -i .personal/avd-lab.yml -c .personal/ceos-catalog.yml --tags arista --username admin --password admin123 --table --timeout 1 --loglevel info
    ╭──────────────────────── Settings ─────────────────────────╮
    │ Running check-devices with:                               │
    │               - Inventory: .personal/avd-lab.yml          │
    │               - Tests catalog: .personal/ceos-catalog.yml │
    ╰───────────────────────────────────────────────────────────╯
    [14:08:41] WARNING  Service not running on device 10.73.1.254 with:                                                                                                                                                                                                                                                                                    __init__.py:191
               WARNING  connection return: f('admin:[email protected]/command-api', 302, 'Found', <http.client.HTTPMessage object at 0x1062952b0>)                                                                                                                                                                                                      __init__.py:192
               WARNING  Service not running on device 10.73.1.254                                                                                                                                                                                                                                                                                          __init__.py:222
    [14:08:42] WARNING  Service not running on device 10.100.164.150 with:                                                                                                                                                                                                                                                                                 __init__.py:191
               WARNING  connection return: ftimed out                                                                                                                                                                                                                                                                                                      __init__.py:192
    [14:08:43] WARNING  Service not running on device 10.100.164.150                                                                                                                                                                                                                                                                                       __init__.py:222
               INFO     Inventory .personal/avd-lab.yml loaded                                                                                                                                                                                                                                                                                        check-devices.py:202
               INFO     starting running test on inventory ...                                                                                                                                                                                                                                                                                        check-devices.py:208
               INFO     testing done !                                                                                                                                                                                                                                                                                                                check-devices.py:226
               INFO     retrieve list of result using output_format list                                                                                                                                                                                                                                                                                   __init__.py:106
                     All tests results                  
    ┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
    ┃ Device IP ┃ Test Name ┃ Test Status ┃ Message(s) ┃
    ┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
    └───────────┴───────────┴─────────────┴────────────┘
    
    bug 
    opened by titom73 3
  • feat(reporter): Implement initial reporter using rich

    feat(reporter): Implement initial reporter using rich

    Reporting

    Provides easy reporting module

    • Get all tests results:
    ┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃ Device IP    ┃ Test Name           ┃ Test Status ┃ Message(s)                                                                                   ┃
    ┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │ 192.168.0.10 │ verify_eos_version  │ failure     │ device is running version 4.27.3F-26379303.4273F (engineering build) not in expected version │
    │ 192.168.0.11 │ verify_eos_version  │ failure     │ device is running version 4.27.3F-26379303.4273F (engineering build) not in expected version │
    │ 192.168.0.12 │ verify_eos_version  │ failure     │ device is running version 4.27.3F-26379303.4273F (engineering build) not in expected version │
    │ 192.168.0.13 │ verify_eos_version  │ failure     │ device is running version 4.27.3F-26379303.4273F (engineering build) not in expected version │
    │ 192.168.0.14 │ verify_eos_version  │ failure     │ device is running version 4.27.3F-26379303.4273F (engineering build) not in expected version │
    │ 192.168.0.15 │ verify_eos_version  │ failure     │ device is running version 4.27.3F-26379303.4273F (engineering build) not in expected version │
    │ 192.168.0.15 │ verify_uptime       │ success     │                                                                                              │
    │ 192.168.0.15 │ verify_reload_cause │ error       │ 'response'                                                                                   │
    │ 192.168.0.15 │ verify_coredump     │ success     │                                                                                              │
    │ 192.168.0.15 │ verify_agent_logs   │ success     │                                                                                              │
    └──────────────┴─────────────────────┴─────────────┴──────────────────────────────────────────────────────────────────────────────────────────────┘
    
    • Get tests summarized
    ┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃ Test Case          ┃ # of success ┃ # of failure ┃ # of errors ┃ List of failed or error nodes                                                                    ┃
    ┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │ verify_eos_version │ 0            │ 6            │ 0           │ ['192.168.0.10', '192.168.0.11', '192.168.0.12', '192.168.0.13', '192.168.0.14', '192.168.0.15'] │
    └────────────────────┴──────────────┴──────────────┴─────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────┘
    
    • Get hosts summarized
    ┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃ Host IP      ┃ # of success ┃ # of failure ┃ # of errors ┃ List of failed ortest case                    ┃
    ┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │ 192.168.0.15 │ 3            │ 1            │ 1           │ ['verify_eos_version', 'verify_reload_cause'] │
    └──────────────┴──────────────┴──────────────┴─────────────┴───────────────────────────────────────────────┘
    

    Testing

    python .github/scripts/anta-tester.py -h
    [17:03:13] INFO     ANTA program is loaded                                                                                                                                                                                                                                                                                                                 anta-tester.py:99
    usage: anta-tester.py [-h] [--inventory INVENTORY] [--catalog CATALOG] [--username USERNAME] [--password PASSWORD] [--search_ip SEARCH_IP] [--search_test SEARCH_TEST] [--verbose] [--list] [--full] [--devices] [--testcases] [--table] [--save] [--save_file SAVE_FILE]
    
    ANTA test & demo script
    
    optional arguments:
      -h, --help            show this help message and exit
      --inventory INVENTORY, -i INVENTORY
                            ANTA Inventory file
      --catalog CATALOG, -c CATALOG
                            ANTA Tests cagtalog
      --username USERNAME, -u USERNAME
                            EOS Username
      --password PASSWORD, -p PASSWORD
                            EOS Password
      --search_ip SEARCH_IP
                            search result for host
      --search_test SEARCH_TEST
                            search result for test
      --verbose, -v         Set script to verbose mode
      --list, -l            Display internal data
      --full                Display all test cases results
      --devices             Provides summary of test results per device
      --testcases           Provides summary of test results per test case
      --table               Result represented in tables
      --save                Save result to file
      --save_file SAVE_FILE
                            Generated report file
    
    framework-enhancement 
    opened by titom73 2
  • collect sheduled show tech

    collect sheduled show tech

    #!/usr/bin/env python3
    
    ## This script collects all the tech-support files stored on Arista switches flash and copies them locally
    
    import ssl
    import paramiko
    from jsonrpclib import Server
    from scp import SCPClient
    from time import strftime, gmtime
    from argparse import ArgumentParser
    from getpass import getpass
    import os
    from sys import exit
    from socket import setdefaulttimeout
    
    port = 22
    ssl._create_default_https_context = ssl._create_unverified_context
    date = strftime("%d %b %Y %H:%M:%S", gmtime())
    
    def device_directories (device, root_dir):
        cwd = os.getcwd()
        show_tech_directory = cwd + '/' + root_dir
        device_directory = show_tech_directory + '/' + device
        for directory in [show_tech_directory, device_directory]:
            if not os.path.exists(directory):
                os.makedirs(directory)
        result = show_tech_directory, device_directory
        return result
    
    def createSSHClient (device, port, username, password):
        client = paramiko.SSHClient()
        client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        client.connect(device, port, username, password)
        return client
    
    def main():
        parser = ArgumentParser(
            description='Collect all the tech-support files'
            )
        parser.add_argument(
            '-i',
            help='Text file containing a list of switches',
            dest='file',
            required=True
            )
        parser.add_argument(
            '-u',
            help='Devices username',
            dest='username',
            required=True
            )
        parser.add_argument(
            '-o',
            help='Output directory',
            dest='output_directory',
            required=True
            )
        args = parser.parse_args()
        args.password = getpass(prompt='Device password: ')
    
        try:
            with open(args.file, 'r') as file:
                devices = file.readlines()
        except:
            print('Error opening ' + args.file)
            exit(1)
    
        for i,device in enumerate(devices):
            devices[i] = device.strip()
    
        # Deletes unreachable devices from devices list
    
        unreachable = []
        
        print('Connecting to devices .... please be patient ... ')
        
        for device in devices:
            try:
                setdefaulttimeout(5)
                url = 'https://%s:%s@%s/command-api' %(args.username, args.password, device)
                switch = Server(url)
                response = switch.runCmds(1, ['enable'])
            except:
                unreachable.append(device)
        
        for item in unreachable:
            devices.remove(item)
            print("Can not connect to device " + item)
    
        # Collects all the tech-support files stored on Arista switches flash and copies them locally
    
        for device in devices:
            url = "https://" + args.username + ":" + args.password + "@" + device + "/command-api"
            try:
                ## Create one zip file named all_files.zip on the device with the all the show tech-support files in it
                cmds=['enable','bash timeout 30 zip /mnt/flash/schedule/all_files.zip /mnt/flash/schedule/tech-support/*']
                switch = Server(url)
                switch.runCmds(1,cmds, 'text')
                ## Connect to the device using SSH
                ssh = createSSHClient(device, port, args.username, args.password)
                ## Create directories 
                output_dir = device_directories (device, args.output_directory)
                ## Get the zipped file all_files.zip using SCP and save it locally
                my_path = output_dir[1] + '/' + date + '.zip'
                ssh = createSSHClient(device, port, args.username, args.password)
                scp = SCPClient(ssh.get_transport())
                scp.get("/mnt/flash/schedule/all_files.zip",local_path = my_path)
                scp.close()
                ## Delete the created zip file on the device
                cmds=['enable','bash timeout 30 rm /mnt/flash/schedule/all_files.zip']
                switch.runCmds(1,cmds, 'text')
            except:
                print('You are unlucky today! ' + device + ' does not like this script')
    
    if __name__ == '__main__':
        main()
        ```
    opened by ksator 2
  • Implement

    Implement "AntaTest" and "AntaAsyncTest" as protocols

    Sharing some ideas that could improve the framework... Need a way to enforce the format of the test functions to ensure all functions are written correctly (return boolean, etc) and the user can introduce more functions on runtime. Both normal/sequential and async modes could be supported, running the async tests in parallel on a "AntaAsyncTestRunner" and the other functions in a "AntaTestRunner". A test manager could handle the runners and dispatch the tests to each of them delegating the execution. So having the tests modelled in a flexible way could be the first step towards a more generic execution framework that can be customized/extended by the user on runtime.

    opened by gusmb 1
  • split the module `tests` of the package `anta` into multiples modules

    split the module `tests` of the package `anta` into multiples modules

    The module is too large (>1600 lines). we can split it into diff modules:

    • routing
    • hardware
    • interfaces
    • ... This will require an update of the check-devices.py
    opened by ksator 1
  • Scripts can't run due to 'jsonrpclib' incompatibility

    Scripts can't run due to 'jsonrpclib' incompatibility

    I am having an issue running the scripts, as jsonrpclib doesn't work with Python3:

    testvm:~/network_tests_automation# ./check-devices-reachability.py --help
    Traceback (most recent call last):
      File "./check-devices-reachability.py", line 5, in <module>
        from jsonrpclib import Server
    ModuleNotFoundError: No module named 'jsonrpclib'
    

    I've installed PIP dependencies as instructed, and even added 'pip install jsonrpclib' separately:

    test-vm:~/network_tests_automation# /usr/bin/env python3 -V
    Python 3.6.9
    
    testvm:~# pip list | grep jsonrpc
    jsonrpclib (0.1.7)
    jsonrpclib-pelix (0.4.3.2)
    
    

    There seems to be a known issue with this library and Python3: https://stackoverflow.com/questions/38664102/python-jsonrpclib-not-working-after-upgrade-to-python-3-5-2

    Any ideas how to get around this?

    opened by ctyrider 1
  • feat(anta.inventory): use aioeapi to handle a device session

    feat(anta.inventory): use aioeapi to handle a device session

    Using asyncio framework will help improving performances when running tests on a large network with a high number of tests. First step is to refactor the anta.inventory module to use an async library, this PR tries an attempt with aio-eapi. This PR must also refactor all the tests as coroutines and modify the scripts to run the asyncio event loop.


    Modifications to anta.inventory

    The session attribute of InventoryDevice is now a aioeapi.Device object. auto_connect is removed from the constructor: this would cause to run an asyncio event loop in an object constructor and prevent fine control from a ANTA user script on the asyncio scheduling.

    _is_device_online() uses aioeapi.Device.check_connection() to only tries to open a port on the device. _ refresh_device_fact() updates all the device flags and reduce the number of API calls compared to the previous implementation.

    connect_inventory() is a coroutine that needs to be called in a ANTA user script to establish connection towards the devices.

    async def main():
       await anta_inventory.connect_inventory()
    

    When using a containerlab topology on macOS, we need to use port forwarding to localhost to access eAPI endpoints. The current inventory only supports IP addresses. This PR supports hostnames (DNS names instead of IP), ports and alias names in the inventory.

    The following inventory works with this PR:

    anta_inventory:
      hosts:
      - name: spine1
        host: localhost
        port: 44301
      - name: spine2
        host: localhost
        port: 44302
      - name: leaf1
        host: localhost
        port: 44311
      - name: leaf1
        host: localhost
        port: 44312
      - name: leaf3
        host: localhost
        port: 44313
      - name: leaf4
        host: localhost
        port: 44314
    
    opened by mtache 2
  • Add support for custom defined tests

    Add support for custom defined tests

    Description

    In order to allow users to build their own test collection, it would be nice to add option to support catalog pointing to custom libraries.

    Today, we only look at anta.tests package but we could extend YAML format to support both anta.tests and a custom package with the following format:

    <package.name>:
          - <test function>:
               <function options>
    

    An example would be:

    anta.tests.software:
      - verify_eos_version: # Verifies the device is running one of the allowed EOS version.
          versions: # List of allowed EOS versions.
            - 4.25.4M
            - 4.26.1F
    inetsix.tests.mydesign:
      - check_topology:
    

    Potential impacts:

    • anta.loader.parse_catalog
    framework-enhancement 
    opened by titom73 0
  • Add attributes 'label' and 'description' with function decorator

    Add attributes 'label' and 'description' with function decorator

    Hello,

    Currently the only precision about the tests is the name of the function itself. It is therefore necessary to read the code of the function to understand how it works and to have more details.

    An idea would be to add to each test function a python decorator with two attributes: label and description.

    Example with this sample piece of code:

    def decorator(**kwargs):
        def wrapper(function):
            for k, v in kwargs.items():
                setattr(function, k, v)
            return function
        return wrapper
    
    @decorator(label='EOS version', description='Verifies the device is running one of the allowed EOS version')
    def verify_eos_version():
        pass
    
    print(f'label: {verify_eos_version.label}\ndescription: {verify_eos_version.description}')
    
    >>> label: EOS version
    >>> description: Verifies the device is running one of the allowed EOS version
    

    Now, we can access easily this information, without having to add two fields in a .yaml file, for example. Thank you very much.

    opened by noam-zeitoun 2
  • Implement

    Implement "AntaTest" and "AntaAsyncTest" as protocols

    Sharing some ideas that could improve the framework... Need a way to enforce the format of the test functions to ensure all functions are written correctly (return boolean, etc) and the user can introduce more functions on runtime. Both normal/sequential and async modes could be supported, running the async tests in parallel on a "AntaAsyncTestRunner" and the other functions in a "AntaTestRunner". A test manager could handle the runners and dispatch the tests to each of them delegating the execution. So having the tests modelled in a flexible way could be the first step towards a more generic execution framework that can be customized/extended by the user on runtime.

    framework-enhancement 
    opened by gusmb 2
  • Map tags to inventory devices and test catalog

    Map tags to inventory devices and test catalog

    Increase flexibility by using tags on inventory devices and the test catalog, so that catalog tests can run on relevant devices only. For instance, in s typical EVPN scenario there.might be some tests that should run only in spines and some other tests only on leaf switches, having also some tests that run on all devices. Using tags or filters on tests and inventory devices could achieve this easily and add flexibility to the way tests execute. For inventory networks/ranges, tags would apply to the whole range/network...

    framework-enhancement 
    opened by gusmb 1
Releases(v0.3.0)
  • v0.3.0(Sep 26, 2022)

    v0.3.0

    Features

    • 836f995 [anta.reporter] Add skipped column in aggregated results (#110)
    • 8c3404b [inventory] Add tag support to filter test execution (#112)
    • 40f6977 [result_manager] Support JSON output and save option (#113)

    Bug fixes

    • c010bae [check-devices] Update --timeout management + add logging management (#117)
    • cfd9d59 [fix] Missing docstring for tags (#114)

    Documentation

    Other changes

    • 39f80e4 [make] Build system: Prepare version management (#120)
    • 3262f09 [mkdocs] Build system: Fix incorrect requirement path
    Source code(tar.gz)
    Source code(zip)
  • v0.2.0-alpha(Jul 22, 2022)

    Add the Inventory module in the anta package

    Migrated the following scripts to use the Inventory module :

    check-devices-reachability.py
    clear-counters.py
    collect-eos-commands.py
    collect-sheduled-show-tech.py
    evpn-blacklist-recovery.py
    

    This release is not in PyPi

    Source code(tar.gz)
    Source code(zip)
  • v0.1.0(Jul 2, 2022)

    Includes the following package:

    • anta with 53 functions to test EOS

    Includes the following scripts:

    • check-devices-reachability.py
    • check-devices.py
    • clear-counters.py
    • collect-eos-commands.py
    • collect-sheduled-show-tech.py
    • evpn-blacklist-recovery.py

    This release is not in PyPi

    Source code(tar.gz)
    Source code(zip)
Owner
Netdevops Community
Arista Network Automation Community (Community projects only, no direct affiliation)
Netdevops Community
nose is nicer testing for python

On some platforms, brp-compress zips man pages without distutils knowing about it. This results in an error when building an rpm for nose. The rpm bui

1.4k Dec 12, 2022
This is a simple software for fetching new changes to remote repositories automatically.

Git Autofetch Git Autofetch is a simple software for fetching new changes from a repo to local repositories after a set time interval. This program is

Shreyas Ashtamkar 10 Jul 21, 2022
Nokia SR OS automation

Nokia SR OS automation Nokia is one of the biggest vendors of the telecommunication equipment, which is very popular in the Service Provider segment.

Karneliuk.com 7 Jul 23, 2022
Automação de Processos (obtenção de informações com o Selenium), atualização de Planilha e Envio de E-mail.

Automação de Processo: Código para acompanhar o valor de algumas ações na B3. O código entra no Google Drive, puxa os valores das ações (pré estabelec

Hemili Beatriz 1 Jan 08, 2022
Automated tests for OKAY websites in Python (Selenium) - user friendly version

Okay Selenium Testy Aplikace určená k testování produkčních webů společnosti OKAY s.r.o. Závislosti K běhu aplikace je potřeba mít v počítači nainstal

Viktor Bem 0 Oct 01, 2022
Test django schema and data migrations, including migrations' order and best practices.

django-test-migrations Features Allows to test django schema and data migrations Allows to test both forward and rollback migrations Allows to test th

wemake.services 382 Dec 27, 2022
Generates realistic traffic for load testing tile servers

Generates realistic traffic for load testing tile servers. Useful for: Measuring throughput, latency and concurrency of your tile serving stack. Ident

Brandon Liu 23 Dec 05, 2022
Webscreener is a tool for mass web domains pentesting.

Webscreener is a tool for mass web domains pentesting. It is used to take snapshots for domains that is generated by a tool like knockpy or Sublist3r. It cuts out most of the pentesting time by scree

Seekurity 3 Jun 07, 2021
A set of pytest fixtures to test Flask applications

pytest-flask An extension of pytest test runner which provides a set of useful tools to simplify testing and development of the Flask extensions and a

pytest-dev 433 Dec 23, 2022
Playwright Python tool practice pytest pytest-bdd screen-play page-object allure cucumber-report

pytest-ui-automatic Playwright Python tool practice pytest pytest-bdd screen-play page-object allure cucumber-report How to run Run tests execute_test

moyu6027 11 Nov 08, 2022
A configurable set of panels that display various debug information about the current request/response.

Django Debug Toolbar The Django Debug Toolbar is a configurable set of panels that display various debug information about the current request/respons

Jazzband 7.3k Jan 02, 2023
This repository has automation content to test Arista devices.

Network tests automation Network tests automation About this repository Requirements Requirements on your laptop Requirements on the switches Quick te

Netdevops Community 17 Nov 04, 2022
A collection of benchmarking tools.

Benchmark Utilities About A collection of benchmarking tools. PYPI Package Table of Contents Using the library Installing and using the library Manual

Kostas Georgiou 2 Jan 28, 2022
An improbable web debugger through WebSockets

wdb - Web Debugger Description wdb is a full featured web debugger based on a client-server architecture. The wdb server which is responsible of manag

Kozea 1.6k Dec 09, 2022
splinter - python test framework for web applications

splinter - python tool for testing web applications splinter is an open source tool for testing web applications using Python. It lets you automate br

Cobra Team 2.6k Dec 27, 2022
Minimal example of how to use pytest with automated 'devops' style automated test runs

Pytest python example with automated testing This is a minimal viable example of pytest with an automated run of tests for every push/merge into the m

Karma Computing 2 Jan 02, 2022
Lightweight, scriptable browser as a service with an HTTP API

Splash - A javascript rendering service Splash is a javascript rendering service with an HTTP API. It's a lightweight browser with an HTTP API, implem

Scrapinghub 3.8k Jan 03, 2023
Python Moonlight (Machine Learning) Practice

PyML Python Moonlight (Machine Learning) Practice Contents Design Documentation Prerequisites Checklist Dev Setup Testing Run Prerequisites Python 3 P

Dockerian Seattle 2 Dec 25, 2022
Load Testing ML Microservices for Robustness and Scalability

The demo is aimed at getting started with load testing a microservice before taking it to production. We use FastAPI microservice (to predict weather) and Locust to load test the service (locally or

Emmanuel Raj 13 Jul 05, 2022
Youtube Tool using selenium Python

YT-AutoLikeComment-AutoReportComment-AutoComment Youtube Tool using selenium Python Auto Comment Auto Like Comment Auto Report Comment Usage: 1. Insta

Rahul Joshua Damanik 1 Dec 13, 2021