Python scripts for a generic performance testing infrastructure using Locust.

Related tags

TestingLocust_Scripts
Overview

TODOs

  • Reference to published paper or online version of it
  • loadtest_plotter.py: Cleanup and reading data from files
  • ARS_simulation.py: Cleanup, documentation and control workloads and parameters of the simulation model through CLI
  • locust-parameter-variation.py: Cleanup and Documentation
  • Move the files into subfolders (Executors, Load Testers, Evaluators, Systems under Test)

Locust Performance Testing Infrastructure

In [1] we introduced a generic performance testing infrastructure and used it in an industrial case study. Our idea is to have decoupled components, Python scripts in our case, that together allow to:

  1. reproducible execute a load testing tool with a set of parameters for a particular experiment,
  2. evaluate the performance measurements assisted by visualizations or automatic evaluators.

Generally, we have four types of components in our infrastructure:

  • Executors: execute a particular Load Tester as long as the Load Tester provides a CLI or an API;
  • Load Testers: execute the load test, parametrized with values given by an Executor. Have to output a logfile containing the response times;
  • Evaluators: postprocess the logfile and for example plot the response times;
  • Systems under Test (SUTs): Target systems we want to test. Usually, the target systems will be external systems, e.g., web servers. In our case, we build software that simulates the behavior of a real system, in order to provide the means for others to roughly reproduce our experiments.

More details about our generic performance testing infrastructure can be found in our paper [1].

This repository contains the aforementioned Python scripts:

  • Executors:
    • executor.py: executes Locust with a set of parameters;
    • locust-parameter-variation.py: executes Locust and keeps increasing the load. This is similar to Locust's Step Load Mode, however, our approach increases the number of clients for as long as the ARS complies with real-time requirements in order to find the saturation point of the ARS.
  • Load Testers:
    • locust_tester.py: contains specific code for Locust to perform the actual performance test. For demonstration purposes, this script tests ARS_simulation.py. Outputs a locust_log.log;
    • locust_multiple_requests: an enhanced version of locust_tester that sends additional requests to generate more load.
    • locust_teastore.py: performs load testing against TeaStore, or our simulated TeaStore.
  • Evaluators:
    • loadtest_plotter.py: reads the locust_log.log, plots response times, and additional metrics to better visualize, if the real-time requirements of the EN 50136 are met.
  • SUTs
    • Alarm Receiving Software Simulation (ARS_simulation.py): simulates an industrial ARS based on data measured in the production environment of the GS company group.
    • TeaStore (teastore_simulation.py): simulates TeaStore based on a predictive model generated in a lab environment.

Instructions to reproduce results in our paper

Quick start

  • Clone the repository;
  • run pip3 install -r requirements.txt;
  • In the file ARS_simulation.py make sure that the constant MASCOTS2020 is set to True.
  • open two terminal shells:
    1. run python3 ARS_simulation.py in one of them;
    2. run python3 executor.py. in the other.
  • to stop the test, terminate the executor.py script;
  • run python3 loadtest_plotter.py, pass the locust_log.log and see the results. :)

Details

Using the performance testing infrastructure available in this repository, we conducted performance tests in a real-world alarm system provided by the GS company. To provide a way to reproduce our results without the particular alarm system, we build a software simulating the Alarm Receiving Software. The simulation model uses variables, we identified as relevant and also performed some measurements in the production environment, to initialize the variables correctly.

To reproduce our results, follow the steps in the Section "Quick start". The scripts are already preconfigured, to simulate a realistic workload, inject faults, and automatically recover from them. The recovery is performed after the time, the real fault management mechanism requires.

If you follow the steps and, for example, let the test run for about an hour, you will get similar results to the ones you can find in the Folder "Tests under Fault".

Results after running our scripts for about an hour:

Results


Keep in mind that we use a simulated ARS here; in our paper we present measurements performed with a real system, thus the results reproduced with the code here are slightly different.

Nonetheless, the overall observations we made in our paper, are in fact reproducible.


Instructions on how to adapt our performance testing infrastructure to other uses

After cloning the repository, take a look at the locust_tester.py. This is, basically, an ordinary Locust script that sends request to the target system and measures the response time, when the response arrives. Our locust_tester.py is special, because:

  • we implemented a custom client instead of using the default;
  • we additionally log the response times to a logfile instead of using the .csv files Locust provides.

So, write a performance test using Locust, following the instructions of the Locust developers on how to write a Locust script. The only thing to keep in mind is, that your Locust script has to output the measured response times to a logfile in the same way our script does it. Use logger.info("Response time %s ms", total_time) to log the response times.

When you have your Locust script ready, execute it with python3 executor.py, pass the path to your script as argument, and when you want to finish the load test, terminate it with Ctrl + C.

Use python3 executor.py --help to get additional information.

Example call:

% python3 executor.py locust_scripts/locust_tester.py

After that, plot your results:

% python3 loadtest_plotter.py
Path to the logfile: locust_log.log
Owner
Juri Tomak
Juri Tomak
Argument matchers for unittest.mock

callee Argument matchers for unittest.mock More robust tests Python's mocking library (or its backport for Python 3.3) is simple, reliable, and easy

Karol Kuczmarski 77 Nov 03, 2022
masscan + nmap 快速端口存活检测和服务识别

masnmap masscan + nmap 快速端口存活检测和服务识别。 思路很简单,将masscan在端口探测的高速和nmap服务探测的准确性结合起来,达到一种相对比较理想的效果。 先使用masscan以较高速率对ip存活端口进行探测,再以多进程的方式,使用nmap对开放的端口进行服务探测。 安

starnightcyber 75 Dec 19, 2022
The evaluator covering all of the metrics required by tasks within the DUE Benchmark.

DUE Evaluator The repository contains the evaluator covering all of the metrics required by tasks within the DUE Benchmark, i.e., set-based F1 (for KI

DUE Benchmark 4 Jan 21, 2022
A python bot using the Selenium library to auto-buy specified sneakers on the nike.com website.

Sneaker-Bot-UK A python bot using the Selenium library to auto-buy specified sneakers on the nike.com website. This bot is still in development and is

Daniel Hinds 4 Dec 14, 2022
A simple Python script I wrote that scrapes NASA's James Webb Space Telescope tracker website using Selenium and returns its current status and location.

A simple Python script I wrote that scrapes NASA's James Webb Space Telescope tracker website using Selenium and returns its current status and location.

9 Feb 10, 2022
PyAutoEasy is a extension / wrapper around the famous PyAutoGUI, a cross-platform GUI automation tool to replace your boooring repetitive tasks.

PyAutoEasy PyAutoEasy is a extension / wrapper around the famous PyAutoGUI, a cross-platform GUI automation tool to replace your boooring repetitive t

Dingu Sagar 7 Oct 27, 2022
Donors data of Tamil Nadu Chief Ministers Relief Fund scrapped from https://ereceipt.tn.gov.in/cmprf/Interface/CMPRF/MonthWiseReport

Tamil Nadu Chief Minister's Relief Fund Donors Scrapped data from https://ereceipt.tn.gov.in/cmprf/Interface/CMPRF/MonthWiseReport Scrapper scrapper.p

Arunmozhi 5 May 18, 2021
A pure Python script to easily get a reverse shell

easy-shell A pure Python script to easily get a reverse shell. How it works? After sending a request, it generates a payload with different commands a

Cristian Souza 48 Dec 12, 2022
Getting the most out of your hobby servo

ServoProject by Adam Bäckström Getting the most out of your hobby servo Theory The control system of a regular hobby servo looks something like this:

209 Dec 20, 2022
Web testing library for Robot Framework

SeleniumLibrary Contents Introduction Keyword Documentation Installation Browser drivers Usage Extending SeleniumLibrary Community Versions History In

Robot Framework 1.2k Jan 03, 2023
AutoExploitSwagger is an automated API security testing exploit tool that can be combined with xray, BurpSuite and other scanners.

AutoExploitSwagger is an automated API security testing exploit tool that can be combined with xray, BurpSuite and other scanners.

6 Jan 28, 2022
LuluTest is a Python framework for creating automated browser tests.

LuluTest LuluTest is an open source browser automation framework using Python and Selenium. It is relatively lightweight in that it mostly provides wr

Erik Whiting 14 Sep 26, 2022
Wraps any WSGI application and makes it easy to send test requests to that application, without starting up an HTTP server.

WebTest This wraps any WSGI application and makes it easy to send test requests to that application, without starting up an HTTP server. This provides

Pylons Project 325 Dec 30, 2022
🐍 Material for PyData Global 2021 Presentation: Effective Testing for Machine Learning Projects

Effective Testing for Machine Learning Projects Code for PyData Global 2021 Presentation by @edublancas. Slides available here. The project is develop

Eduardo Blancas 73 Nov 06, 2022
Kent - Fake Sentry server for local development, debugging, and integration testing

Kent is a service for debugging and integration testing Sentry.

Will Kahn-Greene 100 Dec 15, 2022
A modern API testing tool for web applications built with Open API and GraphQL specifications.

Schemathesis Schemathesis is a modern API testing tool for web applications built with Open API and GraphQL specifications. It reads the application s

Schemathesis.io 1.6k Dec 30, 2022
Useful additions to Django's default TestCase

django-test-plus Useful additions to Django's default TestCase from REVSYS Rationale Let's face it, writing tests isn't always fun. Part of the reason

REVSYS 546 Dec 22, 2022
This repository contnains sample problems with test cases using Cormen-Lib

Cormen Lib Sample Problems Description This repository contnains sample problems with test cases using Cormen-Lib. These problems were made for the pu

Cormen Lib 3 Jun 30, 2022
A rewrite of Python's builtin doctest module (with pytest plugin integration) but without all the weirdness

The xdoctest package is a re-write of Python's builtin doctest module. It replaces the old regex-based parser with a new abstract-syntax-tree based pa

Jon Crall 174 Dec 16, 2022