Python scripts for a generic performance testing infrastructure using Locust.

Related tags

TestingLocust_Scripts
Overview

TODOs

  • Reference to published paper or online version of it
  • loadtest_plotter.py: Cleanup and reading data from files
  • ARS_simulation.py: Cleanup, documentation and control workloads and parameters of the simulation model through CLI
  • locust-parameter-variation.py: Cleanup and Documentation
  • Move the files into subfolders (Executors, Load Testers, Evaluators, Systems under Test)

Locust Performance Testing Infrastructure

In [1] we introduced a generic performance testing infrastructure and used it in an industrial case study. Our idea is to have decoupled components, Python scripts in our case, that together allow to:

  1. reproducible execute a load testing tool with a set of parameters for a particular experiment,
  2. evaluate the performance measurements assisted by visualizations or automatic evaluators.

Generally, we have four types of components in our infrastructure:

  • Executors: execute a particular Load Tester as long as the Load Tester provides a CLI or an API;
  • Load Testers: execute the load test, parametrized with values given by an Executor. Have to output a logfile containing the response times;
  • Evaluators: postprocess the logfile and for example plot the response times;
  • Systems under Test (SUTs): Target systems we want to test. Usually, the target systems will be external systems, e.g., web servers. In our case, we build software that simulates the behavior of a real system, in order to provide the means for others to roughly reproduce our experiments.

More details about our generic performance testing infrastructure can be found in our paper [1].

This repository contains the aforementioned Python scripts:

  • Executors:
    • executor.py: executes Locust with a set of parameters;
    • locust-parameter-variation.py: executes Locust and keeps increasing the load. This is similar to Locust's Step Load Mode, however, our approach increases the number of clients for as long as the ARS complies with real-time requirements in order to find the saturation point of the ARS.
  • Load Testers:
    • locust_tester.py: contains specific code for Locust to perform the actual performance test. For demonstration purposes, this script tests ARS_simulation.py. Outputs a locust_log.log;
    • locust_multiple_requests: an enhanced version of locust_tester that sends additional requests to generate more load.
    • locust_teastore.py: performs load testing against TeaStore, or our simulated TeaStore.
  • Evaluators:
    • loadtest_plotter.py: reads the locust_log.log, plots response times, and additional metrics to better visualize, if the real-time requirements of the EN 50136 are met.
  • SUTs
    • Alarm Receiving Software Simulation (ARS_simulation.py): simulates an industrial ARS based on data measured in the production environment of the GS company group.
    • TeaStore (teastore_simulation.py): simulates TeaStore based on a predictive model generated in a lab environment.

Instructions to reproduce results in our paper

Quick start

  • Clone the repository;
  • run pip3 install -r requirements.txt;
  • In the file ARS_simulation.py make sure that the constant MASCOTS2020 is set to True.
  • open two terminal shells:
    1. run python3 ARS_simulation.py in one of them;
    2. run python3 executor.py. in the other.
  • to stop the test, terminate the executor.py script;
  • run python3 loadtest_plotter.py, pass the locust_log.log and see the results. :)

Details

Using the performance testing infrastructure available in this repository, we conducted performance tests in a real-world alarm system provided by the GS company. To provide a way to reproduce our results without the particular alarm system, we build a software simulating the Alarm Receiving Software. The simulation model uses variables, we identified as relevant and also performed some measurements in the production environment, to initialize the variables correctly.

To reproduce our results, follow the steps in the Section "Quick start". The scripts are already preconfigured, to simulate a realistic workload, inject faults, and automatically recover from them. The recovery is performed after the time, the real fault management mechanism requires.

If you follow the steps and, for example, let the test run for about an hour, you will get similar results to the ones you can find in the Folder "Tests under Fault".

Results after running our scripts for about an hour:

Results


Keep in mind that we use a simulated ARS here; in our paper we present measurements performed with a real system, thus the results reproduced with the code here are slightly different.

Nonetheless, the overall observations we made in our paper, are in fact reproducible.


Instructions on how to adapt our performance testing infrastructure to other uses

After cloning the repository, take a look at the locust_tester.py. This is, basically, an ordinary Locust script that sends request to the target system and measures the response time, when the response arrives. Our locust_tester.py is special, because:

  • we implemented a custom client instead of using the default;
  • we additionally log the response times to a logfile instead of using the .csv files Locust provides.

So, write a performance test using Locust, following the instructions of the Locust developers on how to write a Locust script. The only thing to keep in mind is, that your Locust script has to output the measured response times to a logfile in the same way our script does it. Use logger.info("Response time %s ms", total_time) to log the response times.

When you have your Locust script ready, execute it with python3 executor.py, pass the path to your script as argument, and when you want to finish the load test, terminate it with Ctrl + C.

Use python3 executor.py --help to get additional information.

Example call:

% python3 executor.py locust_scripts/locust_tester.py

After that, plot your results:

% python3 loadtest_plotter.py
Path to the logfile: locust_log.log
Owner
Juri Tomak
Juri Tomak
Obsei is a low code AI powered automation tool.

Obsei is a low code AI powered automation tool. It can be used in various business flows like social listening, AI based alerting, brand image analysis, comparative study and more .

Obsei 782 Dec 31, 2022
Free cleverbot without headless browser

Cleverbot Scraper Simple free cleverbot library that doesn't require running a heavy ram wasting headless web browser to actually chat with the bot, a

Matheus Fillipe 3 Sep 25, 2022
Django-google-optimize is a Django application designed to make running server side Google Optimize A/B tests easy.

Django-google-optimize Django-google-optimize is a Django application designed to make running Google Optimize A/B tests easy. Here is a tutorial on t

Adin Hodovic 39 Oct 25, 2022
Language-agnostic HTTP API Testing Tool

Dredd — HTTP API Testing Framework Dredd is a language-agnostic command-line tool for validating API description document against backend implementati

Apiary 4k Jan 05, 2023
Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)

Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)

Leon 3.5k Dec 30, 2022
A Library for Working with Sauce Labs

Robotframework - Sauce Labs Plugin This is a plugin for the SeleniumLibrary to help with using Sauce Labs. This library is a plugin extension of the S

joshin4colours 6 Oct 12, 2021
The best, free, all in one, multichecking, pentesting utility

The best, free, all in one, multichecking, pentesting utility

Mickey 58 Jan 03, 2023
A automated browsing experience.

browser-automation This app is an automated browsing technique where one has to enter the required information, it's just like searching for Animals o

Ojas Barawal 3 Aug 04, 2021
MultiPy lets you conveniently keep track of your python scripts for personal use or showcase by loading and grouping them into categories. It allows you to either run each script individually or together with just one click.

MultiPy About MultiPy is a graphical user interface built using Dear PyGui Python GUI Framework that lets you conveniently keep track of your python s

56 Oct 29, 2022
PyAutoEasy is a extension / wrapper around the famous PyAutoGUI, a cross-platform GUI automation tool to replace your boooring repetitive tasks.

PyAutoEasy PyAutoEasy is a extension / wrapper around the famous PyAutoGUI, a cross-platform GUI automation tool to replace your boooring repetitive t

Dingu Sagar 7 Oct 27, 2022
AutoExploitSwagger is an automated API security testing exploit tool that can be combined with xray, BurpSuite and other scanners.

AutoExploitSwagger is an automated API security testing exploit tool that can be combined with xray, BurpSuite and other scanners.

6 Jan 28, 2022
A rewrite of Python's builtin doctest module (with pytest plugin integration) but without all the weirdness

The xdoctest package is a re-write of Python's builtin doctest module. It replaces the old regex-based parser with a new abstract-syntax-tree based pa

Jon Crall 174 Dec 16, 2022
Python script to automatically download from Zippyshare

Zippyshare downloader and Links Extractor Python script to automatically download from Zippyshare using Selenium package and Internet Download Manager

Daksh Khurana 2 Oct 31, 2022
Redis fixtures and fixture factories for Pytest.

Redis fixtures and fixture factories for Pytest.This is a pytest plugin, that enables you to test your code that relies on a running Redis database. It allows you to specify additional fixtures for R

Clearcode 86 Dec 23, 2022
hyppo is an open-source software package for multivariate hypothesis testing.

hyppo (HYPothesis Testing in PythOn, pronounced "Hippo") is an open-source software package for multivariate hypothesis testing.

neurodata 137 Dec 18, 2022
A Proof of concept of a modern python CLI with click, pydantic, rich and anyio

httpcli This project is a proof of concept of a modern python networking cli which can be simple and easy to maintain using some of the best packages

Kevin Tewouda 17 Nov 15, 2022
WrightEagle AutoTest (Has been updated by Cyrus team members)

Autotest2d WrightEagle AutoTest (Has been updated by Cyrus team members) Thanks go to WrightEagle Members. Steps 1- prepare start_team file. In this s

Cyrus Soccer Simulation 2D Team 3 Sep 01, 2022
Test utility for validating OpenAPI documentation

DRF OpenAPI Tester This is a test utility to validate DRF Test Responses against OpenAPI 2 and 3 schema. It has built-in support for: OpenAPI 2/3 yaml

snok 103 Dec 21, 2022
Data App Performance Tests

Data App Performance Tests My hypothesis is that The different architectures of

Marc Skov Madsen 6 Dec 14, 2022
A single module to link Python ecosystem to the Web

A single module to link Python ecosystem to the Web. Have a quick look at the Gallery first to get convinced ! FAQ For any questions, please use Stack

66 Dec 21, 2022