:game_die: Pytest plugin to randomly order tests and control random.seed

Related tags

Testingpytest
Overview

pytest-randomly

https://img.shields.io/github/workflow/status/pytest-dev/pytest-randomly/CI/main?style=for-the-badge https://img.shields.io/pypi/v/pytest-randomly.svg?style=for-the-badge https://img.shields.io/badge/code%20style-black-000000.svg?style=for-the-badge pre-commit
Randomness power.

Pytest plugin to randomly order tests and control random.seed.

Features

All of these features are on by default but can be disabled with flags.

  • Randomly shuffles the order of test items. This is done first at the level of modules, then at the level of test classes (if you have them), then at the order of functions. This also works with things like doctests.
  • Resets random.seed() at the start of every test case and test to a fixed number - this defaults to time.time() from the start of your test run, but you can pass in --randomly-seed to repeat a randomness-induced failure.
  • If factory boy is installed, its random state is reset at the start of every test. This allows for repeatable use of its random 'fuzzy' features.
  • If faker is installed, its random state is reset at the start of every test. This is also for repeatable fuzzy data in tests - factory boy uses faker for lots of data. This is also done if you're using the faker pytest fixture, by defining the faker_seed fixture (docs).
  • If numpy is installed, its random state is reset at the start of every test.
  • If additional random generators are used, they can be registered under the pytest_randomly.random_seeder entry point and will have their seed reset at the start of every test. Register a function that takes the current seed value.
  • Works with pytest-xdist.

About

Randomness in testing can be quite powerful to discover hidden flaws in the tests themselves, as well as giving a little more coverage to your system.

By randomly ordering the tests, the risk of surprising inter-test dependencies is reduced - a technique used in many places, for example Google's C++ test runner googletest. Research suggests that "dependent tests do exist in practice" and a random order of test executions can effectively detect such dependencies [1]. Alternatively, a reverse order of test executions, as provided by pytest-reverse, may find less dependent tests but can achieve a better benefit/cost ratio.

By resetting the random seed to a repeatable number for each test, tests can create data based on random numbers and yet remain repeatable, for example factory boy's fuzzy values. This is good for ensuring that tests specify the data they need and that the tested system is not affected by any data that is filled in randomly due to not being specified.

I have written a blog post covering the history of pytest-randomly, including how it started life as the nose plugin nose-randomly.

Additionally, I appeared on the Test and Code podcast to talk about pytest-randomly.

Installation

Install from pip with:

python -m pip install pytest-randomly

Python 3.6 to 3.9 supported.


Testing a Django project? Check out my book Speed Up Your Django Tests which covers loads of best practices so you can write faster, more accurate tests.


Usage

Pytest will automatically find the plugin and use it when you run pytest. The output will start with an extra line that tells you the random seed that is being used:

$ pytest
...
platform darwin -- Python 3.7.2, pytest-4.3.1, py-1.8.0, pluggy-0.9.0
Using --randomly-seed=1553614239
...

If the tests fail due to ordering or randomly created data, you can restart them with that seed using the flag as suggested:

pytest --randomly-seed=1234

Or more conveniently, use the special value last:

pytest --randomly-seed=last

Since the ordering is by module, then by class, you can debug inter-test pollution failures by narrowing down which tests are being run to find the bad interaction by rerunning just the module/class:

pytest --randomly-seed=1234 tests/module_that_failed/

You can disable behaviours you don't like with the following flags:

  • --randomly-dont-reset-seed - turn off the reset of random.seed() at the start of every test
  • --randomly-dont-reorganize - turn off the shuffling of the order of tests

The plugin appears to Pytest with the name 'randomly'. To disable it altogether, you can use the -p argument, for example:

pytest -p no:randomly

Entry Point

If you're using a different randomness generator in your third party package, you can register an entrypoint to be called every time pytest-randomly reseeds. Implement the entrypoint pytest_randomly.random_seeder, referring to a function/callable that takes one argument, the new seed (int).

For example in your setup.cfg:

[options.entry_points]
pytest_randomly.random_seeder =
    mypackage = mypackage.reseed

Then implement reseed(new_seed).

References

[1] Sai Zhang, Darioush Jalali, Jochen Wuttke, Kıvanç Muşlu, Wing Lam, Michael D. Ernst, and David Notkin. 2014. Empirically revisiting the test independence assumption. In Proceedings of the 2014 International Symposium on Software Testing and Analysis (ISSTA 2014). Association for Computing Machinery, New York, NY, USA, 385–396. doi:https://doi.org/10.1145/2610384.2610404
Owner
pytest-dev
pytest-dev
A Python program that will log into your scheduled Google Meets hands free

Chrome GMautomation General Information This Python program will open up Chrome and log into your scheduled Google Meet with camera and mic turned off

Jonathan Leow 5 Dec 31, 2021
The best, free, all in one, multichecking, pentesting utility

The best, free, all in one, multichecking, pentesting utility

Mickey 58 Jan 03, 2023
Simple assertion library for unit testing in python with a fluent API

assertpy Simple assertions library for unit testing in Python with a nice fluent API. Supports both Python 2 and 3. Usage Just import the assert_that

19 Sep 10, 2022
输入Google Hacking语句,自动调用Chrome浏览器爬取结果

Google-Hacking-Crawler 该脚本可输入Google Hacking语句,自动调用Chrome浏览器爬取结果 环境配置 python -m pip install -r requirements.txt 下载Chrome浏览器

Jarcis 4 Jun 21, 2022
Enabling easy statistical significance testing for deep neural networks.

deep-significance: Easy and Better Significance Testing for Deep Neural Networks Contents ⁉️ Why 📥 Installation 🔖 Examples Intermezzo: Almost Stocha

Dennis Ulmer 270 Dec 20, 2022
Python Moonlight (Machine Learning) Practice

PyML Python Moonlight (Machine Learning) Practice Contents Design Documentation Prerequisites Checklist Dev Setup Testing Run Prerequisites Python 3 P

Dockerian Seattle 2 Dec 25, 2022
HTTP traffic mocking and testing made easy in Python

pook Versatile, expressive and hackable utility library for HTTP traffic mocking and expectations made easy in Python. Heavily inspired by gock. To ge

Tom 305 Dec 23, 2022
A single module to link Python ecosystem to the Web

A single module to link Python ecosystem to the Web. Have a quick look at the Gallery first to get convinced ! FAQ For any questions, please use Stack

66 Dec 21, 2022
Python 3 wrapper of Microsoft UIAutomation. Support UIAutomation for MFC, WindowsForm, WPF, Modern UI(Metro UI), Qt, IE, Firefox, Chrome ...

Python 3 wrapper of Microsoft UIAutomation. Support UIAutomation for MFC, WindowsForm, WPF, Modern UI(Metro UI), Qt, IE, Firefox, Chrome ...

yin kaisheng 1.6k Dec 29, 2022
Ab testing - The using AB test to test of difference of conversion rate

Facebook recently introduced a new type of offer that is an alternative to the current type of bidding called maximum bidding he introduced average bidding.

5 Nov 21, 2022
Just a small test with lists in cython

Test for lists in cython Algorithm create a list of 10^4 lists each with 10^4 floats values (namely: 0.1) - 2 nested for iterate each list and compute

Federico Simonetta 32 Jul 23, 2022
A test fixtures replacement for Python

factory_boy factory_boy is a fixtures replacement based on thoughtbot's factory_bot. As a fixtures replacement tool, it aims to replace static, hard t

FactoryBoy project 3k Jan 05, 2023
A twitter bot that simply replies with a beautiful screenshot of the tweet, powered by poet.so

Poet this! Replies with a beautiful screenshot of the tweet, powered by poet.so Installation git clone https://github.com/dhravya/poet-this.git cd po

Dhravya Shah 30 Dec 04, 2022
A set of pytest fixtures to test Flask applications

pytest-flask An extension of pytest test runner which provides a set of useful tools to simplify testing and development of the Flask extensions and a

pytest-dev 433 Dec 23, 2022
Make Selenium work on Github Actions

Make Selenium work on Github Actions Scraping with BeautifulSoup on GitHub Actions is easy-peasy. But what about Selenium?? After you jump through som

Jonathan Soma 33 Dec 27, 2022
Automação de Processos (obtenção de informações com o Selenium), atualização de Planilha e Envio de E-mail.

Automação de Processo: Código para acompanhar o valor de algumas ações na B3. O código entra no Google Drive, puxa os valores das ações (pré estabelec

Hemili Beatriz 1 Jan 08, 2022
Youtube Tool using selenium Python

YT-AutoLikeComment-AutoReportComment-AutoComment Youtube Tool using selenium Python Auto Comment Auto Like Comment Auto Report Comment Usage: 1. Insta

Rahul Joshua Damanik 1 Dec 13, 2021
Asyncio http mocking. Similar to the responses library used for 'requests'

aresponses an asyncio testing server for mocking external services Features Fast mocks using actual network connections allows mocking some types of n

93 Nov 16, 2022
Local continuous test runner with pytest and watchdog.

pytest-watch -- Continuous pytest runner pytest-watch a zero-config CLI tool that runs pytest, and re-runs it when a file in your project changes. It

Joe Esposito 675 Dec 23, 2022
Generates realistic traffic for load testing tile servers

Generates realistic traffic for load testing tile servers. Useful for: Measuring throughput, latency and concurrency of your tile serving stack. Ident

Brandon Liu 23 Dec 05, 2022