Parameterized testing with any Python test framework

Related tags

Testingparameterized
Overview

Parameterized testing with any Python test framework

PyPI PyPI - Downloads Circle CI

Parameterized testing in Python sucks.

parameterized fixes that. For everything. Parameterized testing for nose, parameterized testing for py.test, parameterized testing for unittest.

# test_math.py
from nose.tools import assert_equal
from parameterized import parameterized, parameterized_class

import unittest
import math

@parameterized([
    (2, 2, 4),
    (2, 3, 8),
    (1, 9, 1),
    (0, 9, 0),
])
def test_pow(base, exponent, expected):
   assert_equal(math.pow(base, exponent), expected)

class TestMathUnitTest(unittest.TestCase):
   @parameterized.expand([
       ("negative", -1.5, -2.0),
       ("integer", 1, 1.0),
       ("large fraction", 1.6, 1),
   ])
   def test_floor(self, name, input, expected):
       assert_equal(math.floor(input), expected)

@parameterized_class(('a', 'b', 'expected_sum', 'expected_product'), [
   (1, 2, 3, 2),
   (5, 5, 10, 25),
])
class TestMathClass(unittest.TestCase):
   def test_add(self):
      assert_equal(self.a + self.b, self.expected_sum)

   def test_multiply(self):
      assert_equal(self.a * self.b, self.expected_product)

@parameterized_class([
   { "a": 3, "expected": 2 },
   { "b": 5, "expected": -4 },
])
class TestMathClassDict(unittest.TestCase):
   a = 1
   b = 1

   def test_subtract(self):
      assert_equal(self.a - self.b, self.expected)

With nose (and nose2):

$ nosetests -v test_math.py
test_floor_0_negative (test_math.TestMathUnitTest) ... ok
test_floor_1_integer (test_math.TestMathUnitTest) ... ok
test_floor_2_large_fraction (test_math.TestMathUnitTest) ... ok
test_math.test_pow(2, 2, 4, {}) ... ok
test_math.test_pow(2, 3, 8, {}) ... ok
test_math.test_pow(1, 9, 1, {}) ... ok
test_math.test_pow(0, 9, 0, {}) ... ok
test_add (test_math.TestMathClass_0) ... ok
test_multiply (test_math.TestMathClass_0) ... ok
test_add (test_math.TestMathClass_1) ... ok
test_multiply (test_math.TestMathClass_1) ... ok
test_subtract (test_math.TestMathClassDict_0) ... ok

----------------------------------------------------------------------
Ran 12 tests in 0.015s

OK

As the package name suggests, nose is best supported and will be used for all further examples.

With py.test (version 2.0 and above):

$ py.test -v test_math.py
============================= test session starts ==============================
platform darwin -- Python 3.6.1, pytest-3.1.3, py-1.4.34, pluggy-0.4.0
collecting ... collected 13 items

test_math.py::test_pow::[0] PASSED
test_math.py::test_pow::[1] PASSED
test_math.py::test_pow::[2] PASSED
test_math.py::test_pow::[3] PASSED
test_math.py::TestMathUnitTest::test_floor_0_negative PASSED
test_math.py::TestMathUnitTest::test_floor_1_integer PASSED
test_math.py::TestMathUnitTest::test_floor_2_large_fraction PASSED
test_math.py::TestMathClass_0::test_add PASSED
test_math.py::TestMathClass_0::test_multiply PASSED
test_math.py::TestMathClass_1::test_add PASSED
test_math.py::TestMathClass_1::test_multiply PASSED
test_math.py::TestMathClassDict_0::test_subtract PASSED
==================== 12 passed, 4 warnings in 0.16 seconds =====================

With unittest (and unittest2):

$ python -m unittest -v test_math
test_floor_0_negative (test_math.TestMathUnitTest) ... ok
test_floor_1_integer (test_math.TestMathUnitTest) ... ok
test_floor_2_large_fraction (test_math.TestMathUnitTest) ... ok
test_add (test_math.TestMathClass_0) ... ok
test_multiply (test_math.TestMathClass_0) ... ok
test_add (test_math.TestMathClass_1) ... ok
test_multiply (test_math.TestMathClass_1) ... ok
test_subtract (test_math.TestMathClassDict_0) ... ok

----------------------------------------------------------------------
Ran 8 tests in 0.001s

OK

(note: because unittest does not support test decorators, only tests created with @parameterized.expand will be executed)

With green:

$ green test_math.py -vvv
test_math
  TestMathClass_1
.   test_method_a
.   test_method_b
  TestMathClass_2
.   test_method_a
.   test_method_b
  TestMathClass_3
.   test_method_a
.   test_method_b
  TestMathUnitTest
.   test_floor_0_negative
.   test_floor_1_integer
.   test_floor_2_large_fraction
  TestMathClass_0
.   test_add
.   test_multiply
  TestMathClass_1
.   test_add
.   test_multiply
  TestMathClassDict_0
.   test_subtract

Ran 12 tests in 0.121s

OK (passes=9)

Installation

$ pip install parameterized

Compatibility

Yes (mostly).

  Py2.6 Py2.7 Py3.4 Py3.5 Py3.6 Py3.7 Py3.8 Py3.9 PyPy @mock.patch
nose yes yes yes yes yes yes yes yes yes yes
nose2 yes yes yes yes yes yes yes yes yes yes
py.test 2 yes yes no* no* no* no* yes yes yes yes
py.test 3 yes yes yes yes yes yes yes yes yes yes
py.test 4 no** no** no** no** no** no** no** no** no** no**
py.test fixtures no† no† no† no† no† no† no† no† no† no†
unittest
(@parameterized.expand)
yes yes yes yes yes yes yes yes yes yes
unittest2
(@parameterized.expand)
yes yes yes yes yes yes yes yes yes yes

*: py.test 2 does does not appear to work (#71) under Python 3. Please comment on the related issues if you are affected.

**: py.test 4 is not yet supported (but coming!) in issue #34

†: py.test fixture support is documented in issue #81

Dependencies

(this section left intentionally blank)

Exhaustive Usage Examples

The @parameterized and @parameterized.expand decorators accept a list or iterable of tuples or param(...), or a callable which returns a list or iterable:

from parameterized import parameterized, param

# A list of tuples
@parameterized([
    (2, 3, 5),
    (3, 5, 8),
])
def test_add(a, b, expected):
    assert_equal(a + b, expected)

# A list of params
@parameterized([
    param("10", 10),
    param("10", 16, base=16),
])
def test_int(str_val, expected, base=10):
    assert_equal(int(str_val, base=base), expected)

# An iterable of params
@parameterized(
    param.explicit(*json.loads(line))
    for line in open("testcases.jsons")
)
def test_from_json_file(...):
    ...

# A callable which returns a list of tuples
def load_test_cases():
    return [
        ("test1", ),
        ("test2", ),
    ]
@parameterized(load_test_cases)
def test_from_function(name):
    ...

Note that, when using an iterator or a generator, all the items will be loaded into memory before the start of the test run (we do this explicitly to ensure that generators are exhausted exactly once in multi-process or multi-threaded testing environments).

The @parameterized decorator can be used test class methods, and standalone functions:

from parameterized import parameterized

class AddTest(object):
    @parameterized([
        (2, 3, 5),
    ])
    def test_add(self, a, b, expected):
        assert_equal(a + b, expected)

@parameterized([
    (2, 3, 5),
])
def test_add(a, b, expected):
    assert_equal(a + b, expected)

And @parameterized.expand can be used to generate test methods in situations where test generators cannot be used (for example, when the test class is a subclass of unittest.TestCase):

import unittest
from parameterized import parameterized

class AddTestCase(unittest.TestCase):
    @parameterized.expand([
        ("2 and 3", 2, 3, 5),
        ("3 and 5", 2, 3, 5),
    ])
    def test_add(self, _, a, b, expected):
        assert_equal(a + b, expected)

Will create the test cases:

$ nosetests example.py
test_add_0_2_and_3 (example.AddTestCase) ... ok
test_add_1_3_and_5 (example.AddTestCase) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.001s

OK

Note that @parameterized.expand works by creating new methods on the test class. If the first parameter is a string, that string will be added to the end of the method name. For example, the test case above will generate the methods test_add_0_2_and_3 and test_add_1_3_and_5.

The names of the test cases generated by @parameterized.expand can be customized using the name_func keyword argument. The value should be a function which accepts three arguments: testcase_func, param_num, and params, and it should return the name of the test case. testcase_func will be the function to be tested, param_num will be the index of the test case parameters in the list of parameters, and param (an instance of param) will be the parameters which will be used.

import unittest
from parameterized import parameterized

def custom_name_func(testcase_func, param_num, param):
    return "%s_%s" %(
        testcase_func.__name__,
        parameterized.to_safe_name("_".join(str(x) for x in param.args)),
    )

class AddTestCase(unittest.TestCase):
    @parameterized.expand([
        (2, 3, 5),
        (2, 3, 5),
    ], name_func=custom_name_func)
    def test_add(self, a, b, expected):
        assert_equal(a + b, expected)

Will create the test cases:

$ nosetests example.py
test_add_1_2_3 (example.AddTestCase) ... ok
test_add_2_3_5 (example.AddTestCase) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.001s

OK

The param(...) helper class stores the parameters for one specific test case. It can be used to pass keyword arguments to test cases:

from parameterized import parameterized, param

@parameterized([
    param("10", 10),
    param("10", 16, base=16),
])
def test_int(str_val, expected, base=10):
    assert_equal(int(str_val, base=base), expected)

If test cases have a docstring, the parameters for that test case will be appended to the first line of the docstring. This behavior can be controlled with the doc_func argument:

from parameterized import parameterized

@parameterized([
    (1, 2, 3),
    (4, 5, 9),
])
def test_add(a, b, expected):
    """ Test addition. """
    assert_equal(a + b, expected)

def my_doc_func(func, num, param):
    return "%s: %s with %s" %(num, func.__name__, param)

@parameterized([
    (5, 4, 1),
    (9, 6, 3),
], doc_func=my_doc_func)
def test_subtraction(a, b, expected):
    assert_equal(a - b, expected)
$ nosetests example.py
Test addition. [with a=1, b=2, expected=3] ... ok
Test addition. [with a=4, b=5, expected=9] ... ok
0: test_subtraction with param(*(5, 4, 1)) ... ok
1: test_subtraction with param(*(9, 6, 3)) ... ok

----------------------------------------------------------------------
Ran 4 tests in 0.001s

OK

Finally @parameterized_class parameterizes an entire class, using either a list of attributes, or a list of dicts that will be applied to the class:

from yourapp.models import User
from parameterized import parameterized_class

@parameterized_class([
   { "username": "user_1", "access_level": 1 },
   { "username": "user_2", "access_level": 2, "expected_status_code": 404 },
])
class TestUserAccessLevel(TestCase):
   expected_status_code = 200

   def setUp(self):
      self.client.force_login(User.objects.get(username=self.username)[0])

   def test_url_a(self):
      response = self.client.get('/url')
      self.assertEqual(response.status_code, self.expected_status_code)

   def tearDown(self):
      self.client.logout()


@parameterized_class(("username", "access_level", "expected_status_code"), [
   ("user_1", 1, 200),
   ("user_2", 2, 404)
])
class TestUserAccessLevel(TestCase):
   def setUp(self):
      self.client.force_login(User.objects.get(username=self.username)[0])

   def test_url_a(self):
      response = self.client.get("/url")
      self.assertEqual(response.status_code, self.expected_status_code)

   def tearDown(self):
      self.client.logout()

The @parameterized_class decorator accepts a class_name_func argument, which controls the name of the parameterized classes generated by @parameterized_class:

from parameterized import parameterized, parameterized_class

def get_class_name(cls, num, params_dict):
    # By default the generated class named includes either the "name"
    # parameter (if present), or the first string value. This example shows
    # multiple parameters being included in the generated class name:
    return "%s_%s_%s%s" %(
        cls.__name__,
        num,
        parameterized.to_safe_name(params_dict['a']),
        parameterized.to_safe_name(params_dict['b']),
    )

@parameterized_class([
   { "a": "hello", "b": " world!", "expected": "hello world!" },
   { "a": "say ", "b": " cheese :)", "expected": "say cheese :)" },
], class_name_func=get_class_name)
class TestConcatenation(TestCase):
  def test_concat(self):
      self.assertEqual(self.a + self.b, self.expected)
$ nosetests -v test_math.py
test_concat (test_concat.TestConcatenation_0_hello_world_) ... ok
test_concat (test_concat.TestConcatenation_0_say_cheese__) ... ok

Using with Single Parameters

If a test function only accepts one parameter and the value is not iterable, then it is possible to supply a list of values without wrapping each one in a tuple:

@parameterized([1, 2, 3])
def test_greater_than_zero(value):
   assert value > 0

Note, however, that if the single parameter is iterable (such as a list or tuple), then it must be wrapped in a tuple, list, or the param(...) helper:

@parameterized([
   ([1, 2, 3], ),
   ([3, 3], ),
   ([6], ),
])
def test_sums_to_6(numbers):
   assert sum(numbers) == 6

(note, also, that Python requires single element tuples to be defined with a trailing comma: (foo, ))

Using with @mock.patch

parameterized can be used with mock.patch, but the argument ordering can be confusing. The @mock.patch(...) decorator must come below the @parameterized(...), and the mocked parameters must come last:

@mock.patch("os.getpid")
class TestOS(object):
   @parameterized(...)
   @mock.patch("os.fdopen")
   @mock.patch("os.umask")
   def test_method(self, param1, param2, ..., mock_umask, mock_fdopen, mock_getpid):
      ...

Note: the same holds true when using @parameterized.expand.

Migrating from nose-parameterized to parameterized

To migrate a codebase from nose-parameterized to parameterized:

  1. Update your requirements file, replacing nose-parameterized with parameterized.

  2. Replace all references to nose_parameterized with parameterized:

    $ perl -pi -e 's/nose_parameterized/parameterized/g' your-codebase/
    
  3. You're done!

FAQ

What happened to nose-parameterized?
Originally only nose was supported. But now everything is supported, and it only made sense to change the name!
What do you mean when you say "nose is best supported"?
There are small caveates with py.test and unittest: py.test does not show the parameter values (ex, it will show test_add[0] instead of test_add[1, 2, 3]), and unittest/unittest2 do not support test generators so @parameterized.expand must be used.
Why not use @pytest.mark.parametrize?
Because spelling is difficult. Also, parameterized doesn't require you to repeat argument names, and (using param) it supports optional keyword arguments.
Why do I get an AttributeError: 'function' object has no attribute 'expand' with @parameterized.expand?
You've likely installed the parametrized (note the missing e) package. Use parameterized (with the e) instead and you'll be all set.
Owner
David Wolever
David Wolever
PENBUD is penetration testing buddy which helps you in penetration testing by making various important tools interactive.

penbud - Penetration Tester Buddy PENBUD is penetration testing buddy which helps you in penetration testing by making various important tools interac

Himanshu Shukla 15 Feb 01, 2022
0hh1 solver for the web (selenium) and also for mobile (adb)

0hh1 - Solver Aims to solve the '0hh1 puzzle' for all the sizes (4x4, 6x6, 8x8, 10x10 12x12). for both the web version (using selenium) and on android

Adwaith Rajesh 1 Nov 05, 2021
Load Testing ML Microservices for Robustness and Scalability

The demo is aimed at getting started with load testing a microservice before taking it to production. We use FastAPI microservice (to predict weather) and Locust to load test the service (locally or

Emmanuel Raj 13 Jul 05, 2022
pytest plugin for manipulating test data directories and files

pytest-datadir pytest plugin for manipulating test data directories and files. Usage pytest-datadir will look up for a directory with the name of your

Gabriel Reis 191 Dec 21, 2022
자동 건강상태 자가진단 메크로 서버전용

Auto-Self-Diagnosis-for-server 자동 자가진단 메크로 서버전용 이 프로그램은 SaidBySolo님의 auto-self-diagnosis를 참고하여 제작하였습니다. 개인 사용 목적으로 제작하였기 때문에 추후 업데이트는 진행하지 않습니다. 의존성 G

JJooni 3 Dec 04, 2021
Set your Dynaconf environment to testing when running pytest

pytest-dynaconf Set your Dynaconf environment to testing when running pytest. Installation You can install "pytest-dynaconf" via pip from PyPI: $ pip

David Baumgold 3 Mar 11, 2022
This project is used to send a screenshot by email of your MyUMons schedule using Selenium python lib (headless mode)

MyUMonsSchedule Use MyUMonsSchedule python script to send a screenshot by email (Gmail) of your MyUMons schedule. If you use it on Windows, take care

Pierre-Louis D'Agostino 6 May 12, 2022
One-stop solution for HTTP(S) testing.

HttpRunner HttpRunner is a simple & elegant, yet powerful HTTP(S) testing framework. Enjoy! ✨ 🚀 ✨ Design Philosophy Convention over configuration ROI

HttpRunner 3.5k Jan 04, 2023
Auto-hms-action - Automation of NU Health Management System

🦾 Automation of NU Health Management System 🤖 長崎大学 健康管理システムの自動化 🏯 Usage / 使い方

k5-mot 3 Mar 04, 2022
A Proof of concept of a modern python CLI with click, pydantic, rich and anyio

httpcli This project is a proof of concept of a modern python networking cli which can be simple and easy to maintain using some of the best packages

Kevin Tewouda 17 Nov 15, 2022
This is a simple software for fetching new changes to remote repositories automatically.

Git Autofetch Git Autofetch is a simple software for fetching new changes from a repo to local repositories after a set time interval. This program is

Shreyas Ashtamkar 10 Jul 21, 2022
The Penetration Testers Framework (PTF) is a way for modular support for up-to-date tools.

The PenTesters Framework (PTF) is a Python script designed for Debian/Ubuntu/ArchLinux based distributions to create a similar and familiar distribution for Penetration Testing

trustedsec 4.5k Dec 28, 2022
A pytest plugin to skip `@pytest.mark.slow` tests by default.

pytest-skip-slow A pytest plugin to skip @pytest.mark.slow tests by default. Include the slow tests with --slow. Installation $ pip install pytest-ski

Brian Okken 19 Jan 04, 2023
Ab testing - The using AB test to test of difference of conversion rate

Facebook recently introduced a new type of offer that is an alternative to the current type of bidding called maximum bidding he introduced average bidding.

5 Nov 21, 2022
Object factory for Django

Model Bakery: Smart fixtures for better tests Model Bakery offers you a smart way to create fixtures for testing in Django. With a simple and powerful

Model Bakers 632 Jan 08, 2023
Screenplay pattern base for Python automated UI test suites.

ScreenPy TITLE CARD: "ScreenPy" TITLE DISAPPEARS.

Perry Goy 39 Nov 15, 2022
d4rk Ghost is all in one hacking framework For red team Pentesting

d4rk ghost is all in one Hacking framework For red team Pentesting it contains all modules , information_gathering exploitation + vulnerability scanning + ddos attacks with 12 methods + proxy scraper

d4rk sh4d0w 15 Dec 15, 2022
Network automation lab using nornir, scrapli, and containerlab with Arista EOS

nornir-scrapli-eos-lab Network automation lab using nornir, scrapli, and containerlab with Arista EOS. Objectives Deploy base configs to 4xArista devi

Vireak Ouk 13 Jul 07, 2022
Multi-asset backtesting framework. An intuitive API lets analysts try out their strategies right away

Multi-asset backtesting framework. An intuitive API lets analysts try out their strategies right away. Fast execution of profit-take/loss-cut orders is built-in. Seamless with Pandas.

Epymetheus 39 Jan 06, 2023
A test fixtures replacement for Python

factory_boy factory_boy is a fixtures replacement based on thoughtbot's factory_bot. As a fixtures replacement tool, it aims to replace static, hard t

FactoryBoy project 3k Jan 05, 2023