Test Suite

CPMpy has an extensive test suite, covering all major components including variables, constraints, models, solvers, transformations, and tools.

Running Tests

Basic Usage

Run all tests:

pytest

Run a specific test file:

pytest tests/test_model.py

Run a specific test:

pytest tests/test_model.py::TestModel::test_ndarray

Parallelisation

Through the pytest-xdist pytest plugin, running tests can be parallelised. E.g. running with 40 workers:

pytest -n 40 tests/test_model.py

Or letting pytest decide how many workers to use:

pytest -n auto tests/test_model.py

Install using:

pip install pytest-xdist

Solver Selection

The test suite supports changing the solver backend used to run the tests via the --solver command-line option:

Single Solver

Run tests with a specific solver:

pytest --solver=gurobi

Multiple Solvers

Run tests with multiple solvers

non-solver-specific tests will run against all specified solvers
solver-specific tests will be filtered on specified solvers

pytest --solver=ortools,cplex,gurobi

All Installed Solvers

Run tests with all installed solvers:

pytest --solver=all

This automatically detects all installed solvers from SolverLookup and parametrises non-solver-specific tests to run against each one.

Skip Solver Tests

Skip all solver-parametrised tests (only run tests that don’t depend on solver parametrisation). I.e., tests that do not rely on solving a model. Examples are tests that evaluate constructors of expressions.

pytest --solver=None

Default Behavior

If no --solver option is provided:

Non-solver-specific tests run with the default solver (OR-Tools)
All solver-specific tests run for their respective declared solver (if installed)

Test Organization

Test Files

test_model.py - Model creation, manipulation, and I/O
test_expressions.py - Expression types and operations (comparisons, operators, sums, etc.)
test_constraints.py - Constraint types and validation (boolean, comparison, reification, implication)
test_globalconstraints.py - Global constraint implementations (AllDifferent, Circuit, Cumulative, etc.)
test_solvers.py - Solver interface and functionality (high-level solver tests)
test_solverinterface.py - Low-level solver interface tests (constructor, native model, solve methods)
test_variables.py - Variable types (intvar, boolvar, shapes, naming)
test_builtins.py - Python builtin functions (max, min, all, any)
test_cse.py - Common subexpression elimination
test_direct.py - Direct solver constraints (automaton, etc.)
test_flatten.py - Model flattening transformations
test_int2bool.py - Integer to boolean transformation
test_pysat_*.py - PySAT-specific tests (cardinality, interrupt, weighted sum)
test_solveAll.py - solveAll functionality across solvers
test_solvers_solhint.py - Solver hints functionality
test_tocnf.py - Conversion to CNF (Conjunctive Normal Form)
test_tool_dimacs.py - DIMACS format tools
test_trans_*.py - Transformation tests (linearize, safen, simplify)
test_transf_*.py - Additional transformation tests (comp, decompose, reif)
test_tools_*.py - Tool functionality (MUS, tuning, etc.)
test_examples.py - Run examples as a testsuite

Test Markers

Tests can be marked with special markers:

@pytest.mark.requires_solver("solver_name_1", "solver_name_2", ...) - Test requires a specific solver, one of the listed names
@pytest.mark.requires_dependency("package_name") - Test requires a specific Python package
@pytest.mark.generate_constraints.with_args(generator_function) - Parametrise test’s “constraint” argument using the provided generator

Examples:

@pytest.mark.requires_solver("cplex")
def test_cplex_specific_feature():
    # This test only runs if cplex is available
    pass

def randomly_sample_expressions(solver)
    return [...]

@pytest.mark.generate_constraints.with_args(randomly_sample_expressions)
def test_bool_constraints(solver, constraint):
    ...

(for a complete example, have a look at /tests/test_constraints.py)

Writing Tests

Basic Test Structure

import pytest
import cpmpy as cp

def test_basic_model():
    x = cp.intvar(0, 10, name="x")
    m = cp.Model(x >= 5)
    assert m.solve()
    assert x.value() >= 5

Using the Custom TestCase Class

CPMpy’s test suite includes a custom TestCase class (in tests/utils.py) that provides all unittest-style assertion methods without inheriting from unittest.TestCase. This design allows pytest’s pytest_generate_tests parametrization to work properly.

Benefits

Access to all familiar unittest assertions: assertEqual, assertTrue, assertIn, assertIsInstance, etc.
Compatible with pytest’s parametrization and fixture system
Automatic initialization of required unittest internal attributes
Support for setup_method and teardown_method hooks

Usage

from utils import TestCase
import cpmpy as cp

class TestMyFeature(TestCase):
    def setup_method(self):
        # Called before each test method
        self.x = cp.intvar(0, 10, name="x")

    def test_example(self):
        # Use unittest-style assertions
        self.assertEqual(str(self.x), "x")
        self.assertIsInstance(self.x, cp.intvar)
        self.assertTrue(self.x.lb == 0)
        self.assertIn(self.x, [self.x])

Using the Solver Fixture

For tests that should run with different solvers:

from utils import TestCase

@pytest.mark.usefixtures("solver")
class TestMyFeature(TestCase):
    def test_with_solver(self):
        x = cp.intvar(0, 10)
        m = cp.Model(x >= 5)
        self.assertTrue(m.solve(solver=self.solver))
        self.assertGreaterEqual(x.value(), 5)

When multiple solvers are provided via --solver, these tests will automatically be parametrised to run against each solver. The self.solver attribute is automatically set by the test framework.

Solver-Parametrised Tests

For tests that are explicitly parametrised with a selection of solvers:

@pytest.mark.parametrise("solver", ["ortools", "cplex", "gurobi"])
def test_with_explicit_solvers(solver):
    x = cp.intvar(0, 10)
    m = cp.Model(x >= 5)
    assert m.solve(solver=solver)

Solver-Specific Tests

For tests that only work with specific solvers:

@pytest.mark.requires_solver("cplex")
def test_cplex_feature():
    # Test cplex-specific functionality
    pass

Contributing

When adding new tests:

Follow existing test patterns
Use appropriate markers for solver-specific tests
Ensure tests work with multiple solvers when possible
Add docstrings explaining what the test validates
Use descriptive test names