PyFlake: Flaky Test Detection via Perturbation

Most flaky-test tools just re-run a suite and hope. PyFlake instead perturbs specific sources of nondeterminism (RNG seeds, async timing, execution order, wall-clock drift) and uses a two-stage statistical test to distinguish genuine flakiness from environment noise.

Role: Team project
Timeline: Feb - Apr 2026
Impact: A research-grade tool that finds flaky pytest tests by deliberately perturbing sources of nondeterminism, then proves the flakiness statistically.

Attribution: Built with two teammates for an SFU software-engineering systems course. My scope: the perturbation operators, the statistical evaluation methodology, and the folder-execution harness mode.

PythonpytestSciPyDockerPoetryGitHub Actionsmypy

View source

8 re-runs of the same suite, unperturbed

Concepts demonstrated

Software testing & reliability
Statistical hypothesis testing
CI/CD
Applied research methodology

Highlights

Implemented perturbation operators (random-seed, async-delay, execution-order, time-drift) that monkey-patch nondeterminism sources at the right layer, including eagerly seeding the global RNG at setup time to catch module-level randomness that a naive per-test seed would miss.
Designed a two-gate statistical evaluator: a one-sided Fisher's exact test on operator-vs-baseline pass/fail counts to confirm a perturbation actually changed outcomes, followed by a Wald-Wolfowitz runs test on the outcome sequence to separate genuine non-determinism from clustered environmental noise.
Set up the initial monorepo, Poetry dependency management, pre-commit hooks, mypy type checking, and the lint/test/typecheck GitHub Actions pipeline the team shipped through for the rest of the project.

Problem

Flaky tests, the ones that pass or fail nondeterministically on the same code, quietly erode trust in a test suite. The standard mitigation, 'just re-run it a few times,' doesn't tell you why a test is flaky, and doesn't distinguish real nondeterminism from a bad CI runner having a bad day.

Engineering approach

Rather than blind re-running, target specific, well-known sources of test nondeterminism: RNG seeding, async/await scheduling order, test execution order, and time-based assertions.
Each source gets its own 'operator' that monkey-patches the relevant Python primitive (random.seed, numpy.random.seed, event-loop scheduling, etc.) for the duration of a run, then restores it.
Run the suite repeatedly under each operator and compare the resulting pass/fail distribution against an unperturbed baseline using formal statistical tests, rather than an arbitrary re-run threshold.

Architecture

pyflake/operators/: one module per nondeterminism source, each implementing a common patch/unpatch interface.
pyflake/harness/: drives repeated pytest subprocess runs (including a folder-execution mode), writes a conftest that wires in the active operator, and persists results to SQLite.
pyflake/flapy/: a FlaPy-derived pipeline for running the same methodology against real open-source projects inside Docker, parsing JUnit XML output, and classifying results.
pyflake evaluate: compares detector output against a hand-labeled ground-truth dataset to measure the tool's own precision/recall.

Technical challenges

Seeding correctly is subtler than it looks: patching random.seed() alone misses tests that rely on module-level or setUpClass-time randomness that runs before any test-level seed call. The operator eagerly seeds at setup and periodically reverts to system entropy (roughly every 10th run) specifically to catch tests that assume determinism they never actually asked for.
Distinguishing 'this operator caused the flakiness' from 'the CI runner was just slow this one time' takes more than an eyeball threshold. That's what motivated pulling in Fisher's exact test and a runs test rather than a simple pass-rate cutoff.

Decisions

Chose targeted perturbation over blind re-run repetition, trading some implementation complexity for actual diagnostic signal about *why* a test is flaky.
Used subprocess-isolated pytest runs (not in-process) so a corrupted interpreter state from one run can't contaminate the next, which matters when you're deliberately breaking determinism.
Kept the CI pipeline (lint, type-check, test) mandatory on every PR from week one, which is part of why the codebase stayed navigable across two teammates and nine weeks.

Result

A working detector with an honestly-scoped README about its own limitations, a real CI pipeline the team used throughout, and a statistically grounded evaluation methodology rather than a hand-wavy heuristic.

Learnings

Targeting specific nondeterminism sources instead of blind re-running is what gave the tool diagnostic signal, not just a flakiness label. That distinction was worth the extra implementation complexity.
The evaluation dataset is small and hand-labeled. The honest next step is running the FlaPy pipeline against a wider set of real open-source repos to see whether the statistical thresholds hold outside a course-scoped test suite.

Next projectAI Security Operations Platform