CI test management: what a pull request runs #369

What a pull request runs, and how CI keeps the suite trustworthy: run less on pull requests where history says it loses nothing, keep the full matrix before anything reaches main, and keep flaky tests out of the gate without losing what they check.

Plan: CI test management plan (the wiki), with the measurements. This issue started as step 5 alone; its 2026-11-05 measurement of test selection is on that page, unchanged.

Today (last 20 runs, 2026-11-05 to 11-08)

• 31 jobs, about 250 runner-minutes a run, on every pull request, a one-line docs change included. About half is the suite on three Python versions.
• Wall time 12–21 minutes when nothing queues; with several pull requests open, jobs queued 27–33 minutes for a runner.
• Only lint and docs are required checks: a red test job does not block a merge.

Steps

• 1. Path filter and one required check. Done in CI: run the jobs a pull request needs, timing tests alone, one required check #390 (merged 2026-11-09); CI is the required check. Land CI runs the jobs a pull request's files need #368 (docs, examples and tools alone skip the test jobs; 87% of today's test time over 278 merged PRs), and add an aggregate CI job, required in place of lint and docs, that fails if any job failed and passes skipped ones.
• 2. One Python version in full, the other two quick. Done in CI: a pull request runs the suite on 3.13, the quick tier on 3.11 and 3.12 #391. PRs run the full suite on 3.13 (with VTK and MFEM, as today) and -m quick on 3.11 and 3.12 (without them); pushes to main keep the full matrix. About 70 runner-minutes a run less. Of the 18 failing PR runs of 2026-10-14 to 11-05, the five that missed 3.13 differ by VTK more than by version, and their files are in the quick tier (with test_cpu_pool.py added) but for some of test_viskores_datasets.py's tests.
• 3. Timing tests in a job of their own. Done in CI: run the jobs a pull request needs, timing tests alone, one required check #390. The shares run -m "not timing"; the timing tests run alone, a failure re-run once and reported as a flake if it then passes (CPU threading: a contended kernel stays serial unless a recheck wins by the clock #385's test failed a docs-only PR).
• 4. Flaky tests tracked. Done in tools/ci_failures.py: the tests that failed in CI lately, flaky ones flagged #392; its first flake fixed in Compiler preflight: a probe that times out is tried once more, for longer #394. tools/ci_failures.py: failures on main, failures that passed on a re-run of the same commit, failures a PR's files cannot reach; a weekly report.
• 5. Test selection, by file first. Done in CI records which repository files each test file runs: the test map #395 (recording the map) and CI: a pull request runs the test files its changes reach #397 (selecting from it). A map recorded on main of the repository files each test file uses, refreshed on every push to main; PRs run the test files their changes reach (68% of today's test time by file, 63% by test; none of 13 historic failures missed), the shares re-dealt over the selection. Prototype: branch ci/test-selection-prototype of berkgeveci/tack-fork.
• 6. A merge queue. Done in CI: the merge queue runs every test before main moves #398; the main ruleset has the merge-queue rule (merge commits, one PR a group, CI required). PRs run the reduced set; the queue runs the full matrix before the merge reaches main, replacing today's full run after it.

Open questions (on the wiki page): Metal on every code change or by selection; whether l2-cells can drop to the quick tier on PRs.