Skip to content

datadriven: v2 with a testing.TB wrapper and command-line syntax - #66

Open
RaduBerinde wants to merge 40 commits into
masterfrom
claude/datadriven-v2-api-d25b83
Open

RaduBerinde wants to merge 40 commits into
masterfrom
claude/datadriven-v2-api-d25b83

Conversation

@RaduBerinde

@RaduBerinde RaduBerinde commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

This PR (code and description) was generated with Claude Code.

This turns master into v2 of the library: github.com/cockroachdb/datadriven/v2, requiring Go 1.27 (for generic methods). The v1 branch is at the current master. v1 (v1.0.2 or later) and v2 can be linked in the same test binary, so a large repository can migrate one package at a time; see MIGRATION.md.

What changes

The callback gets a testing.TB wrapper. RunTest is generic over the testing.TB and passes the function a *datadriven.T (or *datadriven.B), which annotates errors, skips and logs with the position of the directive in the test file. The wrapper is also stored in TestData.T, so TestData methods no longer take a t.

Directive lines use command-line syntax. Words that start with - or -- followed by a letter are flags (-x, -x=v, -x=(v1, v2)); other words, and everything after --, are positional arguments.

  • An argument or a flag value can be written between single quotes (-msg='hello world'), in which case it is taken literally ('' stands for a quote). Quotes are only special at the start of an argument or of a flag value, so quotes inside parentheses (e.g. SQL) keep their meaning.
  • A flag can be repeated if the test retrieves all the occurrences with FlagValues; the single-value accessors fail on a repeated flag.

The test must use what is on the line.

  • Every flag must be processed by the test, which catches misspelled flags. This applies even if the test returns early for a directive: a flag that is not used should not be in the test file.
  • An argument of the form name or name=value fails the test if the test looked up flag name and it is not on the line: it is most likely a flag which is missing the dash (quoted arguments and those after -- are exempt).
  • Neither stops the test: the output is still compared (or rewritten with -rewrite), the rest of the file is processed, and the problems are reported when the file ends, after any other failure.
  • A test which interprets the line itself (e.g. as the command line of a tool) uses DirectiveLine, which returns the line as written and disables these checks.

Flags and arguments have separate APIs.

  • Flags: HasFlag, BoolFlag, FlagValue[V], FlagValueOr, FlagValues[V], plus MarkFlagProcessed / MarkAllFlagsProcessed for code that iterates over TestData.Flags.
  • Arguments: TestData.Args, SingleArg, TwoArgs, PopArg, PopArgs, ScanArgs.
  • Both parse values with the same code: integers in base 10, time.Duration with time.ParseDuration, named types, slices for list values, and any type that implements encoding.TextUnmarshaler. Unsupported Go types fail on every call, not only when a value is parsed.

Inputs that are sequences of commands. TestData.Pos is a struct, InputLines iterates over the input with per-line positions, and InputDirectives yields each input line as a TestData with the same checks, reporting errors at that line.

Fixes to long-standing problems.

  • A line longer than 64KiB (or a read error) silently ended the file, and -rewrite then truncated it.
  • -rewrite corrupted the file for outputs that start with a ---- line; outputs that can't be represented now fail instead.
  • An unterminated double ---- section swallowed the rest of the file, and -rewrite deleted it; it is now an error.
  • -run could not select a subtest of a test file.
  • RunTestFromString passed vacuously under -rewrite; it now always compares.
  • ClearResults is fixed, and subtest directives in benchmarks fail with a clear error.

Docs. Package documentation in doc.go (file format and API), a shorter README, and MIGRATION.md with mechanical rewrites and the list of behavior changes.

Notes for reviewers

  • The 40 commits are meant to be read one at a time; each has a message explaining why. The last seven (from support quoted arguments and flag values on) came out of evaluating Pebble's tests against the new API.
  • .github: run CI on the v1 branch is the first commit here, but v1 CI only starts once that change is also on v1.
  • CI now tests with Go 1.27 only.
  • Behavior changes that could surprise existing tests are listed in MIGRATION.md. The ones most likely to matter: RunTestFromString comparing under -rewrite, the unterminated double separator error, -rewrite rejecting outputs with lines that end in a carriage return, and an argument or flag value that starts with a single quote losing its quotes.
  • Effect of quoting on existing test files, measured by running the parser with and without it over their directive lines: no line changes in Pebble; in CockroachDB (pkg/, excluding logic tests), three lines lose the quotes around a value (uri='nodelocal://...', oauth_token='...').
  • At the tip, go vet, gofmt, and the tests with -count=2, -race and -rewrite (in a copy of the tree) pass on Go 1.27.2; each of the last seven commits also passes on its own.

🤖 Generated with Claude Code


This change is Reviewable

RaduBerinde and others added 30 commits October 12, 2026 09:26
Problem: CI only runs for pushes and pull requests against `master`, so
changes to the `v1` branch are not tested.

Changes:
- Also run the workflow for pushes and pull requests against `v1`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Start v2 of the module in preparation for breaking API changes; v1
continues on the `v1` branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: v2 doesn't need to support Go versions older than 1.24.

Changes:
- Bump the `go` directive to 1.24.
- CI tests Go 1.24 through 1.27 (the `v1` branch keeps testing 1.21
  through 1.27).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… position

Problem: errors reported via the `testing.T` passed to the callback don't
point to the directive in the test file. Also, `RunTest` and friends are
duplicated for `*testing.T` and `testing.TB`.

Changes:
- Add the generic `TBWrapper[U testing.TB]` (with aliases `T` and `B`
  for `*testing.T` and `*testing.B`), which overrides `Error(f)`,
  `Fatal(f)`, `Skip(f)`, `Log(f)` to prefix messages with the position of
  the directive currently running; the underlying value is available as
  the `TB` field. `Wrap` creates a wrapper outside of `RunTest`.
- `RunTest`, `RunTestFromString` and `Walk` are generic over the
  `testing.TB` type; the `RunTest*` callbacks receive a `*TBWrapper[U]`
  (the `Walk` callback still receives a `U`). Remove `RunTestAny`,
  `RunTestFromStringAny` and `WalkAny`.
- Store the wrapper in the new `TestData.T` field; each directive gets
  its own `TestData`.
- `TestData` methods (and argument scanning) no longer add the position
  themselves; the wrapper passed to them does.
- `subtest` directives are not supported over a `*testing.B` (fail with
  a clear error): `testing.B.Run` runs the function multiple times, but
  the directives can only be read once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: `TestData` methods require passing a `testing.TB`, even though
`TestData` now holds the (annotating) wrapper.

Changes:
- `ScanArgs`, `MaybeScanArgs`, `Retry`, `RetryFor`, `Logf`, `Fatalf` use
  `TestData.T` instead of taking a `testing.TB`.
- `TestData.T` must be set for manually constructed `TestData` values
  that use these methods; they panic with an explanation otherwise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: `ClearResults` (used by `datadriven-clear`) never wrote the
cleared file (writing it moved into `RunTest` in 2020), and it crashed
on files with `subtest` directives or parse errors because it ran over
a zero `testing.T`.

Changes:
- Run over an internal `testing.TB` which records errors and supports
  `FailNow` and subtests outside of a test binary.
- Write the cleared file (sharing the code with `RunTest`); return the
  first error (with its position) and leave the file unchanged if it
  can't be parsed.
- `datadriven-clear`: terminate the error and usage messages with a
  newline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: framework messages (output mismatch, verbose echo) render the
directive from `CmdArgs`, so they are wrong if the test modified them.

Changes:
- Store the original directive line in a private `TestData` field.
- Use it when printing the directive in framework messages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Many tests treat strings on the directive line as positional arguments,
which `CmdArgs` represents as keys; handling them is awkward and they
can be mistaken for keys by `ScanArgs` etc. PopArg removes the first
argument and returns it; its errors include the original directive
line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: tests assign `TestData.Pos` to point messages at a line within
the input; this silently has no effect on the wrapper's annotations in
v2, and the framework's messages used the modified value. Other tests
parse the `Pos` string to get the file or line.

Changes:
- `Pos` is a struct (`File`, `Line`) with a `String` method; assigning
  `TestData.Pos` has no effect on messages.
- Add `TestData.InputLines`, which iterates over the input lines with
  their positions; messages reported through `T` during the loop body
  are annotated with the line's position.
- Framework messages and the output comparison use the directive as
  read from the file, not as modified by the callback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: many tests report errors with the directive position already
in the message (`t.Fatalf("%s: ...", d.Pos, ...)`); with the wrapper,
the position is printed twice.

Changes:
- The wrapper doesn't add the position if the message already starts
  with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: v1 and v2 both define the `-rewrite` and `-datadriven-quiet`
flags, so a test binary that links both panics; this prevents migrating
a large repository to v2 incrementally. Also, there is no way to check
for `-rewrite` outside of a directive other than looking up the flag by
name.

Changes:
- Reuse a flag if it is already defined; this works with v1.0.2 or
  later, which is initialized before v2 (older versions depend on
  `cockroachdb/errors` and can be initialized after v2).
- `DATADRIVEN_QUIET_LOG` sets the flag through the flag set.
- Add `Rewrite()`, which returns the value of `-rewrite`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: reading an argument requires declaring a variable and calling
`ScanArgs`; projects wrote their own generic wrappers (e.g. cockroach's
`pkg/testutils/dd`). Arguments without values can't be scanned into a
bool, and scan errors are misleading ("unsupported type").

Changes:
- Add the generic `ScanArg`, `ScanArgOr` and `MaybeScanArg` functions.
- An argument without values scans into a bool as true.
- Clearer scan errors: missing value, too many values, and the index of
  the value that failed to parse.
- Document the supported destination types.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: some methods are redundant or unused, `TestData.Logf` and
`Fatalf` have value receivers unlike the other methods, and the code
uses `interface{}` and the deprecated `ioutil`.

Changes:
- Remove `CmdArg.ExpectNumVals` and `ExpectNumValsGE` (unused outside
  of `FirstVal`, `SingleVal`, `TwoVals`).
- Remove `TestData.Logf` (equivalent to `td.T.Logf`).
- `TestData.Fatalf` has a pointer receiver.
- Use `any` and `os.ReadDir`; gofmt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: some framework messages are wrong or unhelpful:
- the subtest end mismatch error swaps "expected" and "got";
- outputs that differ only in trailing whitespace look identical in
  the mismatch message;
- `t.Skip` logs a spurious "fatal during" message;
- the double-separator error has no position;
- `CmdArg.Scan` errors don't mention the argument.

Changes:
- Fix the above; add a note to the mismatch message when the outputs
  differ only in trailing whitespace.
- Use the same position header in both forms of the mismatch message.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: the package has no overview documentation; the file format is
described (partly incorrectly) in the `RunTest` doc, and some comments
are stale. There is no guidance for migrating from v1.

Changes:
- Add a package doc with the file format, an example, subtests and
  flags; focus the `RunTest` doc on its behavior.
- Fix stale or incorrect comments.
- Add `MIGRATION.md`, with (tested) rewrite recipes for the mechanical
  changes and a list of the other changes.
- Update the README with a v2 example.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every word on a directive line used to be a key with optional values,
so tests couldn't distinguish options from positional words, and a
misspelled key was silently ignored. Directive lines now follow
command-line conventions: words that start with a dash and a letter are
flags, and other words (and everything after a `--`) are positional
arguments, with separate APIs for each.

All flags must be processed by the test, otherwise the directive fails;
flags looked up by name are processed automatically, and others are
marked explicitly. Existing test files must be converted (see
MIGRATION.md).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaybeScanFlagInto is renamed to MaybeScanFlagVals accordingly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Repeated flags were allowed and needed their own set of accessors, and
values could be scanned in several overlapping ways, which was
confusing. A flag now appears at most once, its value is either a single
value or a list, and it is retrieved with FlagValue or FlagValueOr (a
list as a slice); BoolFlag checks for flags without values. The Flag
type is no longer exported, so ParseLine returns a TestData, which lets
tests use the same API on the lines they parse.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hiding the flag type forced ParseLine to return a TestData and required
a separate way to iterate over flags. Flags are exported again for
direct iteration, ParseLine returns them alongside the command and
arguments, and the processing check keeps using the flags as parsed
from the directive line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Go 1.27 adds generic methods, which the flag API will use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
They were package-level functions only because Go methods couldn't have
type parameters before Go 1.27.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The unprocessed-flag check read its state back from the TestData passed
to the test function: a function that overwrote the TestData caused a
panic or a false error, and one that modified Flags in place (e.g. with
slices.DeleteFunc) caused bogus "unprocessed flags" failures. The check
now keeps its own reference to the processed set and its own copy of
the flags. Also, looking up flags of a TestData constructed manually
lazily allocated a set nobody checks, racing with concurrent lookups;
marking is now a no-op for TestData values that aren't checked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Named scalar and slice types (e.g. `type X int`, `type Elems []X`) are
supported as the type parameter; the docs now say so explicitly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
These parse positional arguments without a format, one argument per
value, with the same code as flag values: integers are in base 10 (a
leading 0 does not indicate octal, which matters e.g. for Pebble file
numbers) and a time.Duration is parsed with time.ParseDuration; other
types are scanned like fmt.Sscan. ScanArgs detects calls of the v1 form
(with a testing.TB and a key), which still compile, and points to
FlagValue.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test file was read with a bufio.Scanner, which stops at a line
longer than 64KiB; this (like any read error) was treated as the end of
the file, so the rest of the file was silently ignored and, with
-rewrite, removed from the file. The file is now read upfront, with no
limit on the length of a line, and a read error fails the test.

Co-Authored-By: Claude <noreply@anthropic.com>
With -rewrite, an output that starts with a ---- line was written after
a single separator, where it reads back as a double separator: the
output and the directives after it were then misparsed. Such an output
is now written between double separators. An output that can't be
written in either form (e.g. one with a blank line and two consecutive
---- lines) now fails the test instead of corrupting the file.

Co-Authored-By: Claude <noreply@anthropic.com>
testing.T.Run doesn't call the function of a subtest that is not
selected by the -run flag; the test then failed with a bogus "EOF
encountered without subtest end directive" error, so -run could not be
used to select a subtest of a test file. The directives of such a
subtest are now skipped (and preserved by -rewrite).

Co-Authored-By: Claude <noreply@anthropic.com>
With -rewrite, RunTestFromString ran in rewrite mode: the outputs were
not compared with the expected outputs, and there is no file to
rewrite, so its tests passed no matter what. It now always compares,
and TestData.Rewrite reflects whether the file is being rewritten
(rather than the flag), so that e.g. Retry keeps retrying.

Co-Authored-By: Claude <noreply@anthropic.com>
A test can now retrieve flag values and positional arguments as its own
types (e.g. keys, timestamps) by implementing encoding.TextUnmarshaler,
with errors that carry the position in the test file; standard types
like netip.Addr, time.Time and *big.Int work too. This is the single
extension point for flags and arguments: PopArgs and ScanArgs no longer
fall back to fmt.Sscan (and fmt.Scanner) for the types that the flag
parser didn't support.

Co-Authored-By: Claude <noreply@anthropic.com>
FlagValue and FlagValueOr only failed for an unsupported type when they
parsed a value, so a call with such a type passed as long as the flag
was missing (or an empty list) and only failed once a test file used
the flag. The type is now checked on every call; the same goes for the
values passed to PopArgs and ScanArgs.

Co-Authored-By: Claude <noreply@anthropic.com>
RaduBerinde and others added 10 commits November 9, 2026 17:56
Tests whose input is a sequence of commands parse each line with
ParseLine and build a TestData by hand to read its flags, which are
then never checked for being processed: a misspelled flag on such a
line goes unnoticed, unlike on a directive line. InputDirectives
returns each line as a TestData, with the same check, and reports
errors at the position of the line.

Co-Authored-By: Claude <noreply@anthropic.com>
An expected output that starts with a double ---- separator but is not
closed by one extended to the end of the file: the directives after it
were taken as expected output, and -rewrite removed them from the file.
This is now an error.

Co-Authored-By: Claude <noreply@anthropic.com>
The macos-14 runner image was retired, so the macOS job failed
immediately with "The configuration 'macos-14-arm64-us-default' is not
supported". Using macos-latest, like the Linux jobs use ubuntu-latest,
keeps the job from breaking again when an image is retired.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A single value could not contain whitespace: parentheses were the only
way to group words, and a parenthesized value is a list (which can only
be retrieved as a slice). An argument or the value of a flag can now be
written between single quotes, in which case it is taken literally; this
also allows unbalanced parentheses, for which there was no notation.

Quotes are only special at the start of an argument or of a flag value,
so that quotes inside parentheses (e.g. SQL) and apostrophes keep their
meaning. FullCmd and Flag.String quote the values that need it, so what
they render is parsed back unchanged.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Repeated flags were rejected when parsing, because a lookup by name has
to pick one of the occurrences (v1 silently used the first). But tests
do use them for a sequence of values that are lists themselves, e.g.
-probe=(a, 1) -probe=(b, 2), for which there was no other notation.

A flag can now be repeated as long as the test retrieves all the
occurrences, with FlagValues; the methods that retrieve a single value
fail on a repeated flag, so an occurrence is never silently ignored.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Most directives with positional arguments take exactly one or two, and
reading Args[0] directly ignores any extra arguments (or panics if there
are none). These return the arguments and fail the directive if their
number is not the expected one.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Some tests pass the directive line to another command-line parser (e.g.
Pebble's tool tests run cobra commands). They could only rebuild it
from Flags and Args, which loses the difference between -x and --x and
the original quoting. DirectiveLine returns the line as it is written.

Such a test doesn't retrieve the flags through TestData, so calling
DirectiveLine also disables the check that all the flags are processed.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
An unprocessed flag failed the directive before its output was compared
or rewritten. When a change makes a directive return early (e.g. with a
new error), the flags it no longer gets to read were all that was
reported: the output which explains them was hidden, and -rewrite could
not record it.

Unprocessed flags still fail the test (a flag that the test doesn't use
should not be in the test file), but they no longer stop it: the output
is compared or rewritten, the rest of the file is processed, and the
flags are reported at the end, after any other failure. The same goes
for the flags on the lines returned by InputDirectives.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A flag written without the dash (sync instead of -sync, or an argument
left as lower=a when converting a v1 test file) is a positional
argument: the test looks up the flag, doesn't find it, and silently does
something other than what the test file says. Unlike a misspelled flag,
nothing reported it.

An argument of the form name or name=value now fails the test (like an
unprocessed flag, without stopping it) if the test looked up a flag
with that name and the flag is not on the line. Arguments that are
quoted or after a -- word are exempt, for the rare argument that has
the name of a flag.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A logger which holds the wrapper and logs from background goroutines
gets its messages attributed to whatever directive is running, which is
misleading; it should use the underlying testing.TB. The migration guide
also didn't cover v1's ScanArgs with several destinations.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant