Repository navigation
datadriven: v2 with a testing.TB wrapper and command-line syntax - #66
Open
RaduBerinde wants to merge 40 commits into
Open
RaduBerinde wants to merge 40 commits into
RaduBerinde wants to merge 40 commits into
Conversation
Problem: CI only runs for pushes and pull requests against `master`, so changes to the `v1` branch are not tested. Changes: - Also run the workflow for pushes and pull requests against `v1`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Start v2 of the module in preparation for breaking API changes; v1 continues on the `v1` branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: v2 doesn't need to support Go versions older than 1.24. Changes: - Bump the `go` directive to 1.24. - CI tests Go 1.24 through 1.27 (the `v1` branch keeps testing 1.21 through 1.27). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… position Problem: errors reported via the `testing.T` passed to the callback don't point to the directive in the test file. Also, `RunTest` and friends are duplicated for `*testing.T` and `testing.TB`. Changes: - Add the generic `TBWrapper[U testing.TB]` (with aliases `T` and `B` for `*testing.T` and `*testing.B`), which overrides `Error(f)`, `Fatal(f)`, `Skip(f)`, `Log(f)` to prefix messages with the position of the directive currently running; the underlying value is available as the `TB` field. `Wrap` creates a wrapper outside of `RunTest`. - `RunTest`, `RunTestFromString` and `Walk` are generic over the `testing.TB` type; the `RunTest*` callbacks receive a `*TBWrapper[U]` (the `Walk` callback still receives a `U`). Remove `RunTestAny`, `RunTestFromStringAny` and `WalkAny`. - Store the wrapper in the new `TestData.T` field; each directive gets its own `TestData`. - `TestData` methods (and argument scanning) no longer add the position themselves; the wrapper passed to them does. - `subtest` directives are not supported over a `*testing.B` (fail with a clear error): `testing.B.Run` runs the function multiple times, but the directives can only be read once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: `TestData` methods require passing a `testing.TB`, even though `TestData` now holds the (annotating) wrapper. Changes: - `ScanArgs`, `MaybeScanArgs`, `Retry`, `RetryFor`, `Logf`, `Fatalf` use `TestData.T` instead of taking a `testing.TB`. - `TestData.T` must be set for manually constructed `TestData` values that use these methods; they panic with an explanation otherwise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: `ClearResults` (used by `datadriven-clear`) never wrote the cleared file (writing it moved into `RunTest` in 2020), and it crashed on files with `subtest` directives or parse errors because it ran over a zero `testing.T`. Changes: - Run over an internal `testing.TB` which records errors and supports `FailNow` and subtests outside of a test binary. - Write the cleared file (sharing the code with `RunTest`); return the first error (with its position) and leave the file unchanged if it can't be parsed. - `datadriven-clear`: terminate the error and usage messages with a newline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: framework messages (output mismatch, verbose echo) render the directive from `CmdArgs`, so they are wrong if the test modified them. Changes: - Store the original directive line in a private `TestData` field. - Use it when printing the directive in framework messages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Many tests treat strings on the directive line as positional arguments, which `CmdArgs` represents as keys; handling them is awkward and they can be mistaken for keys by `ScanArgs` etc. PopArg removes the first argument and returns it; its errors include the original directive line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: tests assign `TestData.Pos` to point messages at a line within the input; this silently has no effect on the wrapper's annotations in v2, and the framework's messages used the modified value. Other tests parse the `Pos` string to get the file or line. Changes: - `Pos` is a struct (`File`, `Line`) with a `String` method; assigning `TestData.Pos` has no effect on messages. - Add `TestData.InputLines`, which iterates over the input lines with their positions; messages reported through `T` during the loop body are annotated with the line's position. - Framework messages and the output comparison use the directive as read from the file, not as modified by the callback. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: many tests report errors with the directive position already
in the message (`t.Fatalf("%s: ...", d.Pos, ...)`); with the wrapper,
the position is printed twice.
Changes:
- The wrapper doesn't add the position if the message already starts
with it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: v1 and v2 both define the `-rewrite` and `-datadriven-quiet` flags, so a test binary that links both panics; this prevents migrating a large repository to v2 incrementally. Also, there is no way to check for `-rewrite` outside of a directive other than looking up the flag by name. Changes: - Reuse a flag if it is already defined; this works with v1.0.2 or later, which is initialized before v2 (older versions depend on `cockroachdb/errors` and can be initialized after v2). - `DATADRIVEN_QUIET_LOG` sets the flag through the flag set. - Add `Rewrite()`, which returns the value of `-rewrite`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: reading an argument requires declaring a variable and calling
`ScanArgs`; projects wrote their own generic wrappers (e.g. cockroach's
`pkg/testutils/dd`). Arguments without values can't be scanned into a
bool, and scan errors are misleading ("unsupported type").
Changes:
- Add the generic `ScanArg`, `ScanArgOr` and `MaybeScanArg` functions.
- An argument without values scans into a bool as true.
- Clearer scan errors: missing value, too many values, and the index of
the value that failed to parse.
- Document the supported destination types.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: some methods are redundant or unused, `TestData.Logf` and
`Fatalf` have value receivers unlike the other methods, and the code
uses `interface{}` and the deprecated `ioutil`.
Changes:
- Remove `CmdArg.ExpectNumVals` and `ExpectNumValsGE` (unused outside
of `FirstVal`, `SingleVal`, `TwoVals`).
- Remove `TestData.Logf` (equivalent to `td.T.Logf`).
- `TestData.Fatalf` has a pointer receiver.
- Use `any` and `os.ReadDir`; gofmt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: some framework messages are wrong or unhelpful: - the subtest end mismatch error swaps "expected" and "got"; - outputs that differ only in trailing whitespace look identical in the mismatch message; - `t.Skip` logs a spurious "fatal during" message; - the double-separator error has no position; - `CmdArg.Scan` errors don't mention the argument. Changes: - Fix the above; add a note to the mismatch message when the outputs differ only in trailing whitespace. - Use the same position header in both forms of the mismatch message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem: the package has no overview documentation; the file format is described (partly incorrectly) in the `RunTest` doc, and some comments are stale. There is no guidance for migrating from v1. Changes: - Add a package doc with the file format, an example, subtests and flags; focus the `RunTest` doc on its behavior. - Fix stale or incorrect comments. - Add `MIGRATION.md`, with (tested) rewrite recipes for the mechanical changes and a list of the other changes. - Update the README with a v2 example. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every word on a directive line used to be a key with optional values, so tests couldn't distinguish options from positional words, and a misspelled key was silently ignored. Directive lines now follow command-line conventions: words that start with a dash and a letter are flags, and other words (and everything after a `--`) are positional arguments, with separate APIs for each. All flags must be processed by the test, otherwise the directive fails; flags looked up by name are processed automatically, and others are marked explicitly. Existing test files must be converted (see MIGRATION.md). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaybeScanFlagInto is renamed to MaybeScanFlagVals accordingly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Repeated flags were allowed and needed their own set of accessors, and values could be scanned in several overlapping ways, which was confusing. A flag now appears at most once, its value is either a single value or a list, and it is retrieved with FlagValue or FlagValueOr (a list as a slice); BoolFlag checks for flags without values. The Flag type is no longer exported, so ParseLine returns a TestData, which lets tests use the same API on the lines they parse. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hiding the flag type forced ParseLine to return a TestData and required a separate way to iterate over flags. Flags are exported again for direct iteration, ParseLine returns them alongside the command and arguments, and the processing check keeps using the flags as parsed from the directive line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Go 1.27 adds generic methods, which the flag API will use. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
They were package-level functions only because Go methods couldn't have type parameters before Go 1.27. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The unprocessed-flag check read its state back from the TestData passed to the test function: a function that overwrote the TestData caused a panic or a false error, and one that modified Flags in place (e.g. with slices.DeleteFunc) caused bogus "unprocessed flags" failures. The check now keeps its own reference to the processed set and its own copy of the flags. Also, looking up flags of a TestData constructed manually lazily allocated a set nobody checks, racing with concurrent lookups; marking is now a no-op for TestData values that aren't checked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Named scalar and slice types (e.g. `type X int`, `type Elems []X`) are supported as the type parameter; the docs now say so explicitly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
These parse positional arguments without a format, one argument per value, with the same code as flag values: integers are in base 10 (a leading 0 does not indicate octal, which matters e.g. for Pebble file numbers) and a time.Duration is parsed with time.ParseDuration; other types are scanned like fmt.Sscan. ScanArgs detects calls of the v1 form (with a testing.TB and a key), which still compile, and points to FlagValue. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test file was read with a bufio.Scanner, which stops at a line longer than 64KiB; this (like any read error) was treated as the end of the file, so the rest of the file was silently ignored and, with -rewrite, removed from the file. The file is now read upfront, with no limit on the length of a line, and a read error fails the test. Co-Authored-By: Claude <noreply@anthropic.com>
With -rewrite, an output that starts with a ---- line was written after a single separator, where it reads back as a double separator: the output and the directives after it were then misparsed. Such an output is now written between double separators. An output that can't be written in either form (e.g. one with a blank line and two consecutive ---- lines) now fails the test instead of corrupting the file. Co-Authored-By: Claude <noreply@anthropic.com>
testing.T.Run doesn't call the function of a subtest that is not selected by the -run flag; the test then failed with a bogus "EOF encountered without subtest end directive" error, so -run could not be used to select a subtest of a test file. The directives of such a subtest are now skipped (and preserved by -rewrite). Co-Authored-By: Claude <noreply@anthropic.com>
With -rewrite, RunTestFromString ran in rewrite mode: the outputs were not compared with the expected outputs, and there is no file to rewrite, so its tests passed no matter what. It now always compares, and TestData.Rewrite reflects whether the file is being rewritten (rather than the flag), so that e.g. Retry keeps retrying. Co-Authored-By: Claude <noreply@anthropic.com>
A test can now retrieve flag values and positional arguments as its own types (e.g. keys, timestamps) by implementing encoding.TextUnmarshaler, with errors that carry the position in the test file; standard types like netip.Addr, time.Time and *big.Int work too. This is the single extension point for flags and arguments: PopArgs and ScanArgs no longer fall back to fmt.Sscan (and fmt.Scanner) for the types that the flag parser didn't support. Co-Authored-By: Claude <noreply@anthropic.com>
FlagValue and FlagValueOr only failed for an unsupported type when they parsed a value, so a call with such a type passed as long as the flag was missing (or an empty list) and only failed once a test file used the flag. The type is now checked on every call; the same goes for the values passed to PopArgs and ScanArgs. Co-Authored-By: Claude <noreply@anthropic.com>
Tests whose input is a sequence of commands parse each line with ParseLine and build a TestData by hand to read its flags, which are then never checked for being processed: a misspelled flag on such a line goes unnoticed, unlike on a directive line. InputDirectives returns each line as a TestData, with the same check, and reports errors at the position of the line. Co-Authored-By: Claude <noreply@anthropic.com>
An expected output that starts with a double ---- separator but is not closed by one extended to the end of the file: the directives after it were taken as expected output, and -rewrite removed them from the file. This is now an error. Co-Authored-By: Claude <noreply@anthropic.com>
The macos-14 runner image was retired, so the macOS job failed immediately with "The configuration 'macos-14-arm64-us-default' is not supported". Using macos-latest, like the Linux jobs use ubuntu-latest, keeps the job from breaking again when an image is retired. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A single value could not contain whitespace: parentheses were the only way to group words, and a parenthesized value is a list (which can only be retrieved as a slice). An argument or the value of a flag can now be written between single quotes, in which case it is taken literally; this also allows unbalanced parentheses, for which there was no notation. Quotes are only special at the start of an argument or of a flag value, so that quotes inside parentheses (e.g. SQL) and apostrophes keep their meaning. FullCmd and Flag.String quote the values that need it, so what they render is parsed back unchanged. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Repeated flags were rejected when parsing, because a lookup by name has to pick one of the occurrences (v1 silently used the first). But tests do use them for a sequence of values that are lists themselves, e.g. -probe=(a, 1) -probe=(b, 2), for which there was no other notation. A flag can now be repeated as long as the test retrieves all the occurrences, with FlagValues; the methods that retrieve a single value fail on a repeated flag, so an occurrence is never silently ignored. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Most directives with positional arguments take exactly one or two, and reading Args[0] directly ignores any extra arguments (or panics if there are none). These return the arguments and fail the directive if their number is not the expected one. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Some tests pass the directive line to another command-line parser (e.g. Pebble's tool tests run cobra commands). They could only rebuild it from Flags and Args, which loses the difference between -x and --x and the original quoting. DirectiveLine returns the line as it is written. Such a test doesn't retrieve the flags through TestData, so calling DirectiveLine also disables the check that all the flags are processed. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
An unprocessed flag failed the directive before its output was compared or rewritten. When a change makes a directive return early (e.g. with a new error), the flags it no longer gets to read were all that was reported: the output which explains them was hidden, and -rewrite could not record it. Unprocessed flags still fail the test (a flag that the test doesn't use should not be in the test file), but they no longer stop it: the output is compared or rewritten, the rest of the file is processed, and the flags are reported at the end, after any other failure. The same goes for the flags on the lines returned by InputDirectives. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A flag written without the dash (sync instead of -sync, or an argument left as lower=a when converting a v1 test file) is a positional argument: the test looks up the flag, doesn't find it, and silently does something other than what the test file says. Unlike a misspelled flag, nothing reported it. An argument of the form name or name=value now fails the test (like an unprocessed flag, without stopping it) if the test looked up a flag with that name and the flag is not on the line. Arguments that are quoted or after a -- word are exempt, for the rare argument that has the name of a flag. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A logger which holds the wrapper and logs from background goroutines gets its messages attributed to whatever directive is running, which is misleading; it should use the underlying testing.TB. The migration guide also didn't cover v1's ScanArgs with several destinations. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR (code and description) was generated with Claude Code.
This turns
masterinto v2 of the library:github.com/cockroachdb/datadriven/v2, requiring Go 1.27 (for generic methods). Thev1branch is at the currentmaster. v1 (v1.0.2 or later) and v2 can be linked in the same test binary, so a large repository can migrate one package at a time; seeMIGRATION.md.What changes
The callback gets a
testing.TBwrapper.RunTestis generic over thetesting.TBand passes the function a*datadriven.T(or*datadriven.B), which annotates errors, skips and logs with the position of the directive in the test file. The wrapper is also stored inTestData.T, soTestDatamethods no longer take at.Directive lines use command-line syntax. Words that start with
-or--followed by a letter are flags (-x,-x=v,-x=(v1, v2)); other words, and everything after--, are positional arguments.-msg='hello world'), in which case it is taken literally (''stands for a quote). Quotes are only special at the start of an argument or of a flag value, so quotes inside parentheses (e.g. SQL) keep their meaning.FlagValues; the single-value accessors fail on a repeated flag.The test must use what is on the line.
nameorname=valuefails the test if the test looked up flagnameand it is not on the line: it is most likely a flag which is missing the dash (quoted arguments and those after--are exempt).-rewrite), the rest of the file is processed, and the problems are reported when the file ends, after any other failure.DirectiveLine, which returns the line as written and disables these checks.Flags and arguments have separate APIs.
HasFlag,BoolFlag,FlagValue[V],FlagValueOr,FlagValues[V], plusMarkFlagProcessed/MarkAllFlagsProcessedfor code that iterates overTestData.Flags.TestData.Args,SingleArg,TwoArgs,PopArg,PopArgs,ScanArgs.time.Durationwithtime.ParseDuration, named types, slices for list values, and any type that implementsencoding.TextUnmarshaler. Unsupported Go types fail on every call, not only when a value is parsed.Inputs that are sequences of commands.
TestData.Posis a struct,InputLinesiterates over the input with per-line positions, andInputDirectivesyields each input line as aTestDatawith the same checks, reporting errors at that line.Fixes to long-standing problems.
-rewritethen truncated it.-rewritecorrupted the file for outputs that start with a----line; outputs that can't be represented now fail instead.----section swallowed the rest of the file, and-rewritedeleted it; it is now an error.-runcould not select asubtestof a test file.RunTestFromStringpassed vacuously under-rewrite; it now always compares.ClearResultsis fixed, andsubtestdirectives in benchmarks fail with a clear error.Docs. Package documentation in
doc.go(file format and API), a shorter README, andMIGRATION.mdwith mechanical rewrites and the list of behavior changes.Notes for reviewers
support quoted arguments and flag valueson) came out of evaluating Pebble's tests against the new API..github: run CI on the v1 branchis the first commit here, but v1 CI only starts once that change is also onv1.MIGRATION.md. The ones most likely to matter:RunTestFromStringcomparing under-rewrite, the unterminated double separator error,-rewriterejecting outputs with lines that end in a carriage return, and an argument or flag value that starts with a single quote losing its quotes.pkg/, excluding logic tests), three lines lose the quotes around a value (uri='nodelocal://...',oauth_token='...').go vet, gofmt, and the tests with-count=2,-raceand-rewrite(in a copy of the tree) pass on Go 1.27.2; each of the last seven commits also passes on its own.🤖 Generated with Claude Code
This change is