Fix(eval): Developer cases pr-1074, pr-1213 and pr-1320 state every behaviour their hidden tests assert by daniboomerang · Pull Request #1833 · atta-labs/vinaya

daniboomerang · 2026-11-02T14:00:09Z

For: Claude Sonnet 5.5, coding-agent CLI on a dev machine, dispatched locally

Project: cli

Decisions

Statement edit: appended the three brief sentences verbatim, after one space, to the pr-1320, pr-1074 and pr-1213 entries of generic-task-statements.json, and regenerated with the re-cut script instead of hand-editing any case file.
Digests: re-pinned all four development entries in DERIVED_FILE_DIGESTS from the regenerated files; held-out pins untouched.

Test plan

bun scripts/test-suites/local-run.ts exact --context 247c5e672700194088ff3264294d045e7ff04051a08745f0aad4dcc4119b7343 -- apps/cli/tests/checks/changeset-coverage-bin.test.ts@default apps/cli/tests/checks/changeset-coverage.test.ts@default apps/cli/tests/checks/check-exec-bits.test.ts@default apps/cli/tests/checks/own-workflow.test.ts@default apps/cli/tests/ci-shards.test.ts@default apps/cli/tests/eval/freeze-record.test.ts@default apps/cli/tests/eval/generic-suites.test.ts@default apps/cli/tests/eval/harbor-practice-fixes.test.ts@default apps/cli/tests/eval/harbor-tasks.test.ts@default apps/cli/tests/eval/step-one.test.ts@default apps/cli/tests/import-boundary.test.ts@default apps/cli/tests/lib/log-callers.test.ts@default apps/cli/tests/run-paths-only.test.ts@default apps/cli/tests/surface-spec-exports.test.ts@default harness/processes/vinaya-strict/tests/commands/task-status.test.ts@default scripts/package-compat/package-compat.test.ts@default → every selected test passes under its own native runner
bun apps/cli/tests/eval/recut-generic-cases.ts development → rewrites the four development case files and no other file
git diff -U0 origin/main -- 'apps/cli/tests/eval/*-generic-cases.json' | grep -E '^[-+] +"taskStatement"' → only pr-1074, pr-1213 and pr-1320 lines, in the developer and whole-loop files

Premise

Premise:

apps/cli/tests/eval/generic-suites.ts contains: export const DERIVED_FILE_DIGESTS = {

Evidence

Head: 1b998ba
Summary: 6 files changed, 17 insertions(+), 17 deletions(-)

Group A — recomputable

git diff 9c309050e0281c5934bf24261803bca2fb6079ad...1b998ba1648c6f4e8171c1f98f855d6b884f9be8 --numstat

4	4	apps/cli/tests/eval/developer-capability-generic-cases.json
4	4	apps/cli/tests/eval/generic-suites.ts
3	3	apps/cli/tests/eval/generic-task-statements.json
1	1	apps/cli/tests/eval/reviewer-quality-generic-cases.json
1	1	apps/cli/tests/eval/source-citation-generic-cases.json
4	4	apps/cli/tests/eval/whole-loop-generic-cases.json

Group B — attested

vinaya check --all --diff-only

Graded body: the live pull-request body (--push)

ready-gate: pass

Group C — Test Plan commands

C1: bun scripts/test-suites/local-run.ts exact --context 247c5e672700194088ff3264294d045e7ff04051a08745f0aad4dcc4119b7343 -- apps/cli/tests/checks/changeset-coverage-bin.test.ts@default apps/cli/tests/checks/changeset-coverage.test.ts@default apps/cli/tests/checks/check-exec-bits.test.ts@default apps/cli/tests/checks/own-workflow.test.ts@default apps/cli/tests/ci-shards.test.ts@default apps/cli/tests/eval/freeze-record.test.ts@default apps/cli/tests/eval/generic-suites.test.ts@default apps/cli/tests/eval/harbor-practice-fixes.test.ts@default apps/cli/tests/eval/harbor-tasks.test.ts@default apps/cli/tests/eval/step-one.test.ts@default apps/cli/tests/import-boundary.test.ts@default apps/cli/tests/lib/log-callers.test.ts@default apps/cli/tests/run-paths-only.test.ts@default apps/cli/tests/surface-spec-exports.test.ts@default harness/processes/vinaya-strict/tests/commands/task-status.test.ts@default scripts/package-compat/package-compat.test.ts@default

[... 54967 earlier characters truncated ...]
og file's existing content, then keeps running (killed by timeout, same as a real tail -f) [1525.40ms]
(pass) vinaya task status --follow (task-run-v1 task 15, O6) > --follow prints actions and detail lines by default [1525.45ms]
(pass) vinaya task status --follow (task-run-v1 task 15, O6) > --follow --quiet prints actions and failures without the detail lines [1523.70ms]

 17 pass
 0 fail
 43 expect() calls
Ran 17 tests across 1 file. [12.73s]
vinaya-test-group-end {"group":"process-vinaya@default","executed":["harness/processes/vinaya-strict/tests/commands/task-status.test.ts"],"failed":[]}

Reused from a green run recorded 2026-11-02T13:59:39.181Z (pre-push) — not re-run.

C2: `bun apps/cli/tests/eval/recut-generic-cases.ts development`

[... 43 earlier characters truncated ...]
eliverableSources": 0
    },
    {
      "file": "apps/cli/tests/eval/source-citation-generic-cases.json",
      "sha256": "70aefe09313be1da626513e358737a7c757cf8801cdc077cf4d53377d280a77e",
      "cases": {
        "source-delivery-neutral.l5@1": 7
      },
      "layerPOnly": 0,
      "undeliverableSources": 0
    },
    {
      "file": "apps/cli/tests/eval/whole-loop-generic-cases.json",
      "sha256": "d2f48226de690a0253ba37df73cd154a7e39a584cefe9bc72408beda8e09998e",
      "cases": {
        "whole-loop.l6@1": 28
      },
      "layerPOnly": 0,
      "undeliverableSources": 5
    }
  ]
}

Reused from a green run recorded 2026-11-02T14:00:01.659Z (pr-report) — not re-run.

C3: `git diff -U0 origin/main -- 'apps/cli/tests/eval/*-generic-cases.json' | grep -E '^[-+] +"taskStatement"'`

[... 45 earlier characters truncated ...]
nput.universes: Partial<Record<ComponentKind, readonly string[]>>`; field `ExitKindReport.covered: number`; field `ExitKindReport.emptyFields: number`; field `ExitKindReport.unmapped: number`; field `ExitReport.causes: ExitCause[]`; field `ExitReport.emptyFields: number`; field `ExitReport.kinds: Record<ComponentKind, ExitKindReport>`; field `ExitReport.pass: boolean`; field `ExitReport.unmapped: number`; string `count_mismatch`; string `empty_field`; string `kind_missing`; string `module_unnamed`; string `no_module`; string `shard_absent`; string `universe_extra`; string `universe_missing`.",

Reused from a green run recorded 2026-11-02T14:00:02.493Z (pr-report) — not re-run.

Scope

Only the evals data under apps/cli/tests/eval changes: the statements source, the four development case files and their digest pins. Held-out split, re-cut script and every other statement untouched.

Tier: 1

vercel · 2026-11-02T14:00:15Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
vinaya	Skipped		Nov 2, 2026 2:00pm UTC

daniboomerang · 2026-11-02T14:00:15Z

VERDICT: APPROVE

Judged head: 1b998ba

Objectives version: 2f2439683458fec466dfb9c92a656a9d9a77b74f6fad9a7e150927711c3ea8e1

Ruling ordinal: 0

Brief hash: f6dfe0e4229053802fca801497ca4f7972bcc235f823eda64c8fdbef3047d16d

Policy digest: bf51ddb9c7bd4776744d56ac91f4005ee8b2c1e4a5c50d1653eaa0ba8789accc

Judged base: 9c30905

Manifest version: 2

Pause-bound ruling: (none)

BRIEF CONFORMANCE: Brief conformance: the diff does what the brief asks. Three sentences are appended verbatim after one space to pr-1320, pr-1074 and pr-1213 in generic-task-statements.json. The case files carry the same text. Spec: apps/cli/specs/evals.md is unchanged, which fits a data-only re-cut with no behaviour change. Scope: only the seven surface-map files under apps/cli/tests/eval changed. There are no src, held-out or re-cut script changes. Tests: none were added and none were needed. I could not run the digest and re-cut checks because only the staged diff was available, so the digest values are unverified on my side. I checked that all four development files record the same new statements digest (888116f2…) and that the four DERIVED_FILE_DIGESTS development pins changed while the held-out pins did not. Only the developer and whole-loop files change any taskStatement text, and only for the three named cases. Docs: no doc owner bindings apply and the PR body is adequate. No findings.
SPEC CONFORMANCE: (covered by the review summary)

OBJECTIVES:
O1: MET — The pr-1320 statement in generic-task-statements.json (diff lines 95-96) and in the whole-loop case file (diff line 162) ends with the exact O1 sentence, appended after one space and before the Interface block. The developer-capability case file carries no pr-1320 taskStatement change in the diff; the sentence is in the statements file and the whole-loop copy.
O2: MET — The pr-1074 statement in the statements file, the developer-capability file and the whole-loop file now includes 'A control-record write made while one of those driver tool calls runs is still the driver's when the call throws.' after the 'Document in the isolation specification…' sentence.
O3: MET — The pr-1213 statement in the statements file, the developer-capability file and the whole-loop file now includes the verbatim applyQuiet/logQuiet() sentence after 'Add tests for both modes.'.
O4: MET — All four development case files (developer-capability, reviewer-quality, source-citation, whole-loop) are in the diff with the derivedFrom statements digest updated from 326aee9c… to 888116f2…. The four DERIVED_FILE_DIGESTS development pins in generic-suites.ts changed and the heldOut pins are untouched. The taskStatement changes are limited to pr-1074, pr-1213 and pr-1320 in the developer and whole-loop files. I did not recompute the digests.

FINDINGS (ordered by severity):
None.

SCOPE: (covered by the review summary)
TESTS: (covered by the review summary)
DOCS: (covered by the review summary)

Tokens: 1831: review — Reviewer — claude — 6/1865/—
Cast by: Reviewer (session d53341cb-341e-47a0-8f63-386648a40b60)

daniboomerang · 2026-11-02T14:00:19Z

VERDICT: PASS

Judged head: 1b998ba

Objectives version: 2f2439683458fec466dfb9c92a656a9d9a77b74f6fad9a7e150927711c3ea8e1

Ruling ordinal: 0

Brief hash: f6dfe0e4229053802fca801497ca4f7972bcc235f823eda64c8fdbef3047d16d

Policy digest: bf51ddb9c7bd4776744d56ac91f4005ee8b2c1e4a5c50d1653eaa0ba8789accc

Judged base: 9c30905

Manifest version: 2

Pause-bound ruling: (none)

FINDINGS (ordered by severity):
None.

OBJECTIVES:
O1: MET — The pr-1320 statement in generic-task-statements.json, developer-capability-generic-cases.json and whole-loop-generic-cases.json gains the exact objective text, from "Each kind's universe lists..." through "...adapter:codex rows.". It sits before the Interface block.
O2: MET — The pr-1074 statement gains "A control-record write made while one of those driver tool calls runs is still the driver's when the call throws." verbatim, in the statements file and the developer and whole-loop case files.
O3: MET — The pr-1213 statement gains the exact applyQuiet/logQuiet() sentence, in the statements file and the developer and whole-loop case files.
O4: MET — The diff changes the four development case files, via the statements-file sha256 pin in each of developer, reviewer, source-citation and whole-loop. generic-suites.ts DERIVED_FILE_DIGESTS re-pins all four development digests. Only the developer and whole-loop files change taskStatement text, and only for pr-1074, pr-1213 and pr-1320. The reviewer and source-citation files change only the digest pin. No heldOut digest or held-out file changes. I could not recompute the hash values here, so I checked structure and consistency only.

CONFIG SCAN: (covered by the review summary)

SECRETS: SECRETS: none found in my read of the diff. CI has not run (local review), so atta-labs/secret-scan is unconfirmed and I'm not claiming a scan pass. The only hex strings are SHA-256 digests of eval data files. Agent-config: not applicable. The PR changes no .claude/, .mcp.json or .agents/ files, so no scanner ran. The diff has no code, auth, dependency, injection or BYOK changes. It edits eval data JSON and pinned digests only: 3 task statements (pr-1074, pr-1213, pr-1320) in the statements file, mirrored into the developer and whole-loop case files, plus digest re-pins. I could not recompute the digests, because the repo checkout is not available here. No findings.

Tokens: 1831: security — Security — claude — 4/1512/—
Cast by: Security (session 86c608ae-c506-490f-b023-1704b390d1ea)

daniboomerang · 2026-11-02T14:00:22Z

No description provided.

github-actions · 2026-11-02T14:19:07Z

AEG provenance — task (backlog Issue #1831)

Issue: Fix(eval): Developer cases pr-1074, pr-1213 and pr-1320 state every behaviour their hidden tests assert #1831 (closed by merge)
Tier: 1
Brief: Fix(eval): Developer cases pr-1074, pr-1213 and pr-1320 state every behaviour their hidden tests assert #1831 (comment)
Project(s): cli
Model/agent: Claude Sonnet 5.5, coding-agent CLI on a dev machine, dispatched locally
Code review: APPROVE
Security: PASS
Ticket: none
Merged: bd3889b at 2026-11-02T14:18:35Z

Tokens: 1831: archive — Archivist — — — —

Fix(eval): Developer cases state every behaviour their tests assert

1b998ba

daniboomerang merged commit bd3889b into main Nov 2, 2026
22 of 28 checks passed

daniboomerang deleted the task/issue-1831 branch November 2, 2026 14:18

Navigation Menu

Repository navigation

Fix(eval): Developer cases pr-1074, pr-1213 and pr-1320 state every behaviour their hidden tests assert - #1833

Fix(eval): Developer cases pr-1074, pr-1213 and pr-1320 state every behaviour their hidden tests assert#1833
daniboomerang merged 1 commit into
mainfrom
task/issue-1831

daniboomerang commented Nov 2, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Nov 2, 2026

Uh oh!

daniboomerang commented Nov 2, 2026

Uh oh!

daniboomerang commented Nov 2, 2026

Uh oh!

daniboomerang commented Nov 2, 2026

Uh oh!

Uh oh!

github-actions Bot commented Nov 2, 2026

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

daniboomerang commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Decisions

Test plan

Premise

Evidence

Group A — recomputable

Group B — attested

Group C — Test Plan commands

C2: bun apps/cli/tests/eval/recut-generic-cases.ts development

C3: git diff -U0 origin/main -- 'apps/cli/tests/eval/*-generic-cases.json' | grep -E '^[-+] +"taskStatement"'

Scope

Uh oh!

vercel Bot commented Nov 2, 2026

Uh oh!

daniboomerang commented Nov 2, 2026

Uh oh!

daniboomerang commented Nov 2, 2026

Uh oh!

daniboomerang commented Nov 2, 2026

Uh oh!

Uh oh!

github-actions Bot commented Nov 2, 2026

AEG provenance — task (backlog Issue #1831)

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

daniboomerang commented Nov 2, 2026 •

edited

Loading

C2: `bun apps/cli/tests/eval/recut-generic-cases.ts development`

C3: `git diff -U0 origin/main -- 'apps/cli/tests/eval/*-generic-cases.json' | grep -E '^[-+] +"taskStatement"'`