[daniboomerang]

Closes #1825

For: [model] (coding-agent CLI on a dev machine, dispatched locally, unattended)

Project: cli

Decisions

• Statement wording: the new sentences are inserted beside the two sentences they refine (the other-failure sentence and the forty-line sentence) rather than appended at the end, so each behaviour reads next to the rule it sharpens; the statement names behaviour only, no function, and the re-cut's banned-term check accepts it.
• apps/cli/specs/evals.md left unchanged: it holds no derived-file digest and describes the re-cut generically, so the statement change leaves it accurate.
• Commit header shortened to Fix(eval): Case pr-1088 states the push behaviour its tests check because the exact PR title exceeds the commit-header length ceiling; the PR title stays exact.
• Test Plan diff line reads origin/main...HEAD instead of the working tree: the brief's form diffs uncommitted changes, which are empty on a published head, so it exited nonzero with no output. Against the merge base it shows the committed change the line is meant to show.

Test plan

bun scripts/test-suites/local-run.ts exact --context 1f14415fad80d84e92b5744d1d668de2b4db76a1ff8215a49b2d3b48f6861257 -- apps/cli/tests/checks/changeset-coverage-bin.test.ts@default apps/cli/tests/checks/changeset-coverage.test.ts@default apps/cli/tests/checks/check-exec-bits.test.ts@default apps/cli/tests/checks/own-workflow.test.ts@default apps/cli/tests/ci-shards.test.ts@default apps/cli/tests/commands/task-status.test.ts@default apps/cli/tests/import-boundary.test.ts@default apps/cli/tests/lib/log-callers.test.ts@default apps/cli/tests/run-paths-only.test.ts@default apps/cli/tests/surface-spec-exports.test.ts@default → every selected test passes under its own native runner
bun apps/cli/tests/eval/recut-generic-cases.ts development → rewrites the four development case files and no other file
git diff -U0 origin/main...HEAD -- 'apps/cli/tests/eval/*-generic-cases.json' | grep -E '^[-+] +"taskStatement"' → only pr-1088 lines, in the developer and whole-loop files

Premise

Premise:

• apps/cli/tests/eval/generic-task-statements.json contains: return the push's own error output, bounded and sanitized, naming the git exit
• apps/cli/tests/eval/generic-suites.ts contains: export const DERIVED_FILE_DIGESTS = {

Evidence

Head: 87ea3b1
Summary: 6 files changed, 11 insertions(+), 11 deletions(-)

Group A — recomputable

git diff 31d4aeafe15146fbc63c4f3d533e3c07cc4a2824...87ea3b14ad54b813acf84a61d21093af738b33ca --numstat

2	2	apps/cli/tests/eval/developer-capability-generic-cases.json
4	4	apps/cli/tests/eval/generic-suites.ts
1	1	apps/cli/tests/eval/generic-task-statements.json
1	1	apps/cli/tests/eval/reviewer-quality-generic-cases.json
1	1	apps/cli/tests/eval/source-citation-generic-cases.json
2	2	apps/cli/tests/eval/whole-loop-generic-cases.json

Group B — attested

vinaya check --all --diff-only

Graded body: the live pull-request body (--push)

ready-gate: pass

Group C — Test Plan commands

C1: bun scripts/test-suites/local-run.ts exact --context 1f14415fad80d84e92b5744d1d668de2b4db76a1ff8215a49b2d3b48f6861257 -- apps/cli/tests/checks/changeset-coverage-bin.test.ts@default apps/cli/tests/checks/changeset-coverage.test.ts@default apps/cli/tests/checks/check-exec-bits.test.ts@default apps/cli/tests/checks/own-workflow.test.ts@default apps/cli/tests/ci-shards.test.ts@default apps/cli/tests/commands/task-status.test.ts@default apps/cli/tests/import-boundary.test.ts@default apps/cli/tests/lib/log-callers.test.ts@default apps/cli/tests/run-paths-only.test.ts@default apps/cli/tests/surface-spec-exports.test.ts@default

[... 16687 earlier characters truncated ...]


 108 pass
 0 fail
 1506 expect() calls
Ran 108 tests across 10 files. [29.35s]
vinaya-test-group-end {"group":"cli@default","executed":["apps/cli/tests/checks/changeset-coverage-bin.test.ts","apps/cli/tests/checks/changeset-coverage.test.ts","apps/cli/tests/checks/check-exec-bits.test.ts","apps/cli/tests/checks/own-workflow.test.ts","apps/cli/tests/ci-shards.test.ts","apps/cli/tests/commands/task-status.test.ts","apps/cli/tests/import-boundary.test.ts","apps/cli/tests/lib/log-callers.test.ts","apps/cli/tests/run-paths-only.test.ts","apps/cli/tests/surface-spec-exports.test.ts"],"failed":[]}

Reused from a green run recorded 2026-11-02T08:01:19.162Z (pre-push) — not re-run.

C2: bun apps/cli/tests/eval/recut-generic-cases.ts development

[... 43 earlier characters truncated ...]
eliverableSources": 0
    },
    {
      "file": "apps/cli/tests/eval/source-citation-generic-cases.json",
      "sha256": "0a48ca179713b91fce99980ad6edc4e507ad478d7220fdfe056a95804bf418a4",
      "cases": {
        "source-delivery-neutral.l5@1": 7
      },
      "layerPOnly": 0,
      "undeliverableSources": 0
    },
    {
      "file": "apps/cli/tests/eval/whole-loop-generic-cases.json",
      "sha256": "d3e8473f0effa93c0ee4d363677e6bceb9709ca844eb30b84fe5a42473918be3",
      "cases": {
        "whole-loop.l6@1": 28
      },
      "layerPOnly": 0,
      "undeliverableSources": 5
    }
  ]
}

Reused from a green run recorded 2026-11-02T07:58:34.611Z (pr-report) — not re-run.

C3: git diff -U0 origin/main...HEAD -- 'apps/cli/tests/eval/*-generic-cases.json' | grep -E '^[-+] +"taskStatement"'

[... 44 earlier characters truncated ...]
es is still returned as a hook refusal carrying its failing test. Add a test that drives a real git push through a real failing pre-push hook in a temporary repository and asserts both the classification and that a failing test's name reaches the refusal. Document what a refused push returns in the loop specification. Interface: export `apps/cli/src/lib/dev-review-loop.ts: function boundedPushOutput(output: string, mode: 'failing' | 'tail'): string`; export `apps/cli/src/lib/dev-review-loop.ts: function pushBranchClassified(worktreePath: string, branch: string): void`; string `git exit 128`.",

Reused from a green run recorded 2026-11-02T08:01:20.271Z (pr-report) — not re-run.

Scope

Eval fixtures only, in @attalabs/vinaya under apps/cli/tests/eval: the hand-written statement source for case pr-1088, the four development case files the re-cut regenerates (only the developer and whole-loop files change a taskStatement, and only that case's; the reviewer and source-citation files change only their derivedFrom digest), and the four development digest pins in generic-suites.ts. No source code, held-out split, re-cut script or quarantine file touched. Non-goal: any other case statement.

Tier: 1

[daniboomerang]

VERDICT: APPROVE

Judged head: 87ea3b1

Objectives version: 1560b287bfefae9f17e20702ebad74d23e35187bd642a7a85740afb01f1a2097

Ruling ordinal: 0

Brief hash: f02ee40da4de55296a6b1178697cf3310cff30b4374aafb886fb842ed36a5a95

Policy digest: bf51ddb9c7bd4776744d56ac91f4005ee8b2c1e4a5c50d1653eaa0ba8789accc

Judged base: 31d4aea

Manifest version: 2

Pause-bound ruling: (none)

BRIEF CONFORMANCE: Brief conformance: the PR edits the hand-written source generic-task-statements.json for pr-1088 and regenerates the development case files, so no statement was hand-edited. The new statement covers every behaviour O1 lists. It is free of banned terms: I checked it against BANNED_TERMS in banned-terms.ts. It names behaviour, not functions. Only the developer and whole-loop files change a taskStatement, and only pr-1088's, with the Interface block unchanged. All four development files now record the new statements digest 38e5c1f9… under derivedFrom, and all four development pins in DERIVED_FILE_DIGESTS changed. The held-out pins are untouched. Limit: I could not compute file hashes in this sandbox, so the new pin values themselves are unchecked; generic-suites.test.ts in the pre-push hook and CI checks them. Spec conformance: apps/cli/specs/evals.md is unchanged. It holds no digest and no pr-1088 text, and no old digest remains anywhere under apps/cli, so leaving it alone is justified and recorded in the PR body. Scope: every changed path is inside the brief's Modify list; nothing touches src, the re-cut script, quarantine or held-out files. Tests: no new tests are needed for a fixture re-cut; the existing suite test and re-cut test check the digests and banned terms. Docs: nothing else needed to change. No findings.
SPEC CONFORMANCE: (covered by the review summary)

OBJECTIVES:
O1: MET — diff.patch line 60, the generic-task-statements.json entry for pr-1088, adds four things. (1) 'the returned error's message itself, not only its output field, carries the git exit and git's own error text, as in git exit 128 and does not appear to be a git repository'. (2) 'A hook refusal's message names no git exit, and the refusal drops the hook's passing-test lines'. (3) 'when a refusal would pass forty lines, keep every failing-test line and drop detail lines first'. (4) 'a hook that prints tens of thousands of lines is still returned as a hook refusal carrying its failing test'. The same text appears verbatim in the regenerated developer case (diff line 19) and whole-loop case (diff line 108). No BANNED_TERMS word from apps/cli/src/lib/dev-review-eval/banned-terms.ts:32-58 is in the added text.
O2: MET — All four development case files (developer, reviewer-quality, source-citation, whole-loop) record the new statements digest 38e5c1f960ae… under derivedFrom (confirmed in the checkout at HEAD). The four DERIVED_FILE_DIGESTS.*.development values in generic-suites.ts:42-56 changed; every heldOut pin is unchanged. taskStatement changes appear only in the developer and whole-loop files, and only for pr-1088; the reviewer and source-citation files change only their derivedFrom digest. No held-out file is in the diff, and a grep finds none of the old development digests left under apps/cli. I could not compute the new hashes myself; the pinned-digest suite test checks them.

FINDINGS (ordered by severity):
None.

SCOPE: (covered by the review summary)
TESTS: (covered by the review summary)
DOCS: (covered by the review summary)

Tokens: 1825: review — Reviewer — claude — 15/3183/—
Cast by: Reviewer (session 90fd4c21-eda3-427a-9752-a832baab194f)

[daniboomerang]

VERDICT: PASS

Judged head: 87ea3b1

Objectives version: 1560b287bfefae9f17e20702ebad74d23e35187bd642a7a85740afb01f1a2097

Ruling ordinal: 0

Brief hash: f02ee40da4de55296a6b1178697cf3310cff30b4374aafb886fb842ed36a5a95

Policy digest: bf51ddb9c7bd4776744d56ac91f4005ee8b2c1e4a5c50d1653eaa0ba8789accc

Judged base: 31d4aea

Manifest version: 2

Pause-bound ruling: (none)

FINDINGS (ordered by severity):
None.

OBJECTIVES:
O1: MET — In generic-task-statements.json (diff line 60), the pr-1088 statement now says four things. (1) For a failure outside the hook, the returned error's message itself, not only its output field, carries the git exit and git's own error text, as in git exit 128 and does not appear to be a git repository. (2) A hook refusal's message names no git exit, and the refusal drops the hook's passing-test lines. (3) When a refusal would pass forty lines, every failing-test line is kept and detail lines are dropped first. (4) A hook that prints tens of thousands of lines is still read in full and returned as a hook refusal carrying its failing test. The same text appears verbatim in the developer case (diff line 19) and the whole-loop case (diff line 108), followed by the appended Interface block.
O2: MET — All four development case files (developer-capability, reviewer-quality, source-citation, whole-loop) change their derivedFrom digest for generic-task-statements.json from 4ec06b29… to 38e5c1f9…. Only the developer and whole-loop files change a taskStatement, and only pr-1088's. In generic-suites.ts, all four DERIVED_FILE_DIGESTS.*.development pins change, and every heldOut pin is unchanged. No held-out file appears in the diff, so the held-out split was not re-cut. I could not recompute the file digests because no checkout is available. I am relying on the generic-suites.test.ts pin check that CI will run.

CONFIG SCAN: (covered by the review summary)

SECRETS: SECRETS: none found in my own read of the diff. CI has not run, so the atta-labs/secret-scan check has not run on this head and I am not claiming it passed. The diff contains only eval-fixture text and SHA-256 digests. The token=<value> and /Users/<name> strings in the statement are placeholders describing redaction, not credentials. AGENT-CONFIG: not applicable. This pull request changes no agent configuration (.claude/, .mcp.json, .agents/), so no scanner was run, and my own read finds no agent, hook or MCP config in the diff. Scope: 6 files under apps/cli/tests/eval. These are the hand-written statement source for pr-1088, the four regenerated development case files (the developer and whole-loop files change pr-1088's taskStatement; the reviewer and source-citation files change only their derivedFrom digest), and the four development pins in generic-suites.ts. Every heldOut pin is unchanged. No source code, dependency, auth, crypto, shell or SQL surface is touched. The new statement text is static, maintainer-authored eval data with no untrusted input path, so it adds no injection surface. No security findings.

Tokens: 1825: security — Security — claude — 4/1675/—
Cast by: Security (session 80e93d66-c4b1-4663-8746-aa8171f07fbe)

[daniboomerang]

No description provided.

[github-actions]

AEG provenance — task (backlog Issue #1825)

• Issue: Fix(eval): Developer case pr-1088 states the push behaviour its fail-to-pass tests check #1825 (closed by merge)
• Tier: 1
• Brief: Fix(eval): Developer case pr-1088 states the push behaviour its fail-to-pass tests check #1825 (comment)
• Project(s): cli
• Model/agent: [model] (coding-agent CLI on a dev machine, dispatched locally, unattended)
• Code review: APPROVE
• Security: PASS
• Ticket: none
• Merged: 755a5b7 at 2026-11-02T08:06:11Z

Tokens: 1825: archive — Archivist — — — —