Repository navigation
Fix(eval): Every role trial gives its agent the role doctrine the review loop gives it, and its record names that doctrine - #1813
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
|
VERDICT: APPROVE Judged head: 3d8c277 Objectives version: 1c6e8f31755ac4954063a232bc4f14292d0eca5d2ff2ca95d921eb4f3d1925e8 Ruling ordinal: 0 Brief hash: 2894175ed83e23c3d60fff8ff2e8c6b030499ed845243fe1cec30ab5b21a7b56 Policy digest: bf51ddb9c7bd4776744d56ac91f4005ee8b2c1e4a5c50d1653eaa0ba8789accc Judged base: 1578942 Manifest version: 2 Pause-bound ruling: (none) BRIEF CONFORMANCE: Brief conformance: the diff does what the three Parts ask. A new OBJECTIVES: FINDINGS (ordered by severity): SCOPE: (covered by the review summary) Tokens: 1812: review — Reviewer — claude — 21/5668/— |
|
VERDICT: PASS Judged head: 3d8c277 Objectives version: 1c6e8f31755ac4954063a232bc4f14292d0eca5d2ff2ca95d921eb4f3d1925e8 Ruling ordinal: 0 Brief hash: 2894175ed83e23c3d60fff8ff2e8c6b030499ed845243fe1cec30ab5b21a7b56 Policy digest: bf51ddb9c7bd4776744d56ac91f4005ee8b2c1e4a5c50d1653eaa0ba8789accc Judged base: 1578942 Manifest version: 2 Pause-bound ruling: (none) FINDINGS (ordered by severity): OBJECTIVES: CONFIG SCAN: (covered by the review summary) SECRETS: SECRETS: CI has not run, so the atta-labs/secret-scan check has not passed yet and has to be confirmed on the approved head. My own read of the diff found no credential. The only token-shaped value is the placeholder Tokens: 1812: security — Security — claude — 8/2757/— |
|
No description provided. |
|
Head: fb22d13 Checks the Developer reports running (agent-reported context, not evidence):
|
|
VERDICT: APPROVE Judged head: fb22d13 Objectives version: 1c6e8f31755ac4954063a232bc4f14292d0eca5d2ff2ca95d921eb4f3d1925e8 Ruling ordinal: 0 Brief hash: 2894175ed83e23c3d60fff8ff2e8c6b030499ed845243fe1cec30ab5b21a7b56 Policy digest: bf51ddb9c7bd4776744d56ac91f4005ee8b2c1e4a5c50d1653eaa0ba8789accc Judged base: 1578942 Manifest version: 2 Pause-bound ruling: (none) BRIEF CONFORMANCE: Brief conformance: the PR does what the brief asks. All three layer-G prompt sites in step-one.ts (change, review, answer) now start with a doctrine block. That block comes from the loop's own renderers: OBJECTIVES: FINDINGS (ordered by severity): SCOPE: (covered by the review summary) Tokens: 1812: review — Reviewer — claude — 17/5231/— |
|
VERDICT: PASS Judged head: fb22d13 Objectives version: 1c6e8f31755ac4954063a232bc4f14292d0eca5d2ff2ca95d921eb4f3d1925e8 Ruling ordinal: 0 Brief hash: 2894175ed83e23c3d60fff8ff2e8c6b030499ed845243fe1cec30ab5b21a7b56 Policy digest: bf51ddb9c7bd4776744d56ac91f4005ee8b2c1e4a5c50d1653eaa0ba8789accc Judged base: 1578942 Manifest version: 2 Pause-bound ruling: (none) FINDINGS (ordered by severity): OBJECTIVES: CONFIG SCAN: (covered by the review summary) SECRETS: SECRETS: CI is pending, so the atta-labs/secret-scan result is not confirmed yet. Reading the diff by hand found no credential. The only token-shaped string is Tokens: 1812: security — Security — claude — 13/3183/— |
|
No description provided. |
|
Ruling: resume. The worktree was deleted from outside the run, during the publish; that deletion, not this change, made the pre-push hook fail (direct-main-push, the committed-configuration probe and the CI plan tests were cut off while their files were being removed). Nothing in the task needs a decision. Redo the lost work on a fresh worktree: merge the default branch into |
|
The dev-review-loop paused: infrastructure — an uncaught error ended round 2's own processing: Command failed: git worktree add .worktrees/task/issue-1812 -b task/issue-1812 --no-track origin/main. A Principal ruling is needed before this can continue. A loop process still running picks up a posted ruling by itself; use this --resume command only when that process has stopped: |
|
Ruling: resume. The task worktree |
|
The dev-review-loop paused: escalation — the Developer reported the task blocked (tooling_unavailable): publish_changes timed out after 1800s twice in a row; the local merge commit 43bc92b exists, but the push never reached origin/task/issue-1812, which is still 6 commits behind. — Merged 42fbb96 cleanly onto the kept work (no conflicts …. A Principal ruling is needed before this can continue. A loop process still running picks up a posted ruling by itself; use this --resume command only when that process has stopped: |
|
Ruling: resume.
One real failure remains, in required CI "Build, lint & typecheck": the architecture ratchet reports |
|
The dev-review-loop paused: escalation — the Developer reported the task blocked (premise_stale): The handoff says the branch is behind 050f3e3 and conflicts, but 050f3e3 is already merged at b0c083e, the parent of head 4b37f4d; the driver's merge-proof check needs to look at ancestry rather than requiring the head itself to be a merg…. A Principal ruling is needed before this can continue. A loop process still running picks up a posted ruling by itself; use this --resume command only when that process has stopped: |
|
Ruling: resume. Confirmed: head |
|
The dev-review-loop paused: escalation — the Developer reported the task blocked (tooling_unavailable): The pre-push leak guard in apps/cli/tests/lib/test-env-preload.ts fails surface-spec-exports.test.ts and package-compat.test.ts because another loop's live run keeps writing `~/.vinaya/runtime/atta-labs-vinaya/engine-runs/1806/progress/t…. A Principal ruling is needed before this can continue. A loop process still running picks up a posted ruling by itself; use this --resume command only when that process has stopped: |
|
Ruling: resume. Confirmed: the leak guard tripped on another task's concurrent test output, not on this branch. That run stopped writing at 13:02 and no test run is active now. Publish |
|
Head: 7373e28 Checks the Developer reports running (agent-reported context, not evidence):
|
|
VERDICT: APPROVE Judged head: 7373e28 Objectives version: 1c6e8f31755ac4954063a232bc4f14292d0eca5d2ff2ca95d921eb4f3d1925e8 Ruling ordinal: 5 Brief hash: 2894175ed83e23c3d60fff8ff2e8c6b030499ed845243fe1cec30ab5b21a7b56 Policy digest: bf51ddb9c7bd4776744d56ac91f4005ee8b2c1e4a5c50d1653eaa0ba8789accc Judged base: 9dda4a4 Manifest version: 2 Pause-bound ruling: (none) BRIEF CONFORMANCE: Brief conformance: the PR does what the brief asks. The change, review and answer trial prompts in step-one.ts each start with a doctrine block made by the loop's own renderers: renderDeveloperDoctrineBlock for the developer, and joinReviewerPromptPieces(roleDoctrinePieces(role, text)) for a reviewer. Those are the same pieces reviewer-dispatch.ts:1280-1283 uses, so the PR adds no second renderer. Before rendering, the trial checks for a null or blank doctrine and, if it finds one, returns the trial as unavailable, naming the role. The doctrine is resolved through a new resolveRoleDoctrine process step that calls the loop's resolvers in the process's own working directory. The record gains an optional doctrineDigest field, and RENDERER_SCHEMA_VERSION moves to 2. Since round 1 the only changes are the merge of the default branch and the imports repointed to harness/processes/vinaya-strict/loops/dev-review/standalone/, as rulings 1 and 2 asked. Ruling 3 (architecture ratchet): architecture-baseline.json is not in the diff and still says 138. I counted the diff's step-one.ts lines with the ratchet's own rules: the two new process imports, the 'developer' literal and the 'reviewer' literal add four counted tokens. Four comment mentions of Developer/reviewer were reworded to 'agent' or 'role', which removes four. That nets to zero against the default branch, so the file should be back at 138; CI will confirm it. Spec conformance: apps/cli/specs/evals.md now describes each trial's doctrine block and where the doctrine is read from. It also covers the unavailable behaviour, the optional digest field, the schema bump, and the renderer's OBJECTIVES: FINDINGS (ordered by severity): SCOPE: (covered by the review summary) Tokens: 1812: review — Reviewer — claude — 21/7011/— |
|
VERDICT: PASS Judged head: 7373e28 Objectives version: 1c6e8f31755ac4954063a232bc4f14292d0eca5d2ff2ca95d921eb4f3d1925e8 Ruling ordinal: 5 Brief hash: 2894175ed83e23c3d60fff8ff2e8c6b030499ed845243fe1cec30ab5b21a7b56 Policy digest: bf51ddb9c7bd4776744d56ac91f4005ee8b2c1e4a5c50d1653eaa0ba8789accc Judged base: 9dda4a4 Manifest version: 2 Pause-bound ruling: (none) FINDINGS (ordered by severity): OBJECTIVES: CONFIG SCAN: (covered by the review summary) SECRETS: SECRETS: CI is pending, so the atta-labs/secret-scan result is not confirmed yet. I read the diff by hand and found no credential. The only token-shaped string is the placeholder Tokens: 1812: security — Security — claude — 6/2328/— |
|
No description provided. |
AEG provenance — task (backlog Issue #1812)
Tokens: 1812: archive — Archivist — — — — |
Closes #1812
For: [model] (coding-agent CLI on a dev machine, dispatched locally, unattended)
Project: cli
Decisions
trialDoctrineBlockreturns an unavailable run (the role is named in the reason) before any role step runs, instead of throwing. That way the trial reads as unavailable, not as a crashed process run, and no fake or real dispatch happens.joinReviewerPromptPieces(roleDoctrinePieces(role, text))renders it, leading blank lines included. The prompt then holds the loop's exact bytes, and the recorded digest is over that same string. Trimming it would have broken the byte-for-byte match.SHA-256, the same form as the other step-one digests (contentDigest,patchDigest).developer,reviewerorsecurity, which are the loop resolvers' own role ids.code-reviewermaps toreviewerandsecurity-reviewermaps tosecurity.aeg-root. It asserts that the prompt carries the repository's doctrine and none of the tree's.Test plan
SHA-256of the exact developer doctrine block the archive's own loop renders.Premise
Premise:
Evidence
Head: 7373e28
Summary:
7 files changed, 333 insertions(+), 21 deletions(-)Group A — recomputable
git diff 9dda4a41a7baeda648b406a6498683b0a021c2a6...7373e28d8a8878d9d0460f14899cd21f15005d3a --numstatGroup B — attested
vinaya check --all --diff-onlyGraded body: the live pull-request body (
--push)Group C — Test Plan commands
C1:
bun scripts/test-suites/local-run.ts exact --context 1f14415fad80d84e92b5744d1d668de2b4db76a1ff8215a49b2d3b48f6861257 -- apps/cli/tests/checks/changeset-coverage-bin.test.ts@default apps/cli/tests/checks/changeset-coverage.test.ts@default apps/cli/tests/checks/check-exec-bits.test.ts@default apps/cli/tests/checks/own-workflow.test.ts@default apps/cli/tests/ci-shards.test.ts@default apps/cli/tests/commands/task-status.test.ts@default apps/cli/tests/import-boundary.test.ts@default apps/cli/tests/lib/log-callers.test.ts@default apps/cli/tests/run-paths-only.test.ts@default apps/cli/tests/surface-spec-exports.test.ts@defaultReused from a green run recorded
2026-11-02T06:09:32.189Z(pre-push) — not re-run.Scope
This changes the layer-
Gtrial prompts instep-one.ts(the change, review and answer trials), adds an optionaldoctrineDigestfield toVinayaTrialRecord, and bumpsRENDERER_SCHEMA_VERSION, soVINAYA_RENDERER_VERSIONmoves by design. It also adds aresolveRoleDoctrineprocess step. The tests and the evals spec are updated to match. Only@attalabs/vinaya(apps/cli) is touched. Non-goals: the layer-Ptrials, the graders, the case files, the doctrine text, and the loop's own prompt builders, which are reused unchanged.Tier: 1