Skip to content

Fix(eval): Every role trial gives its agent the role doctrine the review loop gives it, and its record names that doctrine - #1813

Merged
daniboomerang merged 7 commits into
mainfrom
task/issue-1812
Nov 2, 2026
Merged

daniboomerang merged 7 commits into
mainfrom
task/issue-1812

Conversation

@daniboomerang

@daniboomerang daniboomerang commented Nov 1, 2026 •

Copy link
Copy Markdown
Member

Closes #1812

For: [model] (coding-agent CLI on a dev machine, dispatched locally, unattended)

Project: cli

Decisions

  • A missing doctrine: trialDoctrineBlock returns an unavailable run (the role is named in the reason) before any role step runs, instead of throwing. That way the trial reads as unavailable, not as a crashed process run, and no fake or real dispatch happens.
  • Reviewer block placement: the reviewer block is prepended exactly as joinReviewerPromptPieces(roleDoctrinePieces(role, text)) renders it, leading blank lines included. The prompt then holds the loop's exact bytes, and the recorded digest is over that same string. Trimming it would have broken the byte-for-byte match.
  • Digest form: the digest is a lowercase hex SHA-256, the same form as the other step-one digests (contentDigest, patchDigest).
  • Doctrine role ids: the step takes developer, reviewer or security, which are the loop resolvers' own role ids. code-reviewer maps to reviewer and security-reviewer maps to security.
  • Stale-tree proof: the step-one test runs the production step in a container trial whose tree is a git repository with its own aeg-root. It asserts that the prompt carries the repository's doctrine and none of the tree's.

Test plan

bun scripts/test-suites/local-run.ts exact --context 1f14415fad80d84e92b5744d1d668de2b4db76a1ff8215a49b2d3b48f6861257 -- apps/cli/tests/checks/changeset-coverage-bin.test.ts@default apps/cli/tests/checks/changeset-coverage.test.ts@default apps/cli/tests/checks/check-exec-bits.test.ts@default apps/cli/tests/checks/own-workflow.test.ts@default apps/cli/tests/ci-shards.test.ts@default apps/cli/tests/commands/task-status.test.ts@default apps/cli/tests/import-boundary.test.ts@default apps/cli/tests/lib/log-callers.test.ts@default apps/cli/tests/run-paths-only.test.ts@default apps/cli/tests/surface-spec-exports.test.ts@default → every selected test passes under its own native runner
  • [principal] One Harbor trial of a developer case: its record names a doctrine digest equal to the SHA-256 of the exact developer doctrine block the archive's own loop renders.

Premise

Premise:

  • harness/processes/vinaya-strict/loops/dev-review/standalone/dev-review-loop/developer-dispatch.ts contains: export function renderDeveloperDoctrineBlock(doctrine: string): string {
  • apps/cli/src/roles/plan.ts contains: export async function resolveRoleDoctrineText(role: 'reviewer' | 'security'): Promise<string | null> {

Evidence

Head: 7373e28
Summary: 7 files changed, 333 insertions(+), 21 deletions(-)

Group A — recomputable

git diff 9dda4a41a7baeda648b406a6498683b0a021c2a6...7373e28d8a8878d9d0460f14899cd21f15005d3a --numstat

33	3	apps/cli/specs/evals.md
87	10	apps/cli/src/lib/dev-review-eval/step-one.ts
6	1	apps/cli/src/lib/dev-review-eval/vinaya-renderer.ts
12	1	apps/cli/tests/eval/harbor-practice-fixes.test.ts
25	2	apps/cli/tests/eval/harbor-tasks.test.ts
154	3	apps/cli/tests/eval/step-one.test.ts
16	1	apps/cli/tests/eval/vinaya-renderer.test.ts

Group B — attested

vinaya check --all --diff-only

Graded body: the live pull-request body (--push)

ready-gate: pass

Group C — Test Plan commands

C1: bun scripts/test-suites/local-run.ts exact --context 1f14415fad80d84e92b5744d1d668de2b4db76a1ff8215a49b2d3b48f6861257 -- apps/cli/tests/checks/changeset-coverage-bin.test.ts@default apps/cli/tests/checks/changeset-coverage.test.ts@default apps/cli/tests/checks/check-exec-bits.test.ts@default apps/cli/tests/checks/own-workflow.test.ts@default apps/cli/tests/ci-shards.test.ts@default apps/cli/tests/commands/task-status.test.ts@default apps/cli/tests/import-boundary.test.ts@default apps/cli/tests/lib/log-callers.test.ts@default apps/cli/tests/run-paths-only.test.ts@default apps/cli/tests/surface-spec-exports.test.ts@default

[... 15633 earlier characters truncated ...]
vel folders as a path segment [64.45ms]

tests/surface-spec-exports.test.ts:
(pass) apps/cli/src/lib/**'s real exports are internally sane (O2) > has at least one real export to check (sanity) [0.07ms]
(pass) apps/cli/src/lib/**'s real exports are internally sane (O2) > has at least one real export to check for @attalabs/aeg-core's barrel (sanity) [1189.18ms]
(pass) apps/cli/src/lib/**'s real exports are internally sane (O2) > every retiresVia target that already names a real lib export names a function [172.20ms]

 108 pass
 0 fail
 1506 expect() calls
Ran 108 tests across 10 files. [33.20s]

Reused from a green run recorded 2026-11-02T06:09:32.189Z (pre-push) — not re-run.

Scope

This changes the layer-G trial prompts in step-one.ts (the change, review and answer trials), adds an optional doctrineDigest field to VinayaTrialRecord, and bumps RENDERER_SCHEMA_VERSION, so VINAYA_RENDERER_VERSION moves by design. It also adds a resolveRoleDoctrine process step. The tests and the evals spec are updated to match. Only @attalabs/vinaya (apps/cli) is touched. Non-goals: the layer-P trials, the graders, the case files, the doctrine text, and the loop's own prompt builders, which are reused unchanged.

Tier: 1

@vercel

vercel Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
vinaya Ignored Ignored Preview Nov 2, 2026 6:06am UTC

@daniboomerang

Copy link
Copy Markdown
Member Author

VERDICT: APPROVE

Judged head: 3d8c277

Objectives version: 1c6e8f31755ac4954063a232bc4f14292d0eca5d2ff2ca95d921eb4f3d1925e8

Ruling ordinal: 0

Brief hash: 2894175ed83e23c3d60fff8ff2e8c6b030499ed845243fe1cec30ab5b21a7b56

Policy digest: bf51ddb9c7bd4776744d56ac91f4005ee8b2c1e4a5c50d1653eaa0ba8789accc

Judged base: 1578942

Manifest version: 2

Pause-bound ruling: (none)

BRIEF CONFORMANCE: Brief conformance: the diff does what the three Parts ask. A new resolveRoleDoctrine step in VinayaProcessSteps runs the loop's own resolvers in the process's working directory. trialDoctrineBlock renders the block with the loop's own renderers: renderDeveloperDoctrineBlock for the developer, and joinReviewerPromptPieces(roleDoctrinePieces(...)) for a reviewer. It rejects a null or blank doctrine before rendering and marks the trial unavailable, naming the role. The block is prepended to the change, review and answer prompts, and each step-one record names its SHA-256 as doctrineDigest. That field is optional on VinayaTrialRecord, and RENDERER_SCHEMA_VERSION moves to 2. Spec conformance: apps/cli/specs/evals.md explains what each prompt holds and where the doctrine is read from. It also notes that the reused developer block names doctrine --role developer --print, and that the layer-P trials are left out. Both claims check out against the code: the Harbor agent runs cd /opt/vinaya, and resolveDoctrineRootInfo resolves from the process cwd. Scope: every path in the diff is in the Surface map, it touches no packages/* path, and it changes no grader, case file or doctrine text. Tests: they compare the trial prompt byte for byte against the loop's real renderers. They cover 4 suites × 2 adapters × host/container in step-one, both backends in harbor-tasks (the engine launch prompt included), and null/blank doctrine giving an unavailable trial with no step run. One test uses the production resolver against a git working tree that holds a stale aeg-root, and the record's round trip is checked in vinaya-renderer.test.ts. No test only asserts what a mock returned. Docs: the spec paragraph and the test-proof paragraphs describe the change accurately. No findings.
SPEC CONFORMANCE: (covered by the review summary)

OBJECTIVES:
O1: MET — step-one.ts runChangeCase calls trialDoctrineBlock(steps,'developer') and puts doctrine.block before input.brief. That block is renderDeveloperDoctrineBlock(text), the loop's own function. step-one.test.ts runs developer-capability-neutral.l5 on claude-code and codex, on host and in a container, and asserts that the prompt equals ${block}\n\n${brief} at its start. harbor-tasks.test.ts asserts launch.prompt.startsWith(doctrineBlock) on both backends. harbor-practice-fixes.test.ts asserts the change prompt starts with the developer block followed by the brief.
O2: MET — runReviewCase maps security-reviewer to 'security' and everything else to 'reviewer', then prepends joinReviewerPromptPieces(roleDoctrinePieces(role,text)). Those are the same pieces renderReviewerPrompt uses (reviewer-dispatch.ts:1272-1275). step-one.test.ts covers code-reviewer-quality-neutral.l5 and security-reviewer-quality-neutral.l5 on both adapters, on host and in a container, byte for byte. harbor-tasks.test.ts checks the review digest and the prompt prefix on both backends.
O3: MET — runAnswerCase calls trialDoctrineBlock(steps,'developer') and puts the block before the brief and the 'Answer from the delivered documents' line. step-one.test.ts covers source-delivery-neutral.l5 on both adapters, on host and in a container. The readOnlyTurn fake now records the prompt it receives.
O4: MET — In PRODUCTION_VINAYA_PROCESS_STEPS, resolveRoleDoctrine calls resolveDeveloperDoctrineText or resolveRoleDoctrineText with no tree argument, so they resolve from process.cwd. That cwd is /opt/vinaya in a container, because vinaya_harbor_agent.py:211 runs cd /opt/vinaya. The step-one test 'reads the doctrine from the Vinaya under test' runs the production step with a container working tree that is a git repository with an aeg-root marked STALE-TREE-DOCTRINE. It asserts that the prompt starts with the repository's own block, does not contain the stale text, and records digest == sha256(block). Every step-one record (change host and container, review, answer) sets doctrineDigest. The field is an optional Observed in vinaya-renderer.ts, and RENDERER_SCHEMA_VERSION is 2.

FINDINGS (ordered by severity):
None.

SCOPE: (covered by the review summary)
TESTS: (covered by the review summary)
DOCS: (covered by the review summary)

Tokens: 1812: review — Reviewer — claude — 21/5668/—
Cast by: Reviewer (session 9cda3dc8-407d-4d28-87e2-63fa8155db46)

@daniboomerang

Copy link
Copy Markdown
Member Author

VERDICT: PASS

Judged head: 3d8c277

Objectives version: 1c6e8f31755ac4954063a232bc4f14292d0eca5d2ff2ca95d921eb4f3d1925e8

Ruling ordinal: 0

Brief hash: 2894175ed83e23c3d60fff8ff2e8c6b030499ed845243fe1cec30ab5b21a7b56

Policy digest: bf51ddb9c7bd4776744d56ac91f4005ee8b2c1e4a5c50d1653eaa0ba8789accc

Judged base: 1578942

Manifest version: 2

Pause-bound ruling: (none)

FINDINGS (ordered by severity):
None.

OBJECTIVES:
O1: MET — step-one.ts:2008-2034: runChangeCase gets trialDoctrineBlock(steps,'developer'), which is renderDeveloperDoctrineBlock(text), and puts doctrine.block first in the prompt, ahead of input.brief. This path has no adapter or backend branch. Tests: step-one.test.ts loops over the claude-code and codex adapters, each on host and in a container, and asserts prompt.startsWith(${block}\n\n${brief}) with the block from the loop's own renderer. harbor-practice-fixes.test.ts and harbor-tasks.test.ts check the same thing on the standalone and Engine backends.
O2: MET — step-one.ts:2110-2114: runReviewCase maps security-reviewer to 'security' and every other reviewer to 'reviewer', and puts joinReviewerPromptPieces(roleDoctrinePieces(role,text)) first in the prompt. The step-one.test.ts doctrine matrix covers the code-reviewer-quality and security-reviewer-quality suites on both adapters, on host and in a container. harbor-tasks.test.ts:1003-1024 asserts the reviewer block and its digest on each backend.
O3: MET — step-one.ts:2187-2226: runAnswerCase puts the developer doctrine.block before the brief. The step-one.test.ts matrix covers the source-delivery-neutral answer trial against the developer block, on both adapters, host and container.
O4: MET — PRODUCTION_VINAYA_PROCESS_STEPS.resolveRoleDoctrine (step-one.ts:1280-1283) calls the loop's resolvers with no working-tree argument. resolveDeveloperDoctrineText (developer-dispatch.ts:1096) resolves from the process's own config path, and dev-review-eval has no chdir. The stale-tree test (step-one.test.ts:567-594) shows the prompt carries the repository's doctrine and none of the tree's STALE-TREE-DOCTRINE. Every step-one record now sets doctrineDigest: digest(block) (changeRecord, review record, answer record). vinaya-renderer.ts adds an optional doctrineDigest?: Observed<Digest> field and RENDERER_SCHEMA_VERSION moves to 2.

CONFIG SCAN: (covered by the review summary)

SECRETS: SECRETS: CI has not run, so the atta-labs/secret-scan check has not passed yet and has to be confirmed on the approved head. My own read of the diff found no credential. The only token-shaped value is the placeholder test-subscription-token, which step-one.test.ts sets in CLAUDE_CODE_OAUTH_TOKEN and then restores. AGENT-CONFIG: not applicable. This PR changes no agent configuration (.claude/, .mcp.json, .agents/), so no scanner was run, and my own read found no agent-config change. Security read: the change adds trusted, operator-controlled doctrine text to the start of each layer-G trial prompt. It does not add untrusted content to the prompts. The production resolveRoleDoctrine step calls the loop's own resolvers (resolveDeveloperDoctrineText and resolveRoleDoctrineText), which resolve from the process's own config and working directory. The dev-review-eval code has no process.chdir, so the case's working tree (/app, which holds an older aeg-root) cannot supply the doctrine. A stale-tree test confirms this: with a working tree carrying its own aeg-root, the prompt contains none of that tree's doctrine. When no doctrine resolves, or it resolves blank, the trial is unavailable before any step runs, so no trial dispatches without its doctrine. The PR adds no dependencies (only node:crypto in tests), no new tools, and no auth or permission change. No findings.

Tokens: 1812: security — Security — claude — 8/2757/—
Cast by: Security (session 1695d3ed-159d-42d6-99df-3f8540e8971f)

@daniboomerang

Copy link
Copy Markdown
Member Author

No description provided.

@daniboomerang

Copy link
Copy Markdown
Member Author

Head: fb22d13
Pause-bound ruling: (none)

Checks the Developer reports running (agent-reported context, not evidence):

  • bun test harness/kernel/tests/architecture.test.ts -t "counted ratchet" — pass
  • bun test apps/cli/tests/eval/step-one.test.ts -t "doctrine|records what the host records" — pass
  • bunx tsc --noEmit -p apps/cli — pass

@daniboomerang

Copy link
Copy Markdown
Member Author

VERDICT: APPROVE

Judged head: fb22d13

Objectives version: 1c6e8f31755ac4954063a232bc4f14292d0eca5d2ff2ca95d921eb4f3d1925e8

Ruling ordinal: 0

Brief hash: 2894175ed83e23c3d60fff8ff2e8c6b030499ed845243fe1cec30ab5b21a7b56

Policy digest: bf51ddb9c7bd4776744d56ac91f4005ee8b2c1e4a5c50d1653eaa0ba8789accc

Judged base: 1578942

Manifest version: 2

Pause-bound ruling: (none)

BRIEF CONFORMANCE: Brief conformance: the PR does what the brief asks. All three layer-G prompt sites in step-one.ts (change, review, answer) now start with a doctrine block. That block comes from the loop's own renderers: renderDeveloperDoctrineBlock, and joinReviewerPromptPieces(roleDoctrinePieces(role, text)) for reviewers. The PR adds no second renderer. Before rendering, the trial checks the resolved text for null or blank and fails as unavailable, naming the role. The text is resolved through a new resolveRoleDoctrine VinayaProcessSteps step. Its production form calls resolveDeveloperDoctrineText and resolveRoleDoctrineText in the process's own working directory. The Harbor agent runs cd /opt/vinaya (vinaya_harbor_agent.py:211), so resolveDoctrineRootInfo does not read /app. doctrineDigest?: Observed<Digest> is optional and every step-one record sets it. RENDERER_SCHEMA_VERSION moved to 2. Spec conformance: apps/cli/specs/evals.md covers what each trial's prompt holds, where the doctrine is read from, the unavailable behaviour, the digest field and the schema bump. It also says the developer block's doctrine --role developer --print hint is reused as it is, which the brief required. Scope: every changed path is in the brief's Surface Modify list; layer-P trials, graders, case files and doctrine text are untouched. Tests: step-one.test.ts runs every combination of developer, code-reviewer, security-reviewer and answer trial, on both adapters, on the host and in a container. Each asserts the prompt is the loop-rendered block byte for byte followed by the brief, plus the sha256 digest. Null and blank doctrine leave the trial unavailable with no step called. A real-production-resolver test against a stale git aeg-root tree checks the prompt carries none of the tree's doctrine. harbor-tasks.test.ts covers both backends and the launch prompt prefix. The renderer test covers the round-trip and that no outcome reads the digest. The fakes stand in for the doctrine source only; the code under test (the prompt assembly) runs for real. CI is still pending, so the pass/fail evidence for these tests is not in yet. Docs read truthfully. No blocking findings.
SPEC CONFORMANCE: (covered by the review summary)

OBJECTIVES:
O1: MET — step-one.ts runChangeCase calls trialDoctrineBlock(steps, 'developer'), which returns renderDeveloperDoctrineBlock(text), and puts doctrine.block first in the prompt, ahead of input.brief. step-one.test.ts tests the developer-capability change suite on claude-code and codex, host and container: prompt.startsWith(${block}\n\n${brief}), where block is the loop's own rendering. harbor-tasks.test.ts checks launch.prompt.startsWith(doctrineBlock) and the digest on both backends. harbor-practice-fixes.test.ts checks the block comes before the brief.
O2: MET — runReviewCase maps security-reviewer to 'security' and other reviewers to 'reviewer'. trialDoctrineBlock renders joinReviewerPromptPieces(roleDoctrinePieces(role, text)), the same pieces the loop's reviewer dispatch uses (reviewer-dispatch.ts:1272), and the trial puts that block first in the prompt. The step-one.test.ts matrix covers code-reviewer-quality and security-reviewer-quality on both adapters, host and container, with a byte-for-byte prefix check. harbor-tasks.test.ts covers the review kind on both backends.
O3: MET — runAnswerCase calls trialDoctrineBlock(steps, 'developer') and puts doctrine.block before the brief and the 'Answer from the delivered documents' line. The source-delivery-neutral answer suite is in the step-one.test.ts DOCTRINE_KINDS matrix with the same byte-for-byte and digest checks, and in the stale-tree production-resolver test.
O4: MET — PRODUCTION_VINAYA_PROCESS_STEPS.resolveRoleDoctrine calls the loop's resolvers with no cwd argument, so they use process.cwd(). The Harbor agent runs from /opt/vinaya (vinaya_harbor_agent.py:211 cd /opt/vinaya), where resolveDoctrineRootInfo (doctrine.ts:83) falls back to the bundled aeg-root. On the host, process.cwd() is the repository. The stale-tree test runs the real production step against a working tree that is a git repository with its own aeg-root and asserts STALE-TREE-DOCTRINE is absent. Every step-one record sets doctrineDigest = digest(block), which tests assert equals the sha256 of the prepended block (vinaya-renderer.ts:301 adds the optional field).

FINDINGS (ordered by severity):
None.

SCOPE: (covered by the review summary)
TESTS: (covered by the review summary)
DOCS: (covered by the review summary)

Tokens: 1812: review — Reviewer — claude — 17/5231/—
Cast by: Reviewer (session c9910180-da3a-46ae-adb1-2e571bacf7b5)

@daniboomerang

Copy link
Copy Markdown
Member Author

VERDICT: PASS

Judged head: fb22d13

Objectives version: 1c6e8f31755ac4954063a232bc4f14292d0eca5d2ff2ca95d921eb4f3d1925e8

Ruling ordinal: 0

Brief hash: 2894175ed83e23c3d60fff8ff2e8c6b030499ed845243fe1cec30ab5b21a7b56

Policy digest: bf51ddb9c7bd4776744d56ac91f4005ee8b2c1e4a5c50d1653eaa0ba8789accc

Judged base: 1578942

Manifest version: 2

Pause-bound ruling: (none)

FINDINGS (ordered by severity):
None.

OBJECTIVES:
O1: MET — In step-one.ts, runChangeCase calls trialDoctrineBlock(steps,'developer'), which calls renderDeveloperDoctrineBlock (the loop's own renderer). It puts doctrine.block first in the prompt array, before input.brief. In step-one.test.ts, the 'every role trial gives its agent the role doctrine' tests loop over claude-code and codex, and over host and container. They assert prompt.startsWith(${block}\n\n${brief}). harbor-practice-fixes.test.ts asserts the same for the container change prompt.
O2: MET — runReviewCase maps security-reviewer to 'security' and anything else to 'reviewer'. trialDoctrineBlock then renders joinReviewerPromptPieces(roleDoctrinePieces(role,text)) and prepends it to the prompt. The step-one.test.ts DOCTRINE_KINDS list includes the code-reviewer-quality and security-reviewer-quality suites, run on both adapters and both modes, with a byte-for-byte prefix assertion. harbor-tasks.test.ts checks the launch prompt prefix and the digest on both backends.
O3: MET — runAnswerCase calls trialDoctrineBlock(steps,'developer') and prepends doctrine.block ahead of the brief. The source-delivery-neutral.l5 answer case is in DOCTRINE_KINDS, and its fake readOnlyTurn now records prompts so the prefix can be asserted.
O4: MET — PRODUCTION_VINAYA_PROCESS_STEPS.resolveRoleDoctrine calls the loop resolvers. Those use process.cwd() through resolveDoctrineRootInfo and findLocalConfig, and dev-review-eval has no chdir. The test "reads the doctrine from the Vinaya under test, never the trial's working tree" builds a git tree with a STALE-TREE-DOCTRINE aeg-root and asserts that the stale text never appears in the prompt. VinayaTrialRecord gains doctrineDigest?: Observed, which every step-one record sets to digest(block). RENDERER_SCHEMA_VERSION is bumped to 2, and the round-trip test is in vinaya-renderer.test.ts.

CONFIG SCAN: (covered by the review summary)

SECRETS: SECRETS: CI is pending, so the atta-labs/secret-scan result is not confirmed yet. Reading the diff by hand found no credential. The only token-shaped string is test-subscription-token, a placeholder that step-one.test.ts sets in CLAUDE_CODE_OAUTH_TOKEN and then puts back in a finally block. The fake doctrine strings are not secrets either. AGENT-CONFIG: the scan was not applicable, because this PR changes no agent configuration (.claude/, .mcp.json, .agents/). My own read finds no agent, hook or MCP change. Security read: the change prepends trusted role doctrine to trial prompts. That doctrine is resolved by the loop's own resolvers (resolveDeveloperDoctrineText and resolveRoleDoctrineText). Both use process.cwd() through resolveDoctrineRootInfo and findLocalConfig, and nothing under dev-review-eval calls chdir. So the case's working tree (/app), which holds attacker-shaped base-revision content, can't feed text into the doctrine block. A test proves this against a stale git tree carrying its own aeg-root. A null or blank doctrine makes the trial unavailable before any step runs, so no trial is dispatched without its doctrine. The PR adds no dependency, auth surface, SQL or shell path. The new imports are in-repo modules, and node:crypto is used only for the SHA-256 digest. No findings.

Tokens: 1812: security — Security — claude — 13/3183/—
Cast by: Security (session 5a9979be-8909-420a-808b-d500498d5162)

@daniboomerang

Copy link
Copy Markdown
Member Author

No description provided.

@daniboomerang

Copy link
Copy Markdown
Member Author

Ruling: resume.

The worktree was deleted from outside the run, during the publish; that deletion, not this change, made the pre-push hook fail (direct-main-push, the committed-configuration probe and the CI plan tests were cut off while their files were being removed). Nothing in the task needs a decision.

Redo the lost work on a fresh worktree: merge the default branch into task/issue-1812, resolve the conflicts as before (point the publishingInstructionLine and the developer-dispatch and reviewer-dispatch imports in step-one.ts, harbor-practice-fixes.test.ts, harbor-tasks.test.ts and step-one.test.ts at the driver's new home under harness/processes/vinaya-strict/loops/dev-review/standalone/), run the tests, and publish.

@daniboomerang

Copy link
Copy Markdown
Member Author

The dev-review-loop paused: infrastructure — an uncaught error ended round 2's own processing: Command failed: git worktree add .worktrees/task/issue-1812 -b task/issue-1812 --no-track origin/main.

A Principal ruling is needed before this can continue. A loop process still running picks up a posted ruling by itself; use this --resume command only when that process has stopped:

vinaya task run --issue 1812 --agent claude --model opus --runner atta-engine

@daniboomerang

Copy link
Copy Markdown
Member Author

Ruling: resume.

The task worktree .worktrees/task/issue-1812 is back on the existing local branch, at the Developer's own merge commit 4c5f851 (the pull request head plus the current default branch, with the doctrine imports repointed). Continue from it: run the tests and publish.

@daniboomerang

Copy link
Copy Markdown
Member Author

The dev-review-loop paused: escalation — the Developer reported the task blocked (tooling_unavailable): publish_changes timed out after 1800s twice in a row; the local merge commit 43bc92b exists, but the push never reached origin/task/issue-1812, which is still 6 commits behind. — Merged 42fbb96 cleanly onto the kept work (no conflicts ….

A Principal ruling is needed before this can continue. A loop process still running picks up a posted ruling by itself; use this --resume command only when that process has stopped:

vinaya task run --issue 1812 --agent claude --model opus --runner atta-engine

@daniboomerang

Copy link
Copy Markdown
Member Author

Ruling: resume.

43bc92b9d is now on origin/task/issue-1812 (pushed for you; the publish timeouts came from the hook running every test file, because the merge of the default branch counted as changed). Your next publish diffs against it.

One real failure remains, in required CI "Build, lint & typecheck": the architecture ratchet reports apps/cli/src/lib/dev-review-eval/step-one.ts: counts 140, baseline 138 — above its baseline entry. Bring step-one.ts back to at most 138 counted entries (the merge's repointed imports added two); do not raise the baseline. Run cd harness/kernel && bun test tests/architecture.test.ts until it passes, then publish.

@daniboomerang

Copy link
Copy Markdown
Member Author

The dev-review-loop paused: escalation — the Developer reported the task blocked (premise_stale): The handoff says the branch is behind 050f3e3 and conflicts, but 050f3e3 is already merged at b0c083e, the parent of head 4b37f4d; the driver's merge-proof check needs to look at ancestry rather than requiring the head itself to be a merg….

A Principal ruling is needed before this can continue. A loop process still running picks up a posted ruling by itself; use this --resume command only when that process has stopped:

vinaya task run --issue 1812 --agent claude --model opus --runner atta-engine

@daniboomerang

Copy link
Copy Markdown
Member Author

Ruling: resume.

Confirmed: head 4b37f4da7 already contains the default branch tip 050f3e3c8 (through merge commit b0c083e61), and GitHub reports the pull request mergeable. There is nothing to merge. Do not make an empty merge commit. Finish the turn as ready on the current head so review and CI run on it.

@daniboomerang

Copy link
Copy Markdown
Member Author

The dev-review-loop paused: escalation — the Developer reported the task blocked (tooling_unavailable): The pre-push leak guard in apps/cli/tests/lib/test-env-preload.ts fails surface-spec-exports.test.ts and package-compat.test.ts because another loop's live run keeps writing `~/.vinaya/runtime/atta-labs-vinaya/engine-runs/1806/progress/t….

A Principal ruling is needed before this can continue. A loop process still running picks up a posted ruling by itself; use this --resume command only when that process has stopped:

vinaya task run --issue 1812 --agent claude --model opus --runner atta-engine

@daniboomerang

Copy link
Copy Markdown
Member Author

Ruling: resume.

Confirmed: the leak guard tripped on another task's concurrent test output, not on this branch. That run stopped writing at 13:02 and no test run is active now. Publish 7373e28d8 again.

@daniboomerang

Copy link
Copy Markdown
Member Author

Head: 7373e28
Pause-bound ruling: (none)

Checks the Developer reports running (agent-reported context, not evidence):

  • cd harness/kernel && bun test tests/architecture.test.ts — pass

@daniboomerang

Copy link
Copy Markdown
Member Author

VERDICT: APPROVE

Judged head: 7373e28

Objectives version: 1c6e8f31755ac4954063a232bc4f14292d0eca5d2ff2ca95d921eb4f3d1925e8

Ruling ordinal: 5

Brief hash: 2894175ed83e23c3d60fff8ff2e8c6b030499ed845243fe1cec30ab5b21a7b56

Policy digest: bf51ddb9c7bd4776744d56ac91f4005ee8b2c1e4a5c50d1653eaa0ba8789accc

Judged base: 9dda4a4

Manifest version: 2

Pause-bound ruling: (none)

BRIEF CONFORMANCE: Brief conformance: the PR does what the brief asks. The change, review and answer trial prompts in step-one.ts each start with a doctrine block made by the loop's own renderers: renderDeveloperDoctrineBlock for the developer, and joinReviewerPromptPieces(roleDoctrinePieces(role, text)) for a reviewer. Those are the same pieces reviewer-dispatch.ts:1280-1283 uses, so the PR adds no second renderer. Before rendering, the trial checks for a null or blank doctrine and, if it finds one, returns the trial as unavailable, naming the role. The doctrine is resolved through a new resolveRoleDoctrine process step that calls the loop's resolvers in the process's own working directory. The record gains an optional doctrineDigest field, and RENDERER_SCHEMA_VERSION moves to 2. Since round 1 the only changes are the merge of the default branch and the imports repointed to harness/processes/vinaya-strict/loops/dev-review/standalone/, as rulings 1 and 2 asked. Ruling 3 (architecture ratchet): architecture-baseline.json is not in the diff and still says 138. I counted the diff's step-one.ts lines with the ratchet's own rules: the two new process imports, the 'developer' literal and the 'reviewer' literal add four counted tokens. Four comment mentions of Developer/reviewer were reworded to 'agent' or 'role', which removes four. That nets to zero against the default branch, so the file should be back at 138; CI will confirm it. Spec conformance: apps/cli/specs/evals.md now describes each trial's doctrine block and where the doctrine is read from. It also covers the unavailable behaviour, the optional digest field, the schema bump, and the renderer's doctrine --role developer --print hint being reused as it is. That matches the code. Scope: all seven changed paths are in the brief's Modify list, and layer-P trials, graders, case files and doctrine text are untouched. Tests: step-one.test.ts runs four role kinds on both adapters, on the host and in a container, and checks each prompt against the loop's rendering byte for byte, followed by the brief, plus the sha256 digest. It also checks the null and blank cases, and runs the real production resolver against a stale git aeg-root tree. The fakes stand in only for the doctrine source; the prompt assembly runs for real. harbor-tasks.test.ts and harbor-practice-fixes.test.ts check the digest and the prompt prefix on both backends, and the renderer test checks the round-trip. CI is pending. Docs read accurately. No findings.
SPEC CONFORMANCE: (covered by the review summary)

OBJECTIVES:
O1: MET — step-one.ts runChangeCase calls trialDoctrineBlock(steps, doctrineRoleOf(request)). For a change case that is 'developer', which renders renderDeveloperDoctrineBlock(text) (developer-dispatch.ts:1127, the loop's own renderer). The prompt array is [doctrine.block, input.brief, ...]. In step-one.test.ts, the 'every role trial gives its agent the role doctrine' matrix runs developer-capability-neutral.l5 on claude-code and codex, host and container, and asserts prompt.slice(0, block.length) === block and prompt.startsWith(${block}\n\n${brief}). harbor-tasks.test.ts asserts launch.prompt.startsWith(doctrineBlock) and the digest on both backends. harbor-practice-fixes.test.ts asserts the block comes before the brief.
O2: MET — In runReviewCase, doctrineRoleOf maps security-reviewer-quality-neutral.l5 to 'security' and other review suites to 'reviewer'. trialDoctrineBlock then renders joinReviewerPromptPieces(roleDoctrinePieces(role, text)), the same pieces the reviewer dispatch builds (reviewer-dispatch.ts:1280-1283), and the prompt starts with that block. The step-one.test.ts matrix covers code-reviewer-quality and security-reviewer-quality on both adapters and both modes with a byte-for-byte prefix check. harbor-tasks.test.ts checks the review digest on both backends.
O3: MET — runAnswerCase calls trialDoctrineBlock(steps, 'developer' via doctrineRoleOf) and puts doctrine.block before input.brief and the 'Answer from the delivered documents' line. source-delivery-neutral.l5 is in DOCTRINE_KINDS, whose readOnlyTurn fake now records prompts, with the same byte-for-byte and digest checks. It is also in the stale-tree production-resolver test.
O4: MET — PRODUCTION_VINAYA_PROCESS_STEPS.resolveRoleDoctrine calls resolveDeveloperDoctrineText() or resolveRoleDoctrineText(role) with no working-directory argument, so they resolve from process.cwd(), the Vinaya under test. The test "reads the doctrine from the Vinaya under test, never the trial's working tree" builds a git working tree with its own STALE-TREE-DOCTRINE aeg-root. It runs the real production step and asserts the prompt starts with the repository's block, never contains the stale text, and records sha256(block). Every step-one record (changeRecord, the review record and the answer record) sets doctrineDigest: doctrine.digest = digest(block). VinayaTrialRecord declares doctrineDigest?: Observed (vinaya-renderer.ts), and RENDERER_SCHEMA_VERSION is 2.

FINDINGS (ordered by severity):
None.

SCOPE: (covered by the review summary)
TESTS: (covered by the review summary)
DOCS: (covered by the review summary)

Tokens: 1812: review — Reviewer — claude — 21/7011/—
Cast by: Reviewer (session 39fe7e02-c282-4afa-a46f-198f3b971b74)

@daniboomerang

Copy link
Copy Markdown
Member Author

VERDICT: PASS

Judged head: 7373e28

Objectives version: 1c6e8f31755ac4954063a232bc4f14292d0eca5d2ff2ca95d921eb4f3d1925e8

Ruling ordinal: 5

Brief hash: 2894175ed83e23c3d60fff8ff2e8c6b030499ed845243fe1cec30ab5b21a7b56

Policy digest: bf51ddb9c7bd4776744d56ac91f4005ee8b2c1e4a5c50d1653eaa0ba8789accc

Judged base: 9dda4a4

Manifest version: 2

Pause-bound ruling: (none)

FINDINGS (ordered by severity):
None.

OBJECTIVES:
O1: MET — step-one.ts runChangeCase calls trialDoctrineBlock(steps, doctrineRoleOf(request)), which resolves to the 'developer' role and returns renderDeveloperDoctrineBlock(text). It puts doctrine.block first in the prompt array, before input.brief. step-one.test.ts runs the developer-capability-neutral.l5 change trial on claude-code and codex, on the host and in a container, and asserts prompt.startsWith(${block}\n\n${brief}), where block is the loop's own rendering. harbor-practice-fixes.test.ts asserts the developer block comes before the brief. harbor-tasks.test.ts checks launch.prompt.startsWith(doctrineBlock) and the digest on both backends.
O2: MET — runReviewCase calls trialDoctrineBlock with 'security' for the security-reviewer-quality suite and 'reviewer' otherwise. The block is joinReviewerPromptPieces(roleDoctrinePieces(role, text)), the loop's own reviewer renderer, and it is prepended to the prompt. The DOCTRINE_KINDS matrix in step-one.test.ts covers the code-reviewer-quality and security-reviewer-quality suites on both adapters and both modes, with a byte-for-byte prefix check. harbor-tasks.test.ts covers the review kind on both backends.
O3: MET — runAnswerCase calls trialDoctrineBlock(steps, 'developer') and puts doctrine.block before the brief and the 'Answer from the delivered documents' line. source-delivery-neutral.l5 is in DOCTRINE_KINDS, and the fake readOnlyTurn now records prompts, so the test asserts the prefix and the digest. The stale-tree production test also covers the answer kind.
O4: MET — PRODUCTION_VINAYA_PROCESS_STEPS.resolveRoleDoctrine calls resolveDeveloperDoctrineText() or resolveRoleDoctrineText(role) with no cwd argument, so they read from the process's working directory: /opt/vinaya in a container, the repository on the host. They never read the trial tree. The test "reads the doctrine from the Vinaya under test, never the trial's working tree" runs the production step against a git tree carrying STALE-TREE-DOCTRINE in its own aeg-root and asserts that text never appears in the prompt. VinayaTrialRecord gains the optional doctrineDigest?: Observed. The change, review and answer records all set it to digest(block), and the tests assert it equals sha256 of the prepended block. RENDERER_SCHEMA_VERSION is bumped to 2.

CONFIG SCAN: (covered by the review summary)

SECRETS: SECRETS: CI is pending, so the atta-labs/secret-scan result is not confirmed yet. I read the diff by hand and found no credential. The only token-shaped string is the placeholder test-subscription-token. step-one.test.ts sets it in CLAUDE_CODE_OAUTH_TOKEN for the doctrine trials and restores the prior value in a finally block. The fake doctrine strings and the 'a'.repeat(64) digest are not secrets either. AGENT-CONFIG: the scan was not applicable, because this PR changes no agent configuration (.claude/, .mcp.json, .agents/). My own read finds no agent, hook or MCP change. Security read: compared with round 1, this head only repoints the doctrine imports to the driver's new home under harness/processes/vinaya-strict/loops/dev-review/standalone/dev-review-loop/, as ruling 1 directed. The logic is the same. The doctrine prepended to trial prompts is trusted text, resolved by the loop's own resolvers (resolveDeveloperDoctrineText and resolveRoleDoctrineText) in the process's own working directory. They are never pointed at the case's working tree (/app), which holds untrusted base-revision content, so that tree cannot inject text into the doctrine block. A test runs the production step against a git tree that carries its own STALE-TREE-DOCTRINE aeg-root and asserts that text never reaches the prompt. A null or blank doctrine makes the trial unavailable before any step runs. The PR adds no dependency (node:crypto is a builtin used only for SHA-256), no auth surface, no SQL and no shell path. No findings.

Tokens: 1812: security — Security — claude — 6/2328/—
Cast by: Security (session 6a0994bf-af0e-4b16-8e3a-69e0c5d5b2e9)

@daniboomerang

Copy link
Copy Markdown
Member Author

No description provided.

@daniboomerang
daniboomerang merged commit c9087be into main Nov 2, 2026
22 of 24 checks passed
@daniboomerang
daniboomerang deleted the task/issue-1812 branch November 2, 2026 06:24
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

AEG provenance — task (backlog Issue #1812)

Tokens: 1812: archive — Archivist — — — —

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix(eval): Every role trial gives its agent the role doctrine the review loop gives it, and its record names that doctrine

1 participant