[developer-review-engine-evals-v1] 3 — A Harbor change or review trial runs on the Engine backend, on both adapters #1653

Activity

1. daniboomerang

   added

   vinaya/tranche:developer-review-engine-evals-v1Tranche: developer-review-engine-evals-v1Tranche: developer-review-engine-evals-v1

   on Oct 25, 2026
2. daniboomerang

   added this to the Productionize developer-review on Atta Engine milestone on Oct 25, 2026
3. daniboomerang commented on Oct 28, 2026

   

   daniboomerang

   on Oct 28, 2026

   MemberAuthor

   More actions

   Brief hash: fbb470551cdf3cf557b226b87d5ede4ddefd9a76e543ce28f2c6cf7ca7b06ffd
   For: [model] (coding-agent CLI on a dev machine, dispatched locally, unattended)
   Reason: Suggested agent-class — high: wiring a role step onto the Engine runtime inside the trial's isolation, across two adapters.
   Owner: the Principal
   Goal: A Harbor change or review trial runs on the Engine backend, on both adapters
   Project: cli
   Tier: 1

   Closes #1653

   You are the AEG Developer. Run bun apps/cli/src/index.ts doctrine --role developer --print and read its output first. Mandatory.

   Objectives

   O1. A Harbor change trial and a Harbor review trial accept --backend engine and run the role's step on the Atta Engine runtime, through the public process-run start with role agents launched by the Engine's role launcher, with no forge access; --backend standalone keeps today's path; a test with stub agents runs one change case and one review case on each backend and asserts both produce a trial record the generic graders accept.
   O2. On the Engine backend both adapters run: a test with stub agents runs the same change case on Claude Code and on Codex and asserts both records grade with the same verdict.
   O3. The trial record names the backend it ran on; HARBOR_TRIAL_BACKENDS accepts standalone and engine, while runStepOne keeps refusing a non-standalone backend with its existing engine_forbidden reason; tests assert both.
   O4. The Harbor agent passes the requested backend to the trial instead of always standalone; a test asserts the agent's command line for each backend.
   O5. A loop case is still refused on both backends with its existing reason; a test asserts it.

   Documentation

   • apps/cli/specs/evals.md — the "Layer-G role trials in Harbor" section and the backend wording it carries (O1, O3, O4)
   • apps/cli/specs/dev-review-evaluation-program.md — §6.17 Engine run-control boundary and §8 held-out order, which this trial backend serves (O1, O3)
   • harness/dev-process/engine-runtime/README.md — what the generic process runner takes and refuses (O1, O2)
   • harness/dev-process/engine-runtime/run-process.ts — startProcessRun and createRoleLauncher (O1, O2)
   • apps/cli/src/lib/dev-review-eval/step-one.ts — VinayaProcessSteps, vinayaProcess and the Engine refusal (O1, O3, O5)

   Premises

   Checked against the code when this task Issue was cut, and again against the default branch at dispatch — facts, not claims.

   • apps/cli/src/lib/dev-review-eval/harbor-trial.ts contains export const HARBOR_TRIAL_BACKENDS = ['standalone'] as const
   • apps/cli/src/lib/dev-review-eval/step-one.ts contains return refusal('engine_forbidden', 'step one runs only on the standalone backend')
   • apps/cli/src/lib/dev-review-eval/step-one.ts contains export interface VinayaProcessSteps {
   • harness/dev-process/engine-runtime/run-process.ts contains export async function startProcessRun(options: ProcessRunOptions, identity?: RunIdentity): Promise<RunOutcome> {
   • harness/dev-process/engine-runtime/run-process.ts contains export function createRoleLauncher(

   2. Context — read before doing anything

   • Tranche: developer-review-engine-evals-v1, task 3, Issue [developer-review-engine-evals-v1] 3 — A Harbor change or review trial runs on the Engine backend, on both adapters #1653. Branch task/developer-review-engine-evals-v1/3. Depends-on: #1652, Conflicts-with: #1643, #1651, #1652, #1654, #1655, #1656.
   • Read Issue [developer-review-engine-evals-v1] 3 — A Harbor change or review trial runs on the Engine backend, on both adapters #1653 in full for the complete rationale — do not re-derive it.
   • Revision: rendered at 3c50cb1cafa26bf0f3c0dfd8543e3d8f55181062 — the checkout's HEAD equaled the remote default branch, and no pinned file below carried an uncommitted change, when these facts were read.
   • Boundary — Give a Harbor role trial an Engine backend, so the same neutral change and review cases run on standalone and on the Engine with the same task, trial, grader and report contracts. It does NOT run the whole loop, read or record results, add Codex login (done by the task it depends on), or change any Engine Flow or runtime file. Pinned files: apps/cli/src/lib/dev-review-eval/harbor-trial.ts, apps/cli/src/lib/dev-review-eval/step-one.ts, apps/cli/src/lib/dev-review-eval/vinaya_harbor_agent.py, apps/cli/tests/eval/harbor-tasks.test.ts, apps/cli/tests/eval/harbor-practice-fixes.test.ts, apps/cli/tests/eval/step-one.test.ts, apps/cli/specs/evals.md, .changeset/harbor-engine-backend.md. Out: every file under harness and processes is read, not edited; the results reader; the freeze record.

   reader-resolvable-prose: code comments, READMEs and specs carry no Issue or pull-request numbers and no tranche slugs.

   doc-claims: every AEG:CLAIM line in a changed file is kept true in the same commit, the vinayaProcess and runHarborTrial claims included.

   • Traps to avoid — Do NOT run the whole dev-review Flow for a role trial; build a one-step Flow for the role's agent step. Do NOT give the Engine trial forge access, a real checkpoint store outside the trial's scratch directory, or any credential beyond what the standalone trial gets. Do NOT lift the Engine refusal for host-side step-one runs. Do NOT edit files under harness or processes; if the runner cannot take a one-step Flow without an edit, stop. A new test file needs one line in an apps/cli/tests/ci-shards/shard-N.txt. A Codex role launch generated by the Engine's adapter requires its vinaya_dev_tools MCP server (required = true in the Codex adapter config); the Engine trial in the container starts the same dev-tools host the standalone trial starts (startToolHost) and points the generated config at it, or the Codex launch refuses to start. New test fixtures use fixtureDir from apps/cli/tests/lib/test-temp-fixture.ts, as harbor-tasks.test.ts and step-one.test.ts do, never mkdtempSync(join(tmpdir(), …)).

   3. Technical dependencies

   Dependency rationale — Conflicts-with: #1643, #1651, #1652, #1654, #1655, #1656 — every one of these also edits the evaluation library directory, which the directory-overlap gate serializes within one milestone; the reasons for the file-level edges follow. Depends-on: #1652 and conflicts with #1652 — the Codex-subscription Harbor task: both pin harbor-trial.ts, vinaya_harbor_agent.py, apps/cli/tests/eval/harbor-tasks.test.ts and apps/cli/tests/eval/harbor-practice-fixes.test.ts, and O2 runs Codex trials that task makes possible. No Conflicts-with with the results-reader task: they share only apps/cli/specs/evals.md.

   4. Technical surface map

   Create:

   • (none — every surface file already exists)

   Modify:

   • apps/cli/src/lib/dev-review-eval
   • apps/cli/tests/eval
   • apps/cli/specs
   • apps/cli/tests/ci-shards
   • .changeset

   Out of surface: harness, processes, packages, apps/log-server — a change there, or to any path no Surface in: glob covers, is still published: the review-loop driver records each such path in the pull request body's beyond-Surface block, and the reviewers judge it like any change.

   Premise pins

   Premise:

   • apps/cli/specs/evals.md sha256: eaef6a004854e34de7076d1ded575697a7a2f1b4d36adc32af23d13adc470f08
   • apps/cli/src/lib/dev-review-eval/harbor-trial.ts sha256: 91ade7bd6bd078dee8878d78e9edaa2a421cd3a46081fdeb1826fd9330d78054
   • apps/cli/src/lib/dev-review-eval/step-one.ts sha256: 81012de4f069d29e52426d1bb92a2f523e00755ebdb241a9d382f7529cc5edcd
   • apps/cli/src/lib/dev-review-eval/vinaya_harbor_agent.py sha256: 59b92301a94db510915f72d6088d6752d4d8930bf7a3c1197e4a990b1f7ddacd
   • apps/cli/tests/eval/harbor-practice-fixes.test.ts sha256: cc5a3ce6269fb0783e5c9f95775fee31f3fa7a03b7a87ed7e9b9976ca371d002
   • apps/cli/tests/eval/harbor-tasks.test.ts sha256: 58b14109bd11de3728a9b529f39fb12196df3f1accc9389ef6b8e9c2b46e842a
   • apps/cli/tests/eval/step-one.test.ts sha256: 2c4a2feb3b024afbf594b584e0be4a0cdeac865d6e1541ff0bf38e647c48d8d1

   5. Pre-flight checks

   Step 0 (mandatory, verbatim):

   bun install --frozen-lockfile --silent
   

   1. Clean status; parent origin/main; branch suffix literal-matches the task id.
   2. Under the review loop (bun apps/cli/src/index.ts task run), the driver has already run the dispatch-readiness gate plus, on this repository toolchain, its own fuller derivation from its own unsandboxed process before dispatching you (re-derived at render time: it was ready), and dispatches you only when that verdict is READY — a NOT READY verdict refuses the turn before it starts, so being dispatched is the verdict: look for no file and run neither script, whose own gh call fails inside your sandbox. Working manually, with no driver behind you, fall back to roles/developer.md's own items 3/5/7 prose — each a complete, hand-run forge check in its own right.

   On any failure: STOP and report.

   6. Numbered parts — publish once with publish_changes when every Part is done, then open_pull_request

   1. Part 1 (O1): An Engine dispatch variant of the role step, built as a one-step Flow on the public process run.

      Files (touches @attalabs/vinaya):

      • apps/cli/specs/evals.md
      • apps/cli/src/lib/dev-review-eval/harbor-trial.ts
      • apps/cli/src/lib/dev-review-eval/step-one.ts
      • apps/cli/src/lib/dev-review-eval/vinaya_harbor_agent.py
      • apps/cli/tests/eval/harbor-practice-fixes.test.ts
      • apps/cli/tests/eval/harbor-tasks.test.ts
      • apps/cli/tests/eval/step-one.test.ts

      The pre-push hook runs the affected suite on your one push and refuses it on failure — do not run it yourself per Part.

   2. Part 2 (O2): The Engine role step runs on both adapters.

   3. Part 3 (O3, O5): The trial record names its backend; the Engine refusal is lifted for Harbor container trials only; loop cases stay refused.

   4. Part 4 (O4): The Harbor agent passes the requested backend.

   7. Documentation-update list

   • apps/cli/specs/evals.md

   8. Verification before claiming done

   • Under the review loop, the controller runs the static gates: the commit hook applies safe format and lint fixes to staged files, typechecks affected packages, and runs the Vinaya checks; the pre-push hook runs the affected suite; run_checks and CI provide the authoritative results. Do not hand-run or paste static-gate commands. Working manually, run this repository's declared typecheck, lint/format, build, and documentation-gate commands yourself.
   • The pre-push hook runs the affected suite once, on your one push, and refuses it on failure; CI runs the full suite.
   • Every blast-radius consumer named in §4, re-verified by name.
   • The tier checklist in bun apps/cli/src/index.ts doctrine --role developer --print genuinely satisfied; use the controller's run_checks result, hooks, and CI as gate evidence.

   9. Test Plan

   cd apps/cli && bun test --timeout=30000 tests/eval/step-one.test.ts → 0 fail; host-side Engine run refused; loop case refused on both backends
   cd apps/cli && bun test --timeout=30000 tests/eval/harbor-tasks.test.ts tests/eval/harbor-practice-fixes.test.ts → 0 fail; change and review cases grade on both backends with stub agents, on both adapters; agent command carries the backend
   cd apps/cli && bun test --timeout=30000 tests/ci-shards.test.ts → 0 fail
   

   10. Stop conditions

   • Running a one-step Flow needs an edit under harness or processes.
   • The Engine trial needs a credential or network host the standalone trial does not have.

   Stop-and-escalate — Stop if running a one-step Flow needs an edit under harness or processes; name the file and the missing option. Stop if the Engine trial needs a credential or network host the standalone trial does not have; name it.

   11. Constraints

   Autonomy: Do not stop to ask clarifying questions. For any ambiguity not covered by a Section 10 stop condition, choose the most reasonable option consistent with this brief, record the choice in the PR body at open, or in a PR comment after open, and continue. Halt only for the explicit Section 10 stop conditions — and when you halt, record the blocker in a PR comment or an Issue comment rather than waiting interactively for input.

   12. Deliverable

   • PR title (exact): [developer-review-engine-evals-v1] 3 — A Harbor change or review trial runs on the Engine backend, on both adapters
   • Open the PR only via the open_pull_request tool (title and body); publish your commits with publish_changes and update the body with update_pull_request_body/refresh_evidence — you hold no gh/git push credential.
   • Publish once, with publish_changes, when every Part is done; then open the pull request with open_pull_request. Publish again only to answer review findings or a red gate — a refused hook, a failing check, a merge conflict.
   • PR body = the Developer's PR report (print it with bun apps/cli/src/index.ts doctrine --template pr-report --print), with Closes #1653 at the top of the header block. This brief stays frozen on the task Issue and never rides in the body.
   • Pre-open: tier checklist satisfied. The documentation gate is the controller's — run_checks and CI run it.
   • Include git diff origin/main...HEAD --stat as context only; Surface is judged by the driver's surface-scope check, never by this hand-run diff. Token use needs no report — the Vinaya log records it.
   • Then STOP. Review and Verification are separate invocations.
4. daniboomerang

   self-assigned this

   on Oct 28, 2026
5. daniboomerang

   closed this as completedin #1716on Oct 28, 2026