[developer-review-engine-evals-v1] 2 — A Harbor trial runs Codex by subscription login, next to Claude Code #1652

Activity

1. daniboomerang

   added

   vinaya/tranche:developer-review-engine-evals-v1Tranche: developer-review-engine-evals-v1Tranche: developer-review-engine-evals-v1

   on Oct 25, 2026
2. daniboomerang

   added this to the Productionize developer-review on Atta Engine milestone on Oct 25, 2026
3. daniboomerang commented on Oct 28, 2026

   

   daniboomerang

   on Oct 28, 2026

   MemberAuthor

   More actions

   Brief hash: d8c1037cf323d2f0ab44e4f932f6c8729d7bc2d18602ec685439f377646f9f33
   For: [model] (coding-agent CLI on a dev machine, dispatched locally, unattended)
   Reason: Suggested agent-class — high: credential staging across a container boundary, where a wrong variable leaks a token or breaks login.
   Owner: the Principal
   Goal: A Harbor trial runs Codex by subscription login, next to Claude Code
   Project: cli
   Tier: 1

   Closes #1652

   You are the AEG Developer. Run bun apps/cli/src/index.ts doctrine --role developer --print and read its output first. Mandatory.

   Objectives

   O1. A Harbor task image installs the Codex CLI at the same pinned version the evaluation already grades with, next to Claude Code, and the image build checks the installed version; a test of the converted task's image definition asserts both installs and both version checks.
   O2. A Harbor trial run with the Codex adapter logs in from the operator's Codex subscription: the harbor run process's environment carries the token in CODEX_ACCESS_TOKEN, which the operator fills from the host Codex login the way resolveCodexAccessToken reads it; the Python agent reads it only from its own environment, passes it only on standard input to codex login --with-access-token inside the container, which stages it into the container's own Codex home, and starts the agent process with that Codex home and no access-token variable; a test with a fake Codex binary asserts the login command, its standard input and the agent's environment.
   O3. A Codex trial refuses to start, with a reason naming the missing login, when no subscription access token is found; no API-key variable is read or forwarded on either adapter; a test asserts the refusal and that no variable ending in API_KEY reaches the container.
   O4. The Harbor agent passes the requested adapter to the trial instead of always Claude Code, the trial accepts claude-code and codex, and the network allowlist for a Codex task names the Codex subscription hosts, starting from api.openai.com and chatgpt.com, while a Claude Code task keeps api.anthropic.com only; tests assert the agent command line and both allowlists.
   O5. The Claude Code path is unchanged: the existing Harbor task, practice-fix and agent tests pass without edits to their Claude expectations.

   Documentation

   • apps/cli/specs/evals.md — the "Layer-G role trials in Harbor" section, the allowed hosts and the forwarded login variables (O1, O2, O4, O5)
   • apps/cli/src/lib/worker-boundary.ts — resolveCodexAccessToken and the Codex login-with-access-token step this task reuses (O2, O3)
   • apps/cli/src/lib/dispatch.ts — why a staged Codex home alone authenticates and an access-token variable on the same process breaks it (O2)
   • apps/cli/src/lib/dev-review-eval/vinaya_harbor_agent.py — the agent's login forwarding and trial command (O2, O3, O4)

   Premises

   Checked against the code when this task Issue was cut, and again against the default branch at dispatch — facts, not claims.

   • apps/cli/src/lib/dev-review-eval/harbor-trial.ts contains export const HARBOR_TRIAL_ADAPTERS = ['claude-code'] as const
   • apps/cli/src/lib/dev-review-eval/vinaya_harbor_agent.py contains "--backend standalone --adapter claude-code",
   • apps/cli/src/lib/dev-review-eval/vinaya_harbor_agent.py contains SUBSCRIPTION_ENV = ("CLAUDE_FORCE_OAUTH", "CLAUDE_CODE_OAUTH_TOKEN")
   • apps/cli/src/lib/dev-review-eval/harbor-tasks.ts contains export const AGENT_ALLOWED_HOSTS = ['api.anthropic.com'] as const
   • apps/cli/src/lib/dev-review-eval/step-one.ts contains export const GRADING_CODEX_PACKAGE = '@openai/codex@0.160.0'
   • apps/cli/src/lib/worker-boundary.ts contains export function resolveCodexAccessToken(

   2. Context — read before doing anything

   • Tranche: developer-review-engine-evals-v1, task 2, Issue [developer-review-engine-evals-v1] 2 — A Harbor trial runs Codex by subscription login, next to Claude Code #1652. Branch task/developer-review-engine-evals-v1/2. Depends-on: —, Conflicts-with: #1643, #1651, #1653, #1654, #1655, #1656.
   • Read Issue [developer-review-engine-evals-v1] 2 — A Harbor trial runs Codex by subscription login, next to Claude Code #1652 in full for the complete rationale — do not re-derive it.
   • Revision: rendered at b9c63aae32df154af0c90bda2fb9e0b2e1d8352c — the checkout's HEAD equaled the remote default branch, and no pinned file below carried an uncommitted change, when these facts were read.
   • Boundary — Let a Harbor trial run the Vinaya role on Codex by subscription login, with the same task, trial, grader and report contracts. It does NOT add the Engine backend, read results, or change how Claude Code logs in. Pinned files: apps/cli/src/lib/dev-review-eval/harbor-tasks.ts, apps/cli/src/lib/dev-review-eval/harbor-trial.ts, apps/cli/src/lib/dev-review-eval/vinaya_harbor_agent.py, apps/cli/tests/eval/harbor-tasks.test.ts, apps/cli/tests/eval/harbor-practice-fixes.test.ts, apps/cli/specs/evals.md, .changeset/harbor-codex-subscription.md. Out: the review loop's worker boundary and dispatch code, which are read, not edited; the results reader; step one's own adapter list, which already holds Codex.

   reader-resolvable-prose: code comments, READMEs and specs carry no Issue or pull-request numbers and no tranche slugs.

   doc-claims: every AEG:CLAIM line in a changed file is kept true in the same commit, the allowed-hosts claim included.

   • Traps to avoid — Do NOT read or forward any API key; subscription only, on both adapters. Do NOT set the Codex access-token variable on the agent process; stage the Codex home and pass only that. Do NOT pass the token on the command line; feed it to the login command's standard input, as the review loop does. Do NOT widen Claude's allowlist; the allowlist is per adapter. Do NOT invent a Codex version; reuse the pinned grading version. Keep the AGENT_ALLOWED_HOSTS AEG:CLAIM in apps/cli/specs/evals.md true. The install hook runs as root while the trial runs as the image's user tester with its home at /home/tester; the staged Codex home must be owned and writable by tester, since the Codex CLI writes its sessions there. A new test file needs one line in an apps/cli/tests/ci-shards/shard-N.txt.

   3. Technical dependencies

   Dependency rationale — Conflicts-with: #1643, #1651, #1653, #1654, #1655, #1656 — every one of these also edits the evaluation library directory, which the directory-overlap gate serializes within one milestone; the reasons for the file-level edges follow. It conflicts with the source-citation held-out task, which also pins harbor-tasks.ts, harbor-tasks.test.ts and apps/cli/specs/evals.md. No file-level conflict with the results-reader task: they share only apps/cli/specs/evals.md. The Engine-backend task of this tranche depends on this one: both pin harbor-trial.ts, vinaya_harbor_agent.py, harbor-tasks.test.ts and harbor-practice-fixes.test.ts.

   4. Technical surface map

   Create:

   • (none — every surface file already exists)

   Modify:

   • apps/cli/src/lib/dev-review-eval
   • apps/cli/tests/eval
   • apps/cli/specs
   • apps/cli/tests/ci-shards
   • .changeset

   Out of surface: harness, processes, packages, apps/log-server — a change there, or to any path no Surface in: glob covers, is still published: the review-loop driver records each such path in the pull request body's beyond-Surface block, and the reviewers judge it like any change.

   Premise pins

   Premise:

   • apps/cli/specs/evals.md sha256: 65cc638c8d226a7a509152f7a9d60e3f1c0fd78f56db9a06ada32f60f1fa0653
   • apps/cli/src/lib/dev-review-eval/harbor-tasks.ts sha256: 913dcbfc1b27377b7cb8b187e5e11131c9532b2ed43ba54e1221306317b011e7
   • apps/cli/src/lib/dev-review-eval/harbor-trial.ts sha256: 6d43b78cea7c7bf449bfbf113ed5afb04a623ac78a475b914a3160782430ded0
   • apps/cli/src/lib/dev-review-eval/vinaya_harbor_agent.py sha256: 0f2cbad699a7cbcf6e18d1579106d74a551a25d6a421abdbc904270c177f373d
   • apps/cli/tests/eval/harbor-practice-fixes.test.ts sha256: 1259491f96b75cd26b4453548805b4a1ac74268eb0a77b59817d6ba267f6ba91
   • apps/cli/tests/eval/harbor-tasks.test.ts sha256: 38630bcf0015ec4f6156eb35ace066389eff4b895ed86eed7d421c40e6799f26

   5. Pre-flight checks

   Step 0 (mandatory, verbatim):

   bun install --frozen-lockfile --silent
   

   1. Clean status; parent origin/main; branch suffix literal-matches the task id.
   2. Under the review loop (bun apps/cli/src/index.ts task run), the driver has already run the dispatch-readiness gate plus, on this repository toolchain, its own fuller derivation from its own unsandboxed process before dispatching you (re-derived at render time: it was ready), and dispatches you only when that verdict is READY — a NOT READY verdict refuses the turn before it starts, so being dispatched is the verdict: look for no file and run neither script, whose own gh call fails inside your sandbox. Working manually, with no driver behind you, fall back to roles/developer.md's own items 3/5/7 prose — each a complete, hand-run forge check in its own right.

   On any failure: STOP and report.

   6. Numbered parts — publish once with publish_changes when every Part is done, then open_pull_request

   1. Part 1 (O1): Task images install and check the pinned Codex CLI beside Claude Code.

      Files (touches @attalabs/vinaya):

      • apps/cli/specs/evals.md
      • apps/cli/src/lib/dev-review-eval/harbor-tasks.ts
      • apps/cli/src/lib/dev-review-eval/harbor-trial.ts
      • apps/cli/src/lib/dev-review-eval/vinaya_harbor_agent.py
      • apps/cli/tests/eval/harbor-practice-fixes.test.ts
      • apps/cli/tests/eval/harbor-tasks.test.ts

      The pre-push hook runs the affected suite on your one push and refuses it on failure — do not run it yourself per Part.

   2. Part 2 (O2, O3): The Codex subscription login is staged into the container's Codex home, and a missing login refuses.

   3. Part 3 (O4): The agent passes the adapter; the trial accepts both; the allowlist follows the adapter.

   4. Part 4 (O5): The Claude Code expectations stay unchanged.

   7. Documentation-update list

   • apps/cli/specs/evals.md

   8. Verification before claiming done

   • Under the review loop, the controller runs the static gates: the commit hook applies safe format and lint fixes to staged files, typechecks affected packages, and runs the Vinaya checks; the pre-push hook runs the affected suite; run_checks and CI provide the authoritative results. Do not hand-run or paste static-gate commands. Working manually, run this repository's declared typecheck, lint/format, build, and documentation-gate commands yourself.
   • The pre-push hook runs the affected suite once, on your one push, and refuses it on failure; CI runs the full suite.
   • Every blast-radius consumer named in §4, re-verified by name.
   • The tier checklist in bun apps/cli/src/index.ts doctrine --role developer --print genuinely satisfied; use the controller's run_checks result, hooks, and CI as gate evidence.

   9. Test Plan

   cd apps/cli && bun test --timeout=30000 tests/eval/harbor-tasks.test.ts → 0 fail; image installs both CLIs; Codex allowlist and Claude allowlist asserted; agent command carries the adapter
   cd apps/cli && bun test --timeout=30000 tests/eval/harbor-practice-fixes.test.ts → 0 fail; fake Codex login gets the token on standard input; agent env has the Codex home and no token or API-key variable
   cd apps/cli && bun test --timeout=30000 tests/ci-shards.test.ts → 0 fail
   

   10. Stop conditions

   • Staging the Codex login needs a change to the review loop's worker boundary or dispatch code.

   Stop-and-escalate — Stop if staging the Codex login needs a change to the review loop's worker boundary or dispatch code; name the function. Run the login in the agent's existing install hook, which already runs commands in the container as root before the trial starts.

   11. Constraints

   Autonomy: Do not stop to ask clarifying questions. For any ambiguity not covered by a Section 10 stop condition, choose the most reasonable option consistent with this brief, record the choice in the PR body at open, or in a PR comment after open, and continue. Halt only for the explicit Section 10 stop conditions — and when you halt, record the blocker in a PR comment or an Issue comment rather than waiting interactively for input.

   12. Deliverable

   • PR title (exact): [developer-review-engine-evals-v1] 2 — A Harbor trial runs Codex by subscription login, next to Claude Code
   • Open the PR only via the open_pull_request tool (title and body); publish your commits with publish_changes and update the body with update_pull_request_body/refresh_evidence — you hold no gh/git push credential.
   • Publish once, with publish_changes, when every Part is done; then open the pull request with open_pull_request. Publish again only to answer review findings or a red gate — a refused hook, a failing check, a merge conflict.
   • PR body = the Developer's PR report (print it with bun apps/cli/src/index.ts doctrine --template pr-report --print), with Closes #1652 at the top of the header block. This brief stays frozen on the task Issue and never rides in the body.
   • Pre-open: tier checklist satisfied. The documentation gate is the controller's — run_checks and CI run it.
   • Include git diff origin/main...HEAD --stat as context only; Surface is judged by the driver's surface-scope check, never by this hand-run diff. Token use needs no report — the Vinaya log records it.
   • Then STOP. Review and Verification are separate invocations.
4. daniboomerang

   self-assigned this

   on Oct 28, 2026
5. daniboomerang commented on Oct 28, 2026

   

   daniboomerang

   on Oct 28, 2026

   MemberAuthor

   More actions

   The dev-review-loop paused: escalation — the Developer reported the task blocked (tooling_unavailable): The enforced addressedFindingIds enum permits only R1-CR-1, so it cannot report the required R2-CR-1; the controller must update the response schema. — Fixed R2-CR-1 in commit 8572d15; publication and controller checks passed, but the ….

   No pull request exists yet for this task, so the pause is recorded on this Issue instead.
   A Principal ruling is needed before this can continue. Post it as a comment on this Issue whose FIRST line is exactly this marker — the second number is the ordinal, and it must exceed the one this pause recorded:

   <!-- aeg:principal:ruling:1652-1 -->
   Your ruling text here.
   

   Then continue with:

   vinaya task run developer-review-engine-evals-v1 2 --agent codex --model gpt-6.1-sol --runner atta-engine
   

   The Operator's own task_resume clears this pause too, reading that same ruling from this Issue; task_cancel ends it instead.
6. daniboomerang commented on Oct 28, 2026

   

   daniboomerang

   on Oct 28, 2026

   MemberAuthor

   More actions

   Decision: proceed. Report this turn as completed.

   The round-2 code review raised R2-CR-1 (the image lacks the Codex sandbox dependency), and commit 8572d15 fixes it. This round's turn-result schema still offers only R1-CR-1 for addressedFindingIds: the controller did not refresh the finding list after round 2, which is a controller defect, not yours. Fill addressedFindingIds with the id the schema offers, and state in summary that the change addresses R2-CR-1 at 8572d15. The reviewers judge the new head in full. No objective, Surface or scope change.
7. daniboomerang

   closed this as completedin #1715on Oct 28, 2026