Skip to content

feat(tool): split media reading out of read into read_image/read_audio/read_video - #2828

Merged
yanyihan-xiaomi merged 5 commits into
mainfrom
feat/read-media-split
Nov 18, 2026
Merged

yanyihan-xiaomi merged 5 commits into
mainfrom
feat/read-media-split

Conversation

@yanyihan-xiaomi

Copy link
Copy Markdown
Collaborator

Issue / context (if applicable)

Spec: docs/compose/spec/read-media-split.md

Type of change

New feature / Refactor

What does this PR do?

read handled text, directories, images, PDFs, audio and video in one tool, and it advertised media support through a description paragraph that changed with the model. A model without the right capability only found out after calling read on a media file.

This PR splits media into three dedicated tools:

  • New tools: read_image, read_audio and read_video, each taking a single file_path.
    • The registry offers a tool only when the turn's model, through its adapter, accepts that media kind. Adapters with an unknown verdict and unresolvable models are excluded.
    • Each tool re-checks this when it runs.
    • Codex mode is unchanged.
  • read: now reads text and directories only. Media fails with Cannot read binary file, and describeMedia is removed.
  • PDF: no PDF tool. The bundled pdf skill covers it.
  • @-mentions: these arrive as text/plain, so their type is now sniffed:
    • Images, audio and video go to the matching media tool.
    • A PDF is inlined only for a model with PDF input; otherwise it stays a path reference and its bytes are not stored.
  • Wiring: the read permission governs the new tools, and they are also wired into parallel-readonly gating, dream/distill allowlists, compaction, the system-prompt vision hint, TUI/run rendering and ACP kinds.

Behavior change: audio and video behind an adapter with no capability declaration (e.g. @ai-sdk/openai) are no longer offered. This fails closed. @ai-sdk/openai-compatible is unaffected.

How did you verify your code works?

  • bun run typecheck (packages/cli): pass.
  • bun test --timeout 30000 test/tool/ test/permission/ test/session/system.test.ts test/session/compaction.test.ts test/session/prompt.test.ts test/agent/agent.test.ts test/acp: 1010 pass, 0 fail.
  • New tests:
    • test/tool/read-media.test.ts
    • registry visibility matrix in registry-toolset.test.ts
    • read media refusals in read.test.ts
    • @-mention routing in prompt.test.ts (this one fails on the pre-fix head)
  • Full suite: the remaining failures are MCP lifecycle and Project/Vcs/Worktree git tests. They also fail on main in this environment.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

@yanyihan-xiaomi
yanyihan-xiaomi merged commit b24e577 into main Nov 18, 2026
6 checks passed
@yanyihan-xiaomi
yanyihan-xiaomi deleted the feat/read-media-split branch November 18, 2026 07:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant