Skip to content

fix(webfetch): block SSRF to metadata, loopback and private ranges - #615

Merged
Temikus merged 21 commits into
mainfrom
fix/web-fetch-ssrf
Nov 18, 2026
Merged

Temikus merged 21 commits into
mainfrom
fix/web-fetch-ssrf

Conversation

@Temikus

@Temikus Temikus commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Purpose

web_fetch only checked the URL scheme, so it could reach cloud metadata, loopback (including Denkeeper's own API) and private ranges, directly or through a redirect. Found in the OWASP Agentic Top 10 review (ASI02).

Closes #589

What a reviewer needs to know

  • Checked at connect time. A new internal/netguard package refuses blocked addresses in the dialer, so redirects, robots.txt lookups and DNS rebinding are covered. A URL-string check runs first only for a clearer error.
  • Closed by default, two opt-ins. [web.fetch] allow_loopback and allow_private open their own ranges. Metadata (incl. Azure 168.63.129.16), link-local and the proxy's own address never open. The docs spell out how broad allow_private is.
  • Blocked URLs never reach Jina. The fallback used to fire on any error. A robots.txt or agents.txt disallow still falls back to Jina, as before.
  • Local-use NAT64 64:ff9b:1::/48 counts as private. A host whose DNS64 uses it needs allow_private to fetch IPv4-only sites.
  • Proxy kept. With HTTP(S)_PROXY set, the proxy must enforce egress. Documented once under "Network policy".
  • allowed_domains / blocked_domains are name-only. Bad entries (.evil.com, IP literals, unmappable hosts) fail at load.
  • Remote MCP is untouched. Its guard allows RFC1918 on purpose for LAN servers.
  • Browser allowlist now uses netguard, but has no production caller. The container's own check is Block SSRF to metadata, loopback, and private ranges denkeeper-browser#89.

Test plan

  • just hook
  • Tests use live httptest servers and the real dialer. The redirect, metadata, IDNA, proxy and default-config tests fail on main

🤖 Generated with Claude Code

https://claude.ai/code/session_01YGMCH6b8gjLdh3ZaC2QwRh


Summary by cubic

Blocks web_fetch, the browser allowlist, and the Jina fallback from reaching cloud metadata, loopback, and private ranges, directly or via redirects. The check runs on the IP each connection reaches at dial time, so redirects, robots/agents.txt lookups, and DNS rebinding are covered. Closes #589.

The guard

  • New internal/netguard package classifies addresses in the dialer; loopback and private ranges open only via [web.fetch] allow_loopback and allow_private.
  • Metadata (incl. Azure 168.63.129.16, AWS fd00:ec2::254, Alibaba 100.100.100.200), link-local, unspecified, multicast, and broadcast never open, even inside private ranges.
  • IPv6 forms carrying an IPv4 address (mapped, NAT64, 6to4) are judged by that IPv4; local-use NAT64 layouts are checked at every RFC 6052 position.

Behavior changes

  • A blocked URL never reaches Jina, and Jina's client is guarded too, checking every redirect hop by URL; refusals name the setting that would open the class.
  • allowed_domains and blocked_domains match the IDNA-mapped host; IP literals, numeric last labels, and malformed entries fail at config load. They cover the URL sent to Jina, not Jina's own redirect targets, and robots/agents.txt disallows still fall back to Jina.
  • The proxy dial is exempt, so the proxy must enforce egress; a direct request to the proxy's own address is refused, and NO_PROXY hosts go through the address check.
  • Empty-label and non-canonical numeric hosts are refused so a pure-Go resolver failure cannot forward them to Jina.
  • The browser allowlist uses the same classifier, IDNA-maps hosts and patterns, and refuses unmappable patterns; the remote-MCP guard is unchanged and still allows RFC1918.
  • Enabling an opt-in logs a single startup warning, and a reload that changes an opt-in logs a warning that running agents keep the old policy until restart.

Written for commit c9bd9d2. Summary will update on new commits.

View guided diff Turn on auto-fix

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 24 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread internal/webfetch/jina.go Resolved
Comment thread README.md Resolved
Comment thread internal/netguard/netguard.go Resolved
Comment thread internal/webfetch/guard.go Resolved
Comment thread cmd/denkeeper/main.go Resolved
Comment thread internal/config/config_test.go Resolved
Comment thread internal/browser/allowlist_test.go Resolved
Comment thread internal/config/config.go Resolved
Comment thread internal/browser/allowlist.go Resolved

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 5 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread internal/netguard/netguard_test.go Resolved
Temikus and others added 9 commits November 17, 2026 09:36
Add internal/netguard: an IP classifier and a guarded transport whose
dialer refuses link-local, metadata, unspecified, multicast and
broadcast addresses always, and loopback and private ranges unless the
policy opts in. IPv4-mapped, NAT64 and 6to4 forms are classified by
their embedded IPv4 address.

web_fetch now checks the first URL and every redirect (scheme, host,
allowed/blocked domains), runs on the guarded transport, and never
forwards a blocked URL to Jina. Refusals wrap netguard.ErrBlocked and
name the [web.fetch] key that opens the class, when one exists.

Refs #589

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YGMCH5yY44TFC5yeXSNicz
AWS's IPv6 metadata endpoint (fd00:ec2::254) sits in ULA and Alibaba's
(100.100.100.200) in CGNAT, so allow_private opened them. Classify both
as metadata ahead of the private ranges.

Refs #589

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YGMCH5yY44TFC5yeXSNicz
- Domain lists and host pre-checks compare the IDNA-mapped host, the
  name net/http actually dials, so fullwidth letters and U+3002 dots no
  longer slip past blocked_domains. A host that fails mapping is refused.
- A direct (non-proxied) request to the configured proxy's host:port is
  refused, so it cannot ride the proxy dial exemption.
- Jina's client uses the guarded transport, so its redirects are checked.
- Classify Azure WireServer 168.63.129.16 as metadata, and decode the
  local-use NAT64 prefix and IPv4-compatible addresses.

Refs #589

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YGMCH5yY44TFC5yeXSNicz
isBlockedHost now asks netguard's default policy, so RFC1918, CGNAT, ULA,
unspecified, multicast, broadcast, the metadata IPs and IPv6 forms carrying
a blocked IPv4 address are refused alongside localhost, loopback, link-local
and the metadata hostnames.

Refs #589

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YGMCH6rFHiCNxPrp9hwTG9
A browser maps hosts through IDNA and reads decimal, octal, hex and short
IPv4 forms as addresses, so http://2130706433/ and http://127.0.0.1/
reached loopback past the string check. Allowed now IDNA-maps the host
before both checks and refuses one that cannot be mapped. isBlockedHost
refuses a host whose last label is numeric (WHATWG "ends in a number") when
netip cannot parse it.

Refs #589

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YGMCH6rFHiCNxPrp9hwTG9
- allow_private, allow_loopback, allowed_domains and blocked_domains
  feed webfetch.Options; both opt-ins default off, and turning one on
  logs a Warn at startup.
- validateWeb rejects a domain entry that is empty, carries a scheme,
  path, port, whitespace or a misplaced wildcard, or does not map
  through IDNA, naming the key and entry.
- Wildcard patterns map only the part after "*.", so a non-ASCII
  wildcard like "*.evil.com" matches.
- Docs: the new keys, the never-open classes, and the proxy and NAT64
  caveats, stated once under [web.fetch] Network policy.

Refs #589

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YGMCH5yY44TFC5yeXSNicz
- Domain entries must be IDNA-mappable DNS labels; malformed ones
  (".evil.com", "user@evil.com", "evil..com", "evil_com", ...) no longer
  pass validation and then silently never match. IP literals are
  rejected with a pointer to allow_private / allow_loopback, a leading
  dot suggests "*.", and every rejection keeps the bare-host guidance.
- The opt-in Warn is logged once at startup; [web.fetch] is global.
- Docs say how broad allow_private is and stop steering home-lab users
  at it without that caveat.

Refs #589

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YGMCH5yY44TFC5yeXSNicz
Temikus and others added 3 commits November 18, 2026 10:46
Allowed maps the request host through IDNA, but patterns were only
lowercased, so a Unicode entry such as "пример.рф" or "*.пример.рф" no
longer matched its own host. NewAllowlist now maps each pattern's host
part (after any "*.") the same way and returns an error for a pattern it
cannot map. Dropping it instead could empty the list, which means
unrestricted.

The unmappable-host test now asserts the error from Allowed.

Refs #589

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YGMCH6rFHiCNxPrp9hwTG9
…ling dot

The pattern "*." kept the suffix ".", so it matched every host with a
trailing dot, and a browser loads evil.com. as evil.com. "*." plus U+00AD
mapped to the same thing, because IDNA drops U+00AD. normalizePattern now
refuses a pattern whose mapped host is empty or has an empty label.

Allowed trims one trailing dot from the request host, as webfetch does, so
github.com. matches a github.com entry.

Refs #589

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YGMCH6rFHiCNxPrp9hwTG9
- Jina checks every redirect hop by URL; the proxy dial exemption hid a
  proxied hop's target from the dial check.
- Refuse numeric hosts that aren't dotted-quad (netguard.EndsInNumber), so
  a pure-Go DNS failure can't send them to Jina.
- Treat local-use NAT64 64:ff9b:1::/48 as private (RFC 8215 leaves its
  layout to the site); never-open IPv4 in the /96 bits stays refused.
- Domain validator: IP checks after "*." and IDNA mapping, numeric last
  labels rejected, one trailing dot accepted like the matcher.
- Warn on reload when the [web.fetch] opt-ins change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YGMCH5yY44TFC5yeXSNicz

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread internal/browser/allowlist.go Resolved
Temikus and others added 6 commits November 18, 2026 10:58
A reload that changes the effective [web.fetch] opt-ins now logs one Warn
stating the mixed state: new agents use the new policy, running agents keep
the old one until restart. Closing access by reload was silent before, and
opening reused the startup text. Docs: [web] is no longer called
restart-only, and the NAT64 bullet says what a refusal looks like.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YGMCH5yY44TFC5yeXSNicz
requestGuard.check refuses hosts with an empty label, and EndsInNumber
trims every trailing dot, so 2852039166.. no longer reaches the resolver
or Jina. Local-use NAT64 addresses are checked for a never-open IPv4 at
every RFC 6052 position, so 64:ff9b:1:a9fe:a9:fe00:: stays refused under
allow_private.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YGMCH5yY44TFC5yeXSNicz
… docs

c1f017f picked up "internal/netguard/numeric 2.go", an iCloud sync copy
that redeclared EndsInNumber and broke the build. The built-in tools page
and the mcp-tools rule still called [web] restart-only; they now match the
config reference: running agents keep their settings until restart, agents
created after a reload use the new values.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YGMCH5yY44TFC5yeXSNicz
The every-position check refuses a public IPv4 whose bytes read as a
never-allowed class at another RFC 6052 position, so a non-/96 local-use
layout can refuse some public addresses (e.g. 8.8.224.1 in the /48 layout).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YGMCH5yY44TFC5yeXSNicz
Drop the browser's local endsInNumber for the shared helper. The local copy
trimmed one trailing dot, so a numeric host with more, such as 127.1.. in
isBlockedHost or 2130706433... through Allowed, was not refused. The netguard
helper trims every trailing dot.

Refs #589

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YGMCH6rFHiCNxPrp9hwTG9
The browser Allowlist trimmed one trailing dot, and CheckHost trims one
more, so localhost..., 127.0.0.1..., 169.254.169.254... and
metadata.google.internal... were allowed. Move webfetch's empty-label rule
into netguard.HasEmptyLabel and call it from both requestGuard.check and
the browser's isBlockedHost.

Refs #589

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YGMCH6rFHiCNxPrp9hwTG9

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 16 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread internal/webfetch/jina.go Resolved
Comment thread .claude/rules/mcp-tools.md Resolved
Temikus and others added 2 commits November 18, 2026 11:28
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Temikus
Temikus merged commit 3d31e4b into main Nov 18, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

web_fetch: block SSRF to metadata, loopback, and private ranges

1 participant