Repository navigation
fix(webfetch): block SSRF to metadata, loopback and private ranges - #615
Conversation
There was a problem hiding this comment.
All reported issues were addressed across 24 files
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
There was a problem hiding this comment.
All reported issues were addressed across 5 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
a181551 to
ba22618
Compare
Add internal/netguard: an IP classifier and a guarded transport whose dialer refuses link-local, metadata, unspecified, multicast and broadcast addresses always, and loopback and private ranges unless the policy opts in. IPv4-mapped, NAT64 and 6to4 forms are classified by their embedded IPv4 address. web_fetch now checks the first URL and every redirect (scheme, host, allowed/blocked domains), runs on the guarded transport, and never forwards a blocked URL to Jina. Refusals wrap netguard.ErrBlocked and name the [web.fetch] key that opens the class, when one exists. Refs #589 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGMCH5yY44TFC5yeXSNicz
AWS's IPv6 metadata endpoint (fd00:ec2::254) sits in ULA and Alibaba's (100.100.100.200) in CGNAT, so allow_private opened them. Classify both as metadata ahead of the private ranges. Refs #589 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGMCH5yY44TFC5yeXSNicz
- Domain lists and host pre-checks compare the IDNA-mapped host, the name net/http actually dials, so fullwidth letters and U+3002 dots no longer slip past blocked_domains. A host that fails mapping is refused. - A direct (non-proxied) request to the configured proxy's host:port is refused, so it cannot ride the proxy dial exemption. - Jina's client uses the guarded transport, so its redirects are checked. - Classify Azure WireServer 168.63.129.16 as metadata, and decode the local-use NAT64 prefix and IPv4-compatible addresses. Refs #589 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGMCH5yY44TFC5yeXSNicz
isBlockedHost now asks netguard's default policy, so RFC1918, CGNAT, ULA, unspecified, multicast, broadcast, the metadata IPs and IPv6 forms carrying a blocked IPv4 address are refused alongside localhost, loopback, link-local and the metadata hostnames. Refs #589 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGMCH6rFHiCNxPrp9hwTG9
A browser maps hosts through IDNA and reads decimal, octal, hex and short IPv4 forms as addresses, so http://2130706433/ and http://127.0.0.1/ reached loopback past the string check. Allowed now IDNA-maps the host before both checks and refuses one that cannot be mapped. isBlockedHost refuses a host whose last label is numeric (WHATWG "ends in a number") when netip cannot parse it. Refs #589 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGMCH6rFHiCNxPrp9hwTG9
- allow_private, allow_loopback, allowed_domains and blocked_domains feed webfetch.Options; both opt-ins default off, and turning one on logs a Warn at startup. - validateWeb rejects a domain entry that is empty, carries a scheme, path, port, whitespace or a misplaced wildcard, or does not map through IDNA, naming the key and entry. - Wildcard patterns map only the part after "*.", so a non-ASCII wildcard like "*.evil.com" matches. - Docs: the new keys, the never-open classes, and the proxy and NAT64 caveats, stated once under [web.fetch] Network policy. Refs #589 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGMCH5yY44TFC5yeXSNicz
- Domain entries must be IDNA-mappable DNS labels; malformed ones
(".evil.com", "user@evil.com", "evil..com", "evil_com", ...) no longer
pass validation and then silently never match. IP literals are
rejected with a pointer to allow_private / allow_loopback, a leading
dot suggests "*.", and every rejection keeps the bare-host guidance.
- The opt-in Warn is logged once at startup; [web.fetch] is global.
- Docs say how broad allow_private is and stop steering home-lab users
at it without that caveat.
Refs #589
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YGMCH5yY44TFC5yeXSNicz
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGMCH6b8gjLdh3ZaC2QwRh
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGMCH6b8gjLdh3ZaC2QwRh
ba22618 to
0ddea04
Compare
Allowed maps the request host through IDNA, but patterns were only lowercased, so a Unicode entry such as "пример.рф" or "*.пример.рф" no longer matched its own host. NewAllowlist now maps each pattern's host part (after any "*.") the same way and returns an error for a pattern it cannot map. Dropping it instead could empty the list, which means unrestricted. The unmappable-host test now asserts the error from Allowed. Refs #589 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGMCH6rFHiCNxPrp9hwTG9
…ling dot The pattern "*." kept the suffix ".", so it matched every host with a trailing dot, and a browser loads evil.com. as evil.com. "*." plus U+00AD mapped to the same thing, because IDNA drops U+00AD. normalizePattern now refuses a pattern whose mapped host is empty or has an empty label. Allowed trims one trailing dot from the request host, as webfetch does, so github.com. matches a github.com entry. Refs #589 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGMCH6rFHiCNxPrp9hwTG9
- Jina checks every redirect hop by URL; the proxy dial exemption hid a proxied hop's target from the dial check. - Refuse numeric hosts that aren't dotted-quad (netguard.EndsInNumber), so a pure-Go DNS failure can't send them to Jina. - Treat local-use NAT64 64:ff9b:1::/48 as private (RFC 8215 leaves its layout to the site); never-open IPv4 in the /96 bits stays refused. - Domain validator: IP checks after "*." and IDNA mapping, numeric last labels rejected, one trailing dot accepted like the matcher. - Warn on reload when the [web.fetch] opt-ins change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGMCH5yY44TFC5yeXSNicz
There was a problem hiding this comment.
All reported issues were addressed across 2 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
View guided diff | Turn on auto-fix | Re-trigger cubic
A reload that changes the effective [web.fetch] opt-ins now logs one Warn stating the mixed state: new agents use the new policy, running agents keep the old one until restart. Closing access by reload was silent before, and opening reused the startup text. Docs: [web] is no longer called restart-only, and the NAT64 bullet says what a refusal looks like. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGMCH5yY44TFC5yeXSNicz
requestGuard.check refuses hosts with an empty label, and EndsInNumber trims every trailing dot, so 2852039166.. no longer reaches the resolver or Jina. Local-use NAT64 addresses are checked for a never-open IPv4 at every RFC 6052 position, so 64:ff9b:1:a9fe:a9:fe00:: stays refused under allow_private. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGMCH5yY44TFC5yeXSNicz
… docs c1f017f picked up "internal/netguard/numeric 2.go", an iCloud sync copy that redeclared EndsInNumber and broke the build. The built-in tools page and the mcp-tools rule still called [web] restart-only; they now match the config reference: running agents keep their settings until restart, agents created after a reload use the new values. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGMCH5yY44TFC5yeXSNicz
The every-position check refuses a public IPv4 whose bytes read as a never-allowed class at another RFC 6052 position, so a non-/96 local-use layout can refuse some public addresses (e.g. 8.8.224.1 in the /48 layout). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGMCH5yY44TFC5yeXSNicz
Drop the browser's local endsInNumber for the shared helper. The local copy trimmed one trailing dot, so a numeric host with more, such as 127.1.. in isBlockedHost or 2130706433... through Allowed, was not refused. The netguard helper trims every trailing dot. Refs #589 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGMCH6rFHiCNxPrp9hwTG9
The browser Allowlist trimmed one trailing dot, and CheckHost trims one more, so localhost..., 127.0.0.1..., 169.254.169.254... and metadata.google.internal... were allowed. Move webfetch's empty-label rule into netguard.HasEmptyLabel and call it from both requestGuard.check and the browser's isBlockedHost. Refs #589 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YGMCH6rFHiCNxPrp9hwTG9
There was a problem hiding this comment.
All reported issues were addressed across 16 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
All reported issues were addressed across 2 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Purpose
web_fetchonly checked the URL scheme, so it could reach cloud metadata, loopback (including Denkeeper's own API) and private ranges, directly or through a redirect. Found in the OWASP Agentic Top 10 review (ASI02).Closes #589
What a reviewer needs to know
internal/netguardpackage refuses blocked addresses in the dialer, so redirects, robots.txt lookups and DNS rebinding are covered. A URL-string check runs first only for a clearer error.[web.fetch] allow_loopbackandallow_privateopen their own ranges. Metadata (incl. Azure168.63.129.16), link-local and the proxy's own address never open. The docs spell out how broadallow_privateis.64:ff9b:1::/48counts as private. A host whose DNS64 uses it needsallow_privateto fetch IPv4-only sites.HTTP(S)_PROXYset, the proxy must enforce egress. Documented once under "Network policy".allowed_domains/blocked_domainsare name-only. Bad entries (.evil.com, IP literals, unmappable hosts) fail at load.netguard, but has no production caller. The container's own check is Block SSRF to metadata, loopback, and private ranges denkeeper-browser#89.Test plan
just hookhttptestservers and the real dialer. The redirect, metadata, IDNA, proxy and default-config tests fail onmain🤖 Generated with Claude Code
https://claude.ai/code/session_01YGMCH6b8gjLdh3ZaC2QwRh
Summary by cubic
Blocks
web_fetch, the browser allowlist, and the Jina fallback from reaching cloud metadata, loopback, and private ranges, directly or via redirects. The check runs on the IP each connection reaches at dial time, so redirects, robots/agents.txt lookups, and DNS rebinding are covered. Closes #589.The guard
internal/netguardpackage classifies addresses in the dialer; loopback and private ranges open only via[web.fetch] allow_loopbackandallow_private.168.63.129.16, AWSfd00:ec2::254, Alibaba100.100.100.200), link-local, unspecified, multicast, and broadcast never open, even inside private ranges.Behavior changes
allowed_domainsandblocked_domainsmatch the IDNA-mapped host; IP literals, numeric last labels, and malformed entries fail at config load. They cover the URL sent to Jina, not Jina's own redirect targets, and robots/agents.txt disallows still fall back to Jina.NO_PROXYhosts go through the address check.Written for commit c9bd9d2. Summary will update on new commits.