You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
web_fetch: robots.txt and agents.txt disallows still fall back to Jina #620
With respect_robots_txt or respect_agents_txt on, a disallowed URL still gets fetched - just through Jina instead of directly.
checkCompliance in internal/webfetch/fetch.go returns plain fmt.Errorf errors for a robots.txt or agents.txt disallow. ChainFetcher only skips the fallback on netguard.ErrBlocked, so any other primary error goes to Jina, and Jina fetches the page without checking either file.
That makes both settings a no-op whenever Jina is enabled. Someone who turned them on is opting out of fetching those paths, not out of fetching them directly.
Proposed fix
Add a sentinel (e.g. webfetch.ErrDisallowed) and wrap both disallow errors with it.
Have ChainFetcher return it without trying the fallback, same as ErrBlocked.
Drop the caveat in the "Network policy" section of website/content/docs/reference/config.md.
A robots.txt lookup that fails (network error, parse error) should keep proceeding as it does now. Only an actual disallow should stop the fallback.
Activity