Skip to content

web_fetch: robots.txt and agents.txt disallows still fall back to Jina #620

Open

Description

@Temikus

With respect_robots_txt or respect_agents_txt on, a disallowed URL still gets fetched - just through Jina instead of directly.

checkCompliance in internal/webfetch/fetch.go returns plain fmt.Errorf errors for a robots.txt or agents.txt disallow. ChainFetcher only skips the fallback on netguard.ErrBlocked, so any other primary error goes to Jina, and Jina fetches the page without checking either file.

That makes both settings a no-op whenever Jina is enabled. Someone who turned them on is opting out of fetching those paths, not out of fetching them directly.

Proposed fix

  1. Add a sentinel (e.g. webfetch.ErrDisallowed) and wrap both disallow errors with it.
  2. Have ChainFetcher return it without trying the fallback, same as ErrBlocked.
  3. Drop the caveat in the "Network policy" section of website/content/docs/reference/config.md.

A robots.txt lookup that fails (network error, parse error) should keep proceeding as it does now. Only an actual disallow should stop the fallback.

Follow-up from #615.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions