Skip to content

federation: verification process and registry publication on sync backend - #364

Open
klpoland wants to merge 13 commits into
masterfrom
feature-kpoland-federation-registry-sign-publish
Open

klpoland wants to merge 13 commits into
masterfrom
feature-kpoland-federation-registry-sign-publish

Conversation

@klpoland

@klpoland klpoland commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Note

High Risk
Touches authentication (admin bearer), Ed25519 signing, SSRF-safe applicant fetches, and the end-to-end peer onboarding path; misconfiguration could block joins or expose internal APIs.

Overview
Moves federation registry ownership from the Django gateway to federation sync. Site identity (/.well-known/sds-federation.json), signed manifest URLs (/.well-known/sds-registry/*), and staff-only POST /internal/registry/* (verify, publish, capabilities) now run on sync behind FEDERATION_SYNC_ADMIN_TOKEN. Applicant domain proof uses FEDERATION_REGISTRY_CHALLENGE_TOKEN on the applicant sync process, not gateway django.env.

The gateway retains intake UI and application DB but delegates verify/publish to sync via FederationSyncClient and FEDERATION_SYNC_INTERNAL_URL. Publish registry builds the site list from approved applications, calls sync to Ed25519-sign the manifest, and marks rows published. Successful verify triggers staff email (FEDERATION_REGISTRY_STAFF_NOTIFY_EMAILS) and optional auto-publish on approve.

Deploy/ops updates: compose mounts for registry signing keys and publish/cache volumes, Traefik routes for well-known to sync, federation-doctor registry checks, sds-federation-registry CLI, operator runbook federation-registry-deploy.md, and generate-secrets.sh now mints FEDERATION_SYNC_ADMIN_TOKEN. Django removes the well-known URL and registry signing/challenge settings that lived on the app container.

Reviewed by Cursor Bugbot for commit 072231f. Bugbot is set up for automated code reviews on this repo. Configure here.

@klpoland klpoland self-assigned this Nov 4, 2026
@klpoland klpoland added feature New feature or request refactoring General code improvements gateway Gateway component federation Federation related work (sync service, document storage, peer configuration, etc.) labels Nov 4, 2026
@semanticdiff-com

semanticdiff-com Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Review changes with  SemanticDiff

Changed Files
File Status
  federation/tests/test_registry_verify.py  31% smaller
  gateway/sds_gateway/templates/users/registry_application_list.html  29% smaller
  federation/sds_federation/services/registry_verify.py  27% smaller
  gateway/sds_gateway/api_methods/federation/registry_application_verify.py  23% smaller
  gateway/config/settings/utils.py  22% smaller
  gateway/config/settings/base.py  21% smaller
  gateway/sds_gateway/api_methods/tests/test_registry_application_verify.py  21% smaller
  gateway/sds_gateway/users/views/registry_application.py  17% smaller
  gateway/sds_gateway/api_methods/tests/test_file_endpoints.py  15% smaller
  gateway/sds_gateway/visualizations/tests/test_views.py  15% smaller
  gateway/sds_gateway/api_methods/tests/test_asset_access_control.py  15% smaller
  gateway/sds_gateway/api_methods/tests/test_authenticate.py  15% smaller
  gateway/sds_gateway/api_methods/tests/test_composite_capture_serialization.py  15% smaller
  gateway/sds_gateway/api_methods/tests/test_permission_updates.py  15% smaller
  gateway/sds_gateway/users/tests/test_check_superuser_exists_command.py  15% smaller
  gateway/sds_gateway/users/tests/test_create_ci_superuser_command.py  15% smaller
  gateway/sds_gateway/api_methods/tests/test_celery_tasks.py  15% smaller
  gateway/sds_gateway/users/tests/test_dataset_authors.py  15% smaller
  gateway/sds_gateway/users/tests/test_managers.py  15% smaller
  gateway/sds_gateway/users/tests/test_registry_application_staff_review.py  15% smaller
  federation/sds_federation/services/site_registry.py  13% smaller
  gateway/sds_gateway/api_methods/tests/test_permissions.py  12% smaller
  gateway/sds_gateway/api_methods/tests/test_federation_hardening.py  12% smaller
  gateway/sds_gateway/api_methods/tests/test_capture_endpoints.py  12% smaller
  gateway/sds_gateway/api_methods/federation/registry_challenge.py  12% smaller
  federation/sds_federation/main.py  12% smaller
  gateway/sds_gateway/api_methods/federation/registry_application_review.py  8% smaller
  gateway/sds_gateway/api_methods/tests/test_registry_challenge.py  2% smaller
  federation-shared.example.env Unsupported file format
  federation/.gitignore Unsupported file format
  federation/README.md Unsupported file format
  federation/compose.local.yaml  0% smaller
  federation/compose.peer.local.yaml  0% smaller
  federation/compose.peer.remote.yaml  0% smaller
  federation/compose.production.yaml  0% smaller
  federation/docs/federation-registry-deploy.md Unsupported file format
  federation/docs/peer-remote-test.md Unsupported file format
  federation/env.local.example Unsupported file format
  federation/env.peer.example Unsupported file format
  federation/federation.example.toml Unsupported file format
  federation/pyproject.toml Unsupported file format
  federation/scripts/federation-doctor.sh Unsupported file format
  federation/scripts/render-site-config.sh Unsupported file format
  federation/scripts/set-registry-challenge.sh Unsupported file format
  federation/scripts/write_site_env.py  0% smaller
  federation/sds_federation/cli/__init__.py  0% smaller
  federation/sds_federation/cli/registry.py  0% smaller
  federation/sds_federation/models.py  0% smaller
  federation/sds_federation/routes/internal_registry.py  0% smaller
  federation/sds_federation/routes/well_known.py  0% smaller
  federation/sds_federation/services/registry_challenge.py  0% smaller
  federation/sds_federation/services/registry_publish.py  0% smaller
  federation/sds_federation/services/registry_sign.py  0% smaller
  federation/sds_federation/services/well_known.py  0% smaller
  federation/secrets/registry/.gitkeep Unsupported file format
  federation/secrets/registry/README.md Unsupported file format
  federation/site.example.env Unsupported file format
  federation/templates/federation.toml.tmpl Unsupported file format
  federation/templates/gateway-federation.env.tmpl Unsupported file format
  federation/templates/traefik-sync.toml.tmpl Unsupported file format
  federation/tests/conftest.py  0% smaller
  federation/tests/test_internal_auth.py  0% smaller
  federation/tests/test_internal_mount.py  0% smaller
  federation/tests/test_registry_challenge.py  0% smaller
  federation/tests/test_registry_publish.py  0% smaller
  federation/tests/test_registry_serve.py  0% smaller
  federation/tests/test_registry_sign.py  0% smaller
  federation/tests/test_site_registry_accept.py  0% smaller
  federation/tests/test_well_known_route.py  0% smaller
  gateway/.envs/example/django.env Unsupported file format
  gateway/.envs/example/django.prod-example.env Unsupported file format
  gateway/.gitignore Unsupported file format
  gateway/config/urls.py  0% smaller
  gateway/docs/federation-registry-verify.md Unsupported file format
  gateway/justfile Unsupported file format
  gateway/pyproject.toml Unsupported file format
  gateway/scripts/generate-secrets.sh Unsupported file format
  gateway/sds_gateway/api_methods/federation/registry_application_helpers.py  0% smaller
  gateway/sds_gateway/api_methods/federation/registry_application_notify.py  0% smaller
  gateway/sds_gateway/api_methods/federation/registry_application_staff_ops.py  0% smaller
  gateway/sds_gateway/api_methods/federation/registry_manifest.py  0% smaller
  gateway/sds_gateway/api_methods/federation/sync_client.py  0% smaller
  gateway/sds_gateway/api_methods/federation/well_known.py  0% smaller
  gateway/sds_gateway/api_methods/migrations/0024_registry_application_drop_verified_status.py  0% smaller
  gateway/sds_gateway/api_methods/migrations/max_migration.txt Unsupported file format
  gateway/sds_gateway/api_methods/models.py  0% smaller
  gateway/sds_gateway/api_methods/tasks.py  0% smaller
  gateway/sds_gateway/api_methods/tests/test_federation_sync_client.py  0% smaller
  gateway/sds_gateway/api_methods/tests/test_federation_well_known.py  0% smaller
  gateway/sds_gateway/api_methods/tests/test_registry_application_notify.py  0% smaller
  gateway/sds_gateway/api_methods/tests/test_registry_application_review.py  0% smaller
  gateway/sds_gateway/api_methods/tests/test_registry_application_staff_ops.py  0% smaller
  gateway/sds_gateway/api_methods/tests/test_registry_manifest.py  0% smaller
  gateway/sds_gateway/api_methods/views/federation_well_known.py  0% smaller
  gateway/sds_gateway/templates/emails/registry_application_pending_approval.html  0% smaller
  gateway/sds_gateway/templates/emails/registry_application_pending_approval.txt Unsupported file format
  gateway/sds_gateway/templates/users/registry_application_detail.html Unsupported file format
  gateway/sds_gateway/users/forms.py  0% smaller
  gateway/sds_gateway/users/tests/test_federation_registry_intake_settings.py  0% smaller
  gateway/sds_gateway/users/tests/test_registry_application_publish_view.py  0% smaller
  gateway/sds_gateway/users/tests/test_registry_application_verify_view.py  0% smaller
  gateway/sds_gateway/users/urls.py  0% smaller
  gateway/sds_gateway/users/views/__init__.py  0% smaller
  gateway/uv.lock Unsupported file format
  network/traefik/conf.d/00-federation-sync.toml Unsupported file format
  sdk/pyproject.toml Unsupported file format

@klpoland
klpoland marked this pull request as ready for review November 4, 2026 20:49

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit fb80b54. Configure here.

Comment thread federation/compose.local.yaml Resolved
@cursor

cursor Bot commented Nov 13, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 439e1317-a02c-4f79-9695-f99f57463ec0)

@cursor

cursor Bot commented Nov 13, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 1d3f5807-fd3a-45f4-bcd8-2ff4ad4a9db5)

@cursor

cursor Bot commented Nov 13, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 2c97926b-9663-4afd-8948-5875c95ee5f1)

@klpoland
klpoland requested a review from mjcurran November 13, 2026 20:17

@mjcurran mjcurran left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gateway builds and runs on my system, so looks good to test on QA and proceed from there.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New feature or request federation Federation related work (sync service, document storage, peer configuration, etc.) gateway Gateway component refactoring General code improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants