Geoconnex Scheduler

The Geoconnex scheduler crawls water data metadata on a schedule and synchronizes with a graph database.

• it crawls sitemaps with nabu harvest and downloads it to an S3 bucket
• it syncs data between the S3 bucket and the graph database using nabu sync

For more information about the Geoconnex project generally and how it aims to improve water data infrastructure, see the Geoconnex docs.

Local / Development Quickstart

Important

You must have uv installed for package management

Install dependencies and spin up necessary Docker services:

make deps && make dev

Then go to localhost:3000

Dockerized / Production Quickstart

• Spin up all services as containers including user code and local db/s3 containers (make sure to set the DAGSTER_POSTGRES_HOST env var to dagster_postgres)

make prod

Spin up user code and essential services but not storage (You will need to specify your db/s3 endpoints and any other remote services in the .env file)

make cloudProd

Cloud deployment

All GCP infrastructure as code lives in opentofu/ (previously the harvest.geoconnex.us repo). It provisions the network, storage buckets, Cloud SQL database, pygeoapi, graphdb, qlever, and the VM that clones this repo and runs make cloudProd.

You must use OpenTofu rather than Terraform since the gcs backend is configured with variables.

cd opentofu
# fill in a tfvars file based on empty.tfvars and supply your GCP credentials file
make pull_state
make apply

Deploying code changes

tofu apply is only needed for infrastructure changes. To ship new code to the running VM, push to main and run from the repo root:

make redeploy

This waits for the push_to_ghcr image build of the latest origin/main commit, then SSHes into the VM (with gcloud compute ssh) and runs make cloudUpdate there. That pulls the repo and the new images and recreates only the containers that changed. It refuses to restart while Dagster runs are in progress, since restarting user code kills them; pass FORCE=true to override. The VM name and zone come from the tofu outputs, or set VM_NAME and VM_ZONE. If you are already on the VM, run sudo make cloudUpdate in /opt/scheduler.

Configuration

• All env vars must be defined in .env at the root of the repo
• The .env.example file will be copied to .env if it does not exist

Testing

• Spin up the local dev environment
• Run make test from the root to execute tests
• If you use VSCode, run the task dagster dev in the debug panel to run the full pipeline with the ability to set breakpoints

Licensing

This repository is a heavily modified version of gleanerio/scheduler and is licensed under Apache 2.0