[RFC] Opt-in Prometheus metrics extension for operational monitoring

[stanGSA]

Motivation

I'd like to contribute an optional metrics extension for self-hosted DeerFlow deployments, so operators can scrape aggregate task, model-call and tool-call metrics into their existing Prometheus setup.

DeerFlow already integrates with LangSmith, Langfuse and Monocle for tracing. This proposal would complement those integrations with a direct Prometheus scrape surface; it does not assume tracing platforms cannot provide aggregation or alerting.

The task/tool monitoring request in #4357 is related, although this would address only the aggregate metrics portion. I also checked Discussion #2699: that work focuses on local diagnostics and execution timelines, while this proposal concerns deployment-level operational metrics.

The backend TODO also lists metrics and monitoring. More concretely, Discussion #6383 led to merged PR #6430, which adds synchronous/asynchronous tool outcome and elapsed-time statistics to the extension example. That provides an observation pattern to build on. Its returned / raised / cancelled categories describe control flow; this proposal would additionally distinguish reported tool errors. These references provide context, not prior approval of this exporter design.

Proposed first version

Following the extension-first guidance, I propose a separately installable deerflow-extension-prometheus package using the public extension API:

• Tasks: lead/subagent starts, finishes, outcomes, paired lifecycle durations and an event-based active-task gauge, through TaskLifecycleContributor.
• Model calls: attempts observed at MODEL_PHYSICAL, duration, outcome, bounded error categories and input/output tokens when usage is available.
• Tool calls: attempts observed at TOOL_VISIBLE, duration and reported outcomes, with interruption and cancellation distinguished from errors.
• Export: an extension-owned registry and a dedicated /metrics listener managed by ExtensionService, proposed default 127.0.0.1:9464, with optional bearer-token protection. No default Compose port publishing or changes to Gateway authentication.
• Deployment: one Gateway worker per instance. Separate single-worker pods could be scraped individually; shared-port multi-worker aggregation would be unsupported in v1.

The intended implementation needs no host-core changes. If a contract gap emerges, I would bring it back for discussion. Package location (examples/ versus a separate community repository) and metric names remain open.

Why a separate listener? Contributed Gateway routes inherit host authentication. The current PAT policy is default-deny outside explicitly allowed routes, and PAT credentials do not carry administrator capability (#5041). An ordinary PAT therefore cannot simply be used to scrape the proposed deployment-wide metrics route. A dedicated listener with its own optional bearer token would avoid changing host authentication for v1; this remains a deployment choice for maintainer feedback.

Candidate v1 metrics

Names and labels below are proposed for review, not a frozen API. Histograms would also expose the standard bucket/count/sum series.

Metric	Type	Proposed labels
deerflow_tasks_started_total	Counter	kind, agent
deerflow_tasks_finished_total	Counter	kind, agent, outcome
deerflow_task_duration_seconds	Histogram	kind, outcome
deerflow_tasks_active	Gauge	kind
deerflow_llm_requests_total	Counter	model, scope, outcome, error_class
deerflow_llm_request_duration_seconds	Histogram	model, scope
deerflow_llm_tokens_total	Counter	model, scope, type
deerflow_tool_calls_total	Counter	tool, scope, outcome
deerflow_tool_call_duration_seconds	Histogram	tool, scope
deerflow_metrics_label_overflow_total	Counter	label

kind/scope distinguish lead and subagent work; token type is limited to input/output in v1. Outcomes and error classes use fixed vocabularies. Agent/tool/model names are bounded; suppressed agent/tool names would use a constant label value. Request and tool counters refer to attempts at the observation boundaries described below.

Measurement and operational boundaries

• MODEL_PHYSICAL sees retries that re-enter the middleware, but may not see retries inside a provider SDK. These would be middleware-observed attempts, not a guarantee of counting every HTTP request or every 429. The current model factory sets max_retries=0 when request_admission is configured and the model class exposes that field. This reduces hidden SDK retries on that path; it does not guarantee visibility into retries in other layers. Other configurations may undercount actual 429 responses, and observed duration can include SDK retry/backoff time. The extension would not change retry policy to improve measurement.
• Lifecycle callbacks can be missing or skipped. Pairing and idempotence would prevent duplicate updates, but the active-task gauge would remain best-effort, not an authoritative concurrency or billing source.
• Tool outcomes would describe observable results, not prove business success. Command results would need explicit handling of any relevant error ToolMessage they carry.
• Labels would be bounded per process, with overflow handling and options to suppress agent/tool names. No user/thread/run/task IDs or message content would be exported. Custom names can still contain business information, and per-process limits are not a global cardinality cap.
• Opt-in packaging limits default impact; it does not make installed middleware, dependencies or a listener risk-free. Validation would cover failure isolation, cancellation, lifecycle cleanup and event-loop behavior.

Scope and validation

V1 would exclude system-model/compaction metrics, HTTP/sandbox/queue instrumentation, dashboards, per-user/tenant metrics, TTFT, OTLP export and Helm integration. System-model and compaction metrics could be small follow-ups. Multi-worker support would need a separate design for exporter ownership/failover, dead-worker cleanup and file I/O.

Tests would cover sync/async hooks, result/exception preservation, lifecycle asymmetry, labels, authentication and listener cleanup, plus host integration using fake models/tools. Extension dependencies would be installed explicitly in the test environment; the existing wheel/package check does not replace running those tests. Real-run validation and any untested cases would be reported accurately in a later PR.

Feedback requested before implementation

1. Does an optional operational-metrics extension fit the project direction, and is there existing work I should coordinate with?
2. Would you prefer a bundled examples/ package or a separately maintained community package?
3. Is prometheus-client as an extension-only dependency appropriate, or would you prefer an OTel Metrics approach?
4. Is a dedicated opt-in listener with single-worker support an acceptable first scope?

I will wait for directional feedback before implementing v1 and open a linked tracking issue if that would help.

[renezander030]

On the tool side I'd split cancelled by where the cancel landed: before the tool body started, or while it was running. For a read the two are the same, but a write cancelled mid-call may already have reached the downstream system, so counting it as a harmless cancellation hides exactly the calls an operator has to go and reconcile. Does the TOOL_VISIBLE wrapper sit close enough to the tool body to tell those two apart?

[stanGSA]

Thanks for raising this. For a write, cancellation can leave the external outcome unknown, so the metric needs to make that limitation explicit.

TOOL_VISIBLE alone cannot reliably distinguish cancellation before the tool body starts from cancellation during execution. It wraps downstream middleware as well as the tool invocation, so entering its handler does not establish that the tool body has started. Cancellation could also be observed after the tool has returned, while downstream result processing is still running.

For v1, I would define cancelled as a cancellation signal propagating through the observation boundary. It would not guarantee that the underlying work stopped or that an external write was prevented or rolled back.

There is a TOOL_RAW placement closer to the callable boundary. I could evaluate pairing it with TOOL_VISIBLE to distinguish observed invocation phases, with an unknown fallback where the evidence is insufficient. I would validate that before promising a before-start/during-execution split; even then, those phases would not establish whether a remote write committed.

I’ll clarify the cancellation semantics in the proposal. Aggregate metrics can flag cancellations for investigation; reconciling individual writes would still require logs, traces, or downstream audit records.