Skip to content

Security: DwellyOrg/OperateBench

SECURITY.md

Security and privacy policy

Status: draft for the technical preview. Response times below are targets for a small maintainer team, not contractual commitments. There is no bug bounty and no paid disclosure programme.

Reporting

Report privately to dmitry@dwelly.group, with OperateBench security in the subject line. If the repository has GitHub private vulnerability reporting enabled, submit a private vulnerability report. That channel is equally acceptable and preferred for code defects. If it is unavailable, use the email address above.

Do not open a public issue for a security or privacy report.

Please include: what you found; how to reproduce it; the affected version, commit or run artifact; and what you think the impact is. If a report contains data you believe is non-synthetic, describe it — do not paste it.

Targets: acknowledgement within 3 working days; an initial assessment within 10 working days; and a fix or a stated decision not to fix, with reasoning, before public disclosure. We will keep you updated if any of those slip.

We will credit reporters who want credit, and respect requests to stay anonymous. We ask for a reasonable disclosure window and will not pursue good-faith research that stays within the scope below.

In scope

  • Non-synthetic data in the repository. Any personal data, customer data, real identifiers, proprietary text or real operational thresholds that appear in a fixture, test, example, document or run artifact. This is the highest priority report this project can receive, and every fixture is meant to be fully synthetic.
  • Credential or secret exposure in code, history, artifacts, CI configuration or logs.
  • Unsafe file handling in the runner or loader: path traversal, symlink following, writing outside a run directory, or overwriting an artifact that should be append-only.
  • Evidence integrity defects: a way to make the ledger, replay or manifest verification accept a run artifact that does not follow from its own recorded evidence.
  • Unintended network or provider activity from a command documented as offline.
  • Injection through fixture content — an operation fixture, message fixture or event payload that causes code execution, file access or command execution in the runner or its tooling.
  • Dependency vulnerabilities that are reachable from this project's code.

Out of scope

  • Model behaviour. An agent that fails a scenario, escalates unnecessarily or produces poor text is a benchmark result, not a vulnerability.
  • Prompt injection of an evaluated agent by authored fixture content. That is a deliberate part of the construct: fixtures contain plausible-but-false claims and pressure, and an agent that acts on them is meant to fail. Injection that escapes the agent and reaches the runner is in scope.
  • Cost, rate-limit or quota exhaustion at a third-party provider caused by your own run configuration.
  • Reports generated by automated scanners without a demonstrated impact on this project's code paths.
  • Findings in third-party services, papers or repositories that this project merely links to.

What this project already assumes

Stated so that reports can target real gaps rather than known ones:

  • Run artifacts are local and unsigned. Append-only ledgers and replay verification are internal-consistency guarantees, not cryptographic ones. Whole-file replacement with self-consistent rows is not locally detectable, and no release-grade immutability claim is made. External immutable storage or a signed published commitment to a run's digests would be needed for one.
  • Credentials are environment-supplied, never stored. No credential is accepted through a configuration file or a command-line setting, and none should ever be written to a manifest, ledger, log, error message or artifact. A report that one is, is in scope and serious.
  • The documented public/offline benchmark commands make no provider calls. Every agent those commands use is deterministic and in-process, and the commands make no benchmark-runtime network call. Separate private operator-only diagnostics require explicit operator authorization and are excluded from published benchmark evidence.

Privacy commitments

Every published fixture is fully synthetic, classified SYNTHETIC_ONLY, and reviewed by someone other than its author before release. No production records, customer data or proprietary material is used. See docs/CONTRIBUTING_OPERATIONS.md and NOTICE.

If non-synthetic data is ever found in this repository, the response is to remove it, publish a correction note under docs/VERSIONING.md describing what was affected, and review how it got past intake — in that order.

Repository dependency controls

Dependabot checks the uv lockfile and GitHub Actions weekly. Minor and patch version updates are grouped within each ecosystem, with at most three open version-update pull requests per ecosystem; major upgrades remain separate. Updates require review and are not automatically merged. Action updates must retain full commit pins and pass the publication checks.

The root requirements.txt is a hash-preserving export of the runtime and all development groups from uv.lock, provided for GitHub's native dependency graph. uv.lock remains the sole dependency source of truth; installation and Dependabot updates continue to use uv, not this repository-only export. After a lockfile update, regenerate it with uv export --frozen --all-groups --no-emit-project --offline > requirements.txt. CI compares a fresh offline export byte-for-byte and rejects drift.

The dependency-review pull-request check rejects newly introduced dependencies with known high or critical vulnerabilities, including development dependencies. It uses GitHub's dependency graph, not a complete audit of existing dependencies or application reachability. Coverage depends on GitHub's supported manifests and available dependency data; a green check is not proof of no vulnerabilities. The workflow has read-only permissions, does not execute pull-request code, and does not post comments or change licensing policy.

Dependency review is available for public repositories without a paid security plan. Dependency graph availability and private vulnerability reporting are repository settings, not enabled or proven by these files. The main-branch policy requires dependency-review alongside the existing test, build, independent secret scan and DCO checks. Maintainers must verify actual forge enforcement and graph coverage; repository prose and a green check alone cannot prove either.

Supported versions

During the technical preview, only the latest release and main receive fixes. There is no long-term support branch, and preview versions are expected to break compatibility — see docs/VERSIONING.md.

There aren't any published security advisories