dataprof profiles CSV, JSON, JSONL, Parquet, DataFrames and Arrow tables in one call: types, nulls, distributions, patterns and a quality score. Then it can gate your pipeline on what it found, including changes from a saved baseline. It runs locally and records when the evidence cannot settle a check.
A Rust library with a Python package on top. Local file profiling needs no server, account or network access.
pip install dataprof # or: uv pip install dataprofWheels for CPython 3.11 to 3.15 on Linux, macOS and Windows, with no Python dependencies.
import dataprof as dp
report = dp.profile("orders.csv") # also .json, .jsonl, .parquet, DataFrames, dicts
print(report.rows, "rows,", report.columns, "columns, quality", report.quality_score)
amount = report["amount"]
print(amount.data_type, amount.mean, amount.null_percentage)Ask it what deserves attention. Each finding has a stable code and the evidence behind it, never a raw value:
for finding in report.findings():
print(finding.severity, finding.code, finding.column)
# warning locale_numbers price ("10,50"-style numbers, left out of the stats)
# warning null_heavy email
# info sensitive_pattern emailState what "good enough" means and get a verdict: pass, fail, or inconclusive when the data can't prove it either way.
result = report.check(
min_quality_score=90, max_null_count={"customer_id": 0}, max_null_percentage={"*": 20}
)
if not result.passed:
for check in result.violations:
print(check.code, check.column, check.message)Or from CI, with no code at all:
python -m dataprof.check orders.csv --min-quality 90 --max-null "*=20"
# exit 0 = pass, 1 = fail, 2 = inconclusive, 3 = an argument, policy, input, baseline or output error
# Or against the last accepted run: fail if quality drops more than 5 points.
# A rolling clock lets each run judge its dates against its own day.
python -m dataprof.check orders.csv --baseline accepted.json --max-quality-drop 5 --timeliness-clock rollingKeep the candidate report with --report candidate.json. Promote it to the
baseline only on a pass, including the first run, which uses absolute rules
alone. A changed or unrecorded metric definition can leave a comparison
definition_not_comparable; inspect the reason and regenerate the baseline
deliberately. See the checkpoint recipe.
0.14 adds these checks:
- Freshness and missing periods:
check(max_date_age={"published_at": 3}, max_date_gap={"published_at": 1})limits the age of the newest date and the largest gap between consecutive distinct days.dp.profile("orders.csv", reference_date="2026-06-10")fixes the timeliness clock. - Volume:
check(baseline=baseline, max_row_count_drop=10, max_row_count_growth=20)limits percentage changes in row count;min_rows=1sets an absolute floor. - Keys:
check(unique_columns=["order_id"], max_null_count={"order_id": 0})requires distinct non-null keys and no nulls. From CI, use--unique-column order_id --max-null-count order_id=0.identifier_columns=["order_id"]selects the key for the uniqueness score andkey_duplicatesfinding. - Schema drift:
forbid_removed_columns=Truerejects removed columns.compare()and baseline diagnostics also report inferred type changes; these diagnostics alone do not fail a gate. A column'snative_typerecords its source's declared type where available.
See the cookbook for complete examples.
A token-bounded summary for an LLM. Values that look like personal data are never echoed:
print(report.to_llm_context(max_tokens=500))cargo add dataprofuse dataprof::{Profiler, QualityPolicy, Verdict};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let report = Profiler::new().analyze_file("orders.csv")?;
let result = QualityPolicy::new().min_quality_score(90.0).evaluate(&report)?;
assert_eq!(result.verdict, Verdict::Pass);
Ok(())
}Minimum supported Rust: 1.96.
- Same data, same serialized numbers. Rust Serde output and Python
to_dict()/to_json()agree across routes and engines for the same logical data, schema semantics, options and analyzed population, within the numeric contract. Source and execution provenance are compared separately. Raw floats may differ in their final digits; extreme values near the limits off64belong to an approximate domain. A retained sample and a full scan need not agree. Reports flag sampled numeric statistics withis_approximate, estimated distinct counts withunique_count_is_approximate, and sampled quality components withquality_sampled_dimensions. - Bounded memory. Files larger than RAM stream through fixed-size accumulators. The exact duplicate-row count holds up to 1,000,000 distinct rows by default, about 10 to 20 bytes each, inside the memory budget, and becomes an estimate under memory pressure. A
duplicate_rows_exact_limityou set yourself is the exception: those rows stay exact and are kept outside the budget. - Honest verdicts. A sampled or partial scan passes a claim about the whole file only when what it read already proves it, such as a row minimum it reached; otherwise the check is inconclusive. A sampled score carries an interval in
score_boundswhenever it can be bounded. - Absence is not zero. A metric that wasn't computed is
None, never a plausible default. - Local by default. Profiling runs on your machine. Reports carry statistics, including numeric minima, maxima and modes;
to_llm_context()leaves raw values out by default.
The Python wheels include file and bytes profiling, Arrow C array and stream
inputs, and dataprof.asyncio file, bytes and HTTP helpers. Install the table
library you use, plus pyarrow for pandas and polars conversion; native Parquet file and bytes
profiling needs none of them. PostgreSQL, MySQL and SQLite require a source
build with connector features. See input routes.
Beta, and moving fast toward a stable 1.0 contract. Some things are still narrow, and we'd rather tell you than have you find out: numbers written with a decimal comma are detected and flagged but not yet parsed, and beyond empty, null and nan, only a fixed set of markers (NA, N/A, n/a, #N/A, \N, None, matched exactly, ignoring surrounding whitespace) counts as null, so a sentinel like - or -999 is still a value. The release notes list what changed and what's known.
Found something wrong? Open an issue. A file that profiles badly is the most useful bug report there is.
- Why dataprof?: profiling and quality gates compared with summaries, visual reports and validation tools, including when to pick each
- Getting started: the metrics and how to read them
- Python API: every function, the gate and the CI entrypoint
- Examples: a messy CSV, an ETL gate, before and after cleaning, a checkpoint against the last accepted run, a collapsing key column, runnable from a clean checkout
- Agent workflows: AGENTS.md, Cursor rules and Claude Code skills
- Report schema: the saved document and its guarantees
- Database connectors and feature flags: Rust-side options
- Changelog and contributing
Use Cite this repository in the GitHub sidebar, which builds APA and BibTeX from CITATION.cff. The citation is for the software; benchmark material lives in scalcom2026-dataprof.
Either the MIT License or the Apache License, Version 2.0, at your option.
