Repository navigation
Conversation
9245bb8 to
21589d6
Compare
cursor
Bot
changed the title
867d02c to
9335a17
Compare
9335a17 to
fc141bb
Compare
fc141bb to
0932ae5
Compare
0932ae5 to
ade156b
Compare
ade156b to
e86448a
Compare
Converts the synthetic source fixtures into mock Atlas seeds and a query-insights dataset. The converter now takes a hand-written reference per case, because the source references quote lifetime totals, and each case replays the Performance Advisor call for the offline scripted run.
e86448a to
ea7ea02
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Several query-insights reference explanations misinterpret join and index behavior or assume an unknown boolean, creating incorrect grading expectations that can penalize accurate reports.
4 open issues
What changed in this PR
Ports 19 synthetic Query Insights cases into the evaluation suite, expanding
coverage beyond smoke tests with mock Atlas data and rewritten references.
Changes:
- Adds and registers a dataset with 19 matching Atlas fixtures.
- Requires hand-written references in the porting CLI and generates scripted
Performance Advisor calls for offline runs. - Adds tests for fixture validity, number-free references, and tool-call coverage.
| File | Description |
|---|---|
packages/evals/src/queryInsightsDataset.test.ts |
Validates fixtures, references, and offline tool coverage. |
packages/evals/src/porting/query-insights-selection.yml |
Selects 19 cases and defines rewritten references. |
packages/evals/src/porting/portQueryInsightsCases.ts |
Requires references and omits generated source comments. |
packages/evals/src/porting/portQueryInsightsCase.ts |
Supports reference overrides and scripted advisor calls. |
packages/evals/src/porting/portQueryInsightsCase.test.ts |
Tests scripts, reference overrides, and seed rendering. |
packages/evals/src/porting/portAssistantCase.ts |
Makes source comments optional in dataset rendering. |
packages/evals/src/porting/portAssistantCase.test.ts |
Tests rendering without source comments. |
packages/evals/src/cli.ts |
Registers the Query Insights dataset. |
packages/evals/fixtures/atlas/query-insights-case-34.yml |
Seeds an unfiltered sort with an index suggestion. |
packages/evals/fixtures/atlas/query-insights-case-33.yml |
Seeds collection storage-statistics queries. |
packages/evals/fixtures/atlas/query-insights-case-32.yml |
Seeds nested-array matching with sorting. |
packages/evals/fixtures/atlas/query-insights-case-31.yml |
Seeds a partially supported multi-field sort. |
packages/evals/fixtures/atlas/query-insights-case-30.yml |
Seeds a two-field ascending sort. |
packages/evals/fixtures/atlas/query-insights-case-29.yml |
Seeds a nested-field range query. |
packages/evals/fixtures/atlas/query-insights-case-28.yml |
Seeds text search with relevance sorting. |
packages/evals/fixtures/atlas/query-insights-case-27.yml |
Seeds matching followed by grouping. |
packages/evals/fixtures/atlas/query-insights-case-26.yml |
Seeds a status query with a broad compound-index suggestion. |
packages/evals/fixtures/atlas/query-insights-case-25.yml |
Seeds a match, sort, and limit aggregation. |
packages/evals/fixtures/atlas/query-insights-case-24.yml |
Seeds an unindexed $in query. |
packages/evals/fixtures/atlas/query-insights-case-23.yml |
Seeds an $or query with blocking sorting. |
packages/evals/fixtures/atlas/query-insights-case-22.yml |
Seeds an $or query using a collection scan. |
packages/evals/fixtures/atlas/query-insights-case-21.yml |
Seeds deep skip-based pagination. |
packages/evals/fixtures/atlas/query-insights-case-19.yml |
Seeds an unindexed field-existence test. |
packages/evals/fixtures/atlas/query-insights-case-17.yml |
Seeds range filtering with in-memory sorting. |
packages/evals/fixtures/atlas/query-insights-case-16.yml |
Seeds a regex query with extensive key scanning. |
packages/evals/fixtures/atlas/query-insights-case-15.yml |
Seeds an inefficient index-prefix query. |
packages/evals/fixtures/atlas/query-insights-case-14.yml |
Seeds an admissions-to-memberships lookup. |
packages/evals/datasets/query-insights.yml |
Defines cases, scripts, expected calls, and grading references. |
Review effort: Balanced
| elaboration: "The `$match` is supported by an index, but each | ||
| admission is joined with several membership documents, so the | ||
| join multiplies the documents the pipeline processes. Consider a |
There was a problem hiding this comment.
Leaving this as is. The join claim comes from the source eval case's reference answer, which we treat as ground truth for these ported cases. We only rewrote it to drop lifetime totals the agent never sees. Whether $lookup multiplies rows is a judgement the source case already made, so we'd rather not change it here. If it turns out to be wrong, the fix belongs in the source case first, and we can re-port it.
| elaboration: "The index scan examines many keys, and every document | ||
| it examines is returned, so the index filters out little. Consider |
There was a problem hiding this comment.
Agreed, and fixed in fb728cb. The inference that the index "filters out little" was ours, not the source case's. The reference now says: "The index scan examines many keys, while every document it fetches is returned." It no longer claims anything about the index filtering. Selection file and datasets/query-insights.yml are both updated, and the six PRs above this one are restacked.
| - file: case_19_synthetic.json | ||
| reference: | ||
| queryOverview: "This query finds tasks where `taskType` equals a string | ||
| value and the `archivedAt` field exists." |
There was a problem hiding this comment.
Leaving this as is. The partial-index advice for the archivedAt existence test is in the source eval case's reference answer, which we treat as ground truth. The port only removes lifetime totals the agent never sees. The ?bool placeholder is how the seed hides the literal value, not a statement about it. If the source advice is wrong, the fix belongs in the source case first.
| elaboration: "Adding the right index will help, since it reduces the | ||
| number of documents that reach the `$group` stage. Review the |
There was a problem hiding this comment.
Leaving this as is. "Reduces the number of documents that reach the $group stage" comes from the source eval case's reference answer, which we treat as ground truth. The port only removes lifetime totals the agent never sees. If the source wording is wrong, the fix belongs in the source case first.
Jira: https://jira.mongodb.org/browse/EAI-2397
Why this change
Changes
fixtures/atlas/query-insights-case-<id>.yml.query-insightsdataset (datasets/query-insights.yml) and registers it in the CLI.pnpm port:query-insightsto take a hand-writtenreferencefor each case, and to emit a scripted Performance Advisor call for offline runs.src/porting/query-insights-selection.yml, which lists the cases and their rewritten references.Notes
referenceper entry.src/queryInsightsDataset.test.tschecks that every case reads a valid seed, that references have no numbers, and that the scripted run calls the Performance Advisor on all 19 cases.gpt-6.1-terra, 5 trials, 19 cases, run on Braintrust.ToolCallCoverageis low for a reason the metric hides. When the agent makes more than twice the required calls, the score becomesceiling / actual. The agent investigates withcollection_indexes,explainand similar tools, so the number reflects extra calls, not a missed Performance Advisor call.ToolCallArgCoverageis the signal to use for thresholds.cursor/eai-2397-judges-9b9a.