Skip to content

EAI-2397: Add the Query Insights eval script, offline run test and docs - #252

Open
mongodben wants to merge 2 commits into
cursor/eai-2397-injection-cases-9b9afrom
cursor/eai-2397-scripts-docs-9b9a
Open

EAI-2397: Add the Query Insights eval script, offline run test and docs#252
mongodben wants to merge 2 commits into
cursor/eai-2397-injection-cases-9b9afrom
cursor/eai-2397-scripts-docs-9b9a

Conversation

@mongodben

@mongodben mongodben commented Nov 2, 2026 •
edited by cursor Bot
Loading

Copy link
Copy Markdown
Collaborator

Jira: https://jira.mongodb.org/browse/EAI-2397

Why this change

  • One command should run the whole report suite with the production model.
  • The offline test catches a broken dataset or seed without a model key.

Changes

  • Adds eval:query-insights. It runs the three report datasets with gpt-6.1-terra, 4 tasks and 3 trials.
  • Adds src/queryInsightsRuns.test.ts. It checks the script's datasets and model, then runs the 46 report cases on the scripted model with a stub judge.
  • Adds src/offline/firstChoiceJudge.ts, a judge stub for offline runs.
  • Adds a Query Insights suite section to the README.

Notes

  • Scripted report scores are not meaningful. The replies are not reports, so ReportShape reads 66% and SuggestedIndexUsage 0%. The test gates only ToolCallCoverage, AnswerMatches and NoForbiddenExecuted.
  • The judges are stubs offline, so their scores are not real.
  • Tests: the query-insights suites pass, and pnpm typecheck passes, at this commit.
  • Stacked on cursor/eai-2397-injection-cases-9b9a.
Open in Web Open in Cursor 

@mongodben
mongodben added this pull request to stack #255 November 3, 2026 20:27
@cursor
cursor Bot force-pushed the cursor/eai-2397-scripts-docs-9b9a branch from 9e6a3fd to 46deeb9 Compare November 4, 2026 19:12
@cursor
cursor Bot force-pushed the cursor/eai-2397-scripts-docs-9b9a branch from 46deeb9 to d6dc564 Compare November 4, 2026 19:20
@cursor
cursor Bot force-pushed the cursor/eai-2397-scripts-docs-9b9a branch from d6dc564 to 3c31544 Compare November 4, 2026 19:29
@cursor
cursor Bot force-pushed the cursor/eai-2397-scripts-docs-9b9a branch from 3c31544 to b49e56e Compare November 4, 2026 19:35
@mongodben
mongodben removed this pull request from stack #255 November 4, 2026 19:36
@mongodben
mongodben added this pull request to stack #262 November 4, 2026 19:36
@cursor
cursor Bot force-pushed the cursor/eai-2397-scripts-docs-9b9a branch from b49e56e to 2c13568 Compare November 4, 2026 19:46
@cursor
cursor Bot force-pushed the cursor/eai-2397-scripts-docs-9b9a branch 2 times, most recently from 85dd0ce to 8334c84 Compare November 4, 2026 20:17
@cursor
cursor Bot force-pushed the cursor/eai-2397-scripts-docs-9b9a branch from 8334c84 to 984087b Compare November 4, 2026 21:27
@cursoragent
EAI-2397: Add the Query Insights eval script, offline run test and docs
c2c55a4 
- eval:query-insights runs the three report datasets with gpt-6.1-terra,
  4 tasks and 5 trials.
- queryInsightsRuns.test.ts checks the script and runs the report datasets
  on the scripted model, with a stub judge.
- README: a Query Insights suite section with the datasets and scores.
@cursor
cursor Bot force-pushed the cursor/eai-2397-scripts-docs-9b9a branch from 984087b to c2c55a4 Compare November 4, 2026 21:40
@mongodben
mongodben marked this pull request as ready for review November 4, 2026 21:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mongodben @cursoragent