Open source
The harness behind the numbers
recount-core is a Python package and command-line tool that runs a question set against a data agent many times and reports how dependable it is. Every figure on this site was produced with it.
Not published yet
The harness is coming as open source, under the Apache-2.0 licence. There is no repository or package to install yet, and this page will say so until there is. Until then, the method on the Method page is the part you can read.
01 · What it does
From a question set to a report
- Repeat runner
- Asks each wording many times, each in a fresh conversation. Runs resume if a budget runs out. Adaptive stopping keeps asking only the wordings that are still mixed.
- Paraphrases
- Can propose extra wordings with a language model. They are marked unreviewed, and a run refuses them until a person has checked them. Only the question text is sent to the model.
- Dynamic ground truth
- Expected answers stored as SQL or DAX and recomputed when the suite runs.
- Deterministic grader
- A number from the answer against a computed value. An LLM judge is optional and scored separately.
- Trace capture
- The agent's rewrite of the question, the source it chose, the query and the answer, for every run.
- Ambiguity probe
- For each business term, computes every defensible definition and reports which one the agent used, and whether its label matches.
- Change diff
- Compares two configurations by category with bootstrap 95% intervals, and lists the questions that moved.
- Change gate
- A check for a pipeline that passes, blocks or says it can't decide. A fall blocks on proof by default, and a question whose expected answer has gone out of date isn't scored. It hasn't run in a real pipeline yet.
- Two-run trace
- Shows what a change did to one question: the answers before and after, and the phrases its rewrites gained or lost.
- Proof
- Replays the recorded live runs through the real gate, on cases where the right answer is known, and fails if any answer is wrong. See the proof page.
- Capacity guard
- Limits on asks and rate, and a warning when the target capacity is shared. It refuses capacities you list as production unless you override it.
02 · Try it
Replay the published runs with no cloud
When it is published, one command will replay the 1,080 recorded runs through the real runner, grader and report, with no cloud account, and write the reports to a folder you can open. Another will replay them through the real gate and check its answers.
Install commands and a repository link will appear here at that point. We'd rather show nothing than something that doesn't work yet.
03 · Platforms
Fabric is the one we have tested in depth
The Microsoft Fabric data agent connector has run against a live agent, and the published runs come from it. The Fabric SDK it uses is a preview.
The Databricks Genie connector has run against test spaces of ours: first on the electricity data, where its expected answers matched Fabric's, then across the nine industries of the census. A baseline connector runs a plain model writing SQL over copies of the same tables, as a second agent to compare against. Connectors for Snowflake Cortex Analyst and Looker were written from each vendor's documentation and haven't run against a live agent. A connector is one small module, so a platform that isn't covered can be added without touching the rest. Power BI Copilot has no connector.
04 · Known limits
What to know before relying on it
- The Fabric trace parser relies on tool names that aren't a documented contract, and the SDK is a preview. A smoke command runs a few asks to check that a full trace still comes back.
- Only a signed-in user has been tested as the identity the agent runs under.
- The grader passes an answer if the right number appears anywhere in it, matching the published method. Reports count how often that wasn't the headline figure.
- Cost covers AI Query usage from reported tokens only, and is a lower bound.
- Decisive stages and rewrite phrases come from small samples. They are leads for a person to confirm.
- The bundled NEM example contains answers about data from the Australian Energy Market Operator (AEMO), used with attribution.