Every vendor tells you their AI is the best. We tell you the same thing, and you have no particular reason to believe us either. So we ran a test, because we wanted to know.
We took 100 real questions our team had asked in our own production Omni instance, and tested five AI setups using the same questions, data, and warehouse:
Omni's AI Agent, which uses Anthropic's models by default
A leading coding agent handed our semantic layer plus a detailed playbook
That same coding agent with the semantic layer, but no playbook
That same coding agent with no semantic layer (raw SQL)
A warehouse-native AI agent
We anonymized the external products because this benchmark focuses on how different architectures perform, rather than vendor rankings. That said:
The results show Omni is better, faster, and less expensive. #
We attribute this more to the harness than the semantic layer alone, but I’ll explain more on that later.
Each question had a predefined rubric that covered the correct answer, failure criteria, and what “close, but wrong” looked like. Claude Opus judged each response, and a second Opus agent ran a supervised check. We also repeated the full matrix twice to test whether the results held.
The Omni results include our validation agent, which reviews the work before returning an answer. This is enabled by default in production, so the results reflect the experience customers receive.
The numbers #
Before getting started, we put in a lot of effort to ensure as much parity as possible across setups. Of course we were rooting for ourselves, but we needed to trust our own numbers. We measured accuracy, reliability, speed, and cost. Omni came out on top, as we hoped. You can learn more about our methodology below, but for now, here’s what we found:
Accuracy: Omni got 95 out of 100 right. That breaks down to 96% on the easy questions, 97% on medium, and 90% on hard. The next-closest alternative hit 60% accuracy on hard questions.
Reliability: Omni was also more consistent, returning the same answer on 96 of the 100 questions across both runs. The next-best alternatives changed 30 answers between runs. On 7 of 100 questions, Omni passed every time while the other setups failed every time. And Omni was the only configuration that returned an answer every single time. The alternatives failed to return any answer at all, or errored out, 19 times across both runs.
Speed and cost: Omni's median response time was roughly twice as fast, and it cost about $1.04 per correct answer, compared to $1.29 for the next-best alternative.
It's in the harness #
Ultimately, the difference boils down to guardrails. Omni’s harness forces the AI agent to compile queries through the semantic model. The other setups have a choice. I could talk all day about the efficiencies and workflows we’ve built into our semantic layer and harness, but what matters is whether or not definitions are enforced.
We gave a leading coding agent our semantic model, along with the playbook explaining exactly how to use it – which fields to pick, joins to follow, etc. It still trailed Omni by 21 points.
I expected the semantic layer to be the equalizer. If two agents see the same metric definitions, join paths, field descriptions, etc., it’s reasonable to assume they converge on the same answers. They didn’t.
Because when you hand a coding agent the semantic layer, it's used as a reference where it should be a rulebook. The agent can still pick a different join path or redefine a metric.
In fact, the coding agent, even given the full context of our semantic model, followed the same pattern almost every time:
Try to read the model export in one shot. Fail (The file is too large to read all at once)
Grep it, and get reams of truncated noise
Write throwaway scripts to parse the model
Find the right definitions (it often did)
“I have everything I need” (some version of this shows up each transcript)
Hand-write SQL based on what it read
Run it against the warehouse, and format a confident answer
Nearly every observed failure happened in that sixth step. The agent’s research was usually a success. But hand-writing SQL is where things fell apart.
Omni works differently, the agent doesn’t hand-write SQL. Instead:
The question routes to a Topic, a curated slice of the semantic model
The agent pulls that Topic’s fields, joins, definitions, and AI context
The agent selects which fields and filters to use
A query engine compiles the selection into SQL
A validation agent reviews the answer against the user’s intent
That difference showed up everywhere in the results. Omni’s agent burned fewer tokens, answered more accurately, and was more reliable across runs.
Omni burned about 670K tokens per correct answer. The alternatives burned one to six million, because each question had to be re-derived against the schema. And because every derivation was independent, the same question could produce different SQL on the next run (which is how you get the 30 flipped answers by a single agent).
To show what this looks like, here are a few of my favorite failures from the runs.
“Europe doesn’t exist” - Coding agent with no semantic layer #
One question asked which EMEA solutions engineer had the most closed-lost opportunities this fiscal year.
The raw-SQL agent started reasonably by listing every database in the warehouse. But the list was too large to read inline, so the agent saved it to a file. Then it only read the first 50 lines.
Skipping the majority of databases in the list (the one holding our CRM data was further down 😅), the agent committed to the wrong database and spent eight minutes combing through it. 60 warehouse queries. Sweeps for anything named “region” or “engineer.”
As a last ditch effort, it tried to find European users by checking whether their longitude was positive. Then, it gave up. It confidently declared that no EMEA region or SE role existed in our warehouse, and suggested our CRM data “hasn’t been loaded into this warehouse yet.”
The whole thing took 489 seconds, cost 6.1 million tokens, and produced no answer.
Omni’s agent routed the same question to the Opportunities Topic, ran a single query, and named the right engineer in 34 seconds, using 466K tokens.
The coding agent spent 13x the tokens and 14x the wall-clock spinning its wheels searching the wrong database, then told us our own data didn’t exist.
“The $55 billion campaign” - Coding agent with semantic layer & playbook #
Another question asked for our top five campaigns by pipeline, with member and opportunity counts.
The coding agent’s research was great. It found the campaign Topic, the canonical pipeline measure, the right stage list, and the right filters.
It even read the comment we left in the model warning about this exact query, which explained that the Topic joins campaigns directly to opportunities, because joining via membership inflates the numbers.
“I have all the definitions I need,” it said.
Then it did its own thing: it hand-wrote a query joining campaigns, members, and opportunities, ignoring the warning in the model, and causing a Cartesian fan-out.
It said our top campaign was worth $55.5 billion in pipeline. I wish.
On the other hand, Omni’s agent read the same Topic and ran a single governed query. The join paths were enforced, and the answer was correct. It also used a third of the tokens and took half the time.
“The phantom ARR crash” - Coding agent with no semantic layer #
The agent skimmed the first 50 lines of the database list, picked wrong, and landed in a demo schema containing synthetic data. It didn’t notice. It invented an ARR formula from the demo tables, and produced a clean-looking analysis.
The headline? “71% ARR crash ⚠️”
The top contributors were companies like “Wagner Group” and “Holden, Atkins and Paul.” None of them exist. The agent closed with analyst commentary on the fake data and praising our “diversified customer base.”
This may not sound like a big deal because I’m talking about demo data, but it’s more real than you’d expect. Every production warehouse has clutter. Demo schemas, test tables, stale uploads. An agent that can see everything can’t tell the demo apart from the business.
Omni's agent answered the same question from an ARR Topic, on about a sixth of the tokens. It never saw the clutter at all.
Every failure proved my expectation wrong. The semantic model isn’t enough; it’s the harness because it forces the agent to obey the model. The institutional knowledge you put into the model comes back as reliability. But as we’ve seen, when the model is optional, most of that effort evaporates.
Refining and improving #
When you're measuring reliability, speed, and cost, you naturally want to improve. So we took our own advice: we added missing measures, wrote clearer field descriptions, fixed a fiscal calendar edge case, and dropped in a handful of sample queries.
It confirmed that our model is what we say it is: tuneable, and the layer where the accuracy gap actually closes. It also further validated that running that loop by hand is miserable, which is what the tooling in AI Hub is for.
Before any of that, Omni was already ahead on accuracy by about eight points. After tuning, the gap grew to 21. Omni also burned roughly three times fewer tokens per correct answer and ran about twice as fast as the next-best setup. The coding agent only gained seven points.
The obvious takeaway is when the semantic layer is optional, better context helps a little. When it's the only path the agent can take, better context helps a lot. We see this internally, and so do our customers.
A note on methodology #
Benchmarking different AI agents on analytics questions involves a ton of moving parts. Every setup needs parity. Every question needs to represent real usage, and be testable on a pass/fail basis. And you need to establish ground truth for each question.
That last part requires subject matter experts from every domain you want to test. For us, that work had already been done: domain experts contributed their expertise to our semantic model, so it already spelled out the business context behind the intent of virtually all of these questions.
We used it to write rubrics for each question, including correct answers, what "close but wrong" looked like, the traps a plausible answer could fall into, and the different possible answers when a question had more than one.
That still left about a thousand answers to grade across both runs. Claude Opus judged each one against its rubric; and since the judge isn’t deterministic, a second supervised pass audited the verdicts and arbitrated the uncertain ones.
This benchmark isn't exact. It's representative. The numbers are directional, and every warehouse is different. The only way to know whether they hold up is to run it yourself.
Run the benchmark on your own data #
This benchmark tells you how these architectures performed on an environment our team has curated over Omni's lifetime. Our results don't tell you how Omni will perform on your data.
Every setup is different, and you should always test for yourself. So I invite you to bring your hardest questions to a free trial, and see for yourself how Omni’s AI performs on your data.
If you’re a customer and want to run the benchmark yourself, reach out to Vinnie on our team for the code.
