Is it the model or the harness? Benchmarking an AI threat hunter across six LLMs
We ran six LLMs through Scout's full harness on ten frozen attack scenarios, three times each. The model sets how good an AI threat hunter can be; the harness decides whether that quality shows. Four lessons for anyone evaluating agents, and a leaderboard.
TL;DR: We put six LLMs, from a 30B open model on two of our own GPUs to the newest Gemini, through Scout's full harness on ten frozen attack scenarios, three times each. Both matter: the model sets how good Scout can be, and the harness decides whether that quality shows at all. One harness fix lifted every model it was tested on by about 0.12 recall, half the gap between our production model and a small self-hosted one. Along the way we learned four lessons about evaluating agents, and we close with a checklist for teams doing the same.
→ See how Scout hunts across your own data
At Huntbase we build Scout, an AI threat hunter that investigates across a security team's cloud, identity, endpoint and SIEM data. It plans a hunt, writes queries in each source's own query language, runs them through Huntbase, and turns the results into findings and a verdict. "The harness matters more than the model" has become a common claim about AI agents, and most of Scout is harness, so we tested the claim on it. We ran six models, from a 30B open model on two of our own GPUs to the newest Gemini, through the same harness, the same frozen environment and the same ten attack scenarios, three times each.
The short answer: both matter. The model sets how good Scout can be, and the harness decides whether that quality shows at all. One harness fix lifted each of the three models we tested it on by about the same amount, roughly 0.12 recall. That is half the gap between our production model and a small self-hosted one, and across all six models the best-to-worst gap is more than three times larger.
Along the way we learned four things about evaluating agents that we wish we had known at the start: single-call benchmarks overstate agent readiness, lenient scoring and single runs make models look interchangeable, the serving stack is part of the model, and one run tells you almost nothing about reliability. This post covers how we tested, what we found, what it means for Huntbase customers, and why we build Scout the way we do.
Why we ran it
Scout is the centre of Huntbase. Security teams connect their data sources to Huntbase, and Scout does the hunting across them. An analyst asks a question in chat, or a detection or intel item seeds an autonomous hunt. Scout then plans the investigation, writes and runs the queries, reads the results and reports what it found. Around the model sits about twenty sub-agents' worth of engineering: an investigation planner, query authoring for each platform's query language, validators and query repair, a pipeline that runs queries across the customer's connected platforms, and the state that ties a hunt together.

The model makes the judgement calls: what to look for, how to ask each source, and what the results mean. Everything else is harness, and that is where most of our engineering goes. That raised two questions we wanted to answer for ourselves.
The first was practical: could Scout run entirely on open-weight models, on infrastructure we or a customer control, with no calls to a hosted LLM? If it could, Scout could go where hosted models can't.
The second is the one every agent builder argues about, and for us it decides where engineering time goes: does the harness (the prompts, tools, planner, validators and execution pipeline around the model) matter more than the model itself? If it does, model choice is a cost decision and our engineering effort belongs in Scout's harness. If it doesn't, we should chase the frontier.
Both questions have the same shape. You can't answer them with a public leaderboard, because public leaderboards measure models in isolation, and Scout is not a model in isolation. So we built a benchmark that measures the whole agent, and made it repeatable so we can run it again whenever a new model ships or the harness changes.
How we tested an agent, not a model
Three design choices did most of the work.
1. Test at several layers, from one call up to the whole hunt. We scored single sub-agent calls (write this query, plan this investigation), live multi-turn chats where the model chooses every tool call, and fully autonomous hunts where Scout's planner turns a seed alert into a graph of queries that the platform executes. The layers differ in who controls the flow of work: the model, or the harness. That turned out to be the most useful axis in the whole study.
2. Freeze the world. Every run starts from the same saved snapshot of a complete Huntbase stack: 22 simulated data sources (cloud, identity, endpoint, SIEM, SaaS) with planted attack evidence for six incident stories, all anchored to a fixed point in time. The ten hunt scenarios come from those six stories: four of the stories are hunted twice, each time from a different first clue. We restore the snapshot before every repeat. Without this, results drifted with simulator randomness, timestamps that moved on every restart, and leftover hunts from earlier runs that the agent could find and copy.
3. Only count evidence the agent established. A hunt scores recall on planted entities (users, hosts, IPs, domains, files) only when they appear in the agent's own conclusions: verdicts, findings, the hunt summary and its facts. Restating the alert, or naming something in a plan step that never ran, earns nothing. Our first scoring version gave credit for words in node labels, and quietly rewarded agents for repeating the question.
Every agent in a trial runs on the model under test (a per-run override follows the request across HTTP and Kafka into every worker), and we turned off Scout's silent fallback to Gemini. Otherwise a typo in a model name would have produced a perfect Gemini score labelled as something else. Each configuration ran three times; we report the average, how often all three repeats passed, and how often at least one did.
A hunt passes only when every check passes: it settles before the time limit (a final status, or every step finished), it plans at least two steps, no more than a third of its steps fail, it finds at least half the planted entities, and its verdict matches the expected one. One of the ten scenarios has no expected verdict, so it is scored on the first four checks. A hunt's score is the share of its checks that pass. The single-call planner score uses a rubric graded by an LLM judge (Gemini 3 Pro), which never runs as the model under test; every other score is deterministic. Where we call a difference significant, its 95% confidence interval, from a paired bootstrap over scenarios, excludes zero.
Lesson 1: single-call benchmarks overstate agent readiness
On the narrow task of writing one detection query from a natural-language request, a 30B open model running on two of our own GPUs scored 90 (with the decoder fix described in Lesson 3) against Gemini 3.8 Flash's 97, statistically indistinguishable, and answered faster. On that evidence alone you would ship it.
Put the same model in Scout's chat loop and it scored 20 against Gemini's 93. The reason was not query quality. It wrote good queries and then stopped: "I've generated queries that will find this activity." It never called the tool that runs them, so its answers never contained the evidence. gpt-oss-120b, larger in total but with only about 5B active parameters, behaved the same way. DeepSeek V4.1 Flash, with about five times the active parameters of the 30B model, wrote, ran, read and followed up, and landed near Gemini.

On one query the gap is small. Once the model has to carry an investigation itself, the gap opens wide. The autonomous hunt sits in between, because Scout's planner and the platform do part of the driving.
The gap between "can do the step" and "will carry the task through" is invisible to any benchmark that scores one call at a time. For agent products, follow-through is a capability in its own right, and in our runs it behaved like a threshold: below it, prompt nudges and tool-level fixes did not help (we tried accepting malformed arguments, recovering tool calls written as text, and "now run it" hints: no measurable change over three repeats), above it models converge quickly.
Lesson 2: lenient scoring can make every model look the same
Our first autonomous-hunt results (one run per model, four scenarios) looked like a clean win for "the harness matters more": the small self-hosted model matched Gemini on hunts, with the same score and slightly better recall. When the planner and the platform drive the work, it seemed, the model barely matters.
That reading was wrong, and the way it was wrong is the most transferable lesson in this post.
Every model, from the smallest to the strongest, seemed to sit at the same 0.35–0.4 recall. We read that as a shared ceiling and spent three rounds removing suspects:
- The environment. Simulated events had random timestamps that moved on every restart, and some tables served different columns from the ones the catalog described. Fixing both nearly doubled the share of queries that returned data (12% to 21%). Recall did not move.
- The planner's routing. It was offering connector variants that could not execute in this environment. Filtering them removed a whole class of failed steps. Recall did not move.
- The data lake. The planner was told to prefer the lake, which was empty for these scenarios. Fixing that moved queries to real sources. Recall did not move.
The ceiling was in how conclusions were formed. The hunt summary that downstream steps read contained row counts ("Returned 14 rows") but never the rows themselves. And a background verdict was never recorded, because the agent's own writes marked the hunt as "analyst engaged", which blocked autonomous verdicts by design. The model saw the evidence while it worked, but nothing it saw reached the conclusion.

Both problems sat after the queries had run, which is why fixing the environment, the routing and the lake never moved recall.
Once we gave the conclusion-forming steps a sample of actual result rows and let autonomous hunts record a verdict, recall rose by +0.10 to +0.15 for each of the three models we tested (Gemini 3.8 Flash, DeepSeek V4.1 Flash and Qwen3-30B; four scenarios, three repeats each). The ranking did not change. And the "parity" between the 30B model and Gemini turned out to be noise: measured properly, with three repeats and evidence-only scoring, Gemini sat about 0.2 recall above it both before and after the fix.
So the shared ceiling was partly an artifact of how we measured: with single runs and lenient scoring, two models 0.2 recall apart looked the same. Per-model confidence intervals are wide at four scenarios. Pooled over all twelve model–scenario pairs, the lift averaged +0.12, and ten of the twelve were zero or better.

Each model moved by a similar amount, and the order did not change.
Two things are true at once. The harness determined whether model quality could show up at all, and one harness fix was worth about half the gap between Gemini and the 30B model. And model quality still mattered, as soon as the harness stopped hiding it.
Lesson 3: the serving stack is part of the model
The single biggest swing we measured on one model came from a serving flag, not the model.
Scout's agents ask for strict JSON output against a schema. With vLLM's default settings, the constrained decoder allows unlimited whitespace between JSON tokens. On long prompts the open model drifted into writing spaces and newlines until it hit the 16k-token cap, and produced no query at all. Query-authoring score: 51.6.
Forcing compact JSON (no whitespace) fixed it: 94.8. It also broke the planner. Denied its natural ": " token, the model filled string fields with literally ", ", and planning fell from 73 to 51. We ended up patching the decoder to allow whitespace but cap it at 32 characters per run, which recovered both (90.2 and 71.9).

Each setting that helped one agent hurt the other, until we capped the whitespace.
This is not a self-hosting quirk. Google's managed endpoint for an open model showed the same whitespace runaway on our planner schema. And Anthropic's API rejected our planner schema in strict structured-output mode: the compiled grammar was too large. In JSON mode, with the schema in the prompt, Claude Opus 5.5 produced the best plans we measured in single-call tests.
The practical rule: a model's score is a score for model + serving stack + output contract. Every serving change has to be re-run across every agent, because a fix that helps one agent can break another. And "strict structured output everywhere" is an assumption that holds for one provider and fails for others; output mode should be configurable per model and per agent.
Lesson 4: report reliability, not just capability
Single runs lied to us twice.
DeepSeek V4.1 Flash scored 97.6 on our live chat suite in one run, apparently matching Gemini. Over three repeats it averaged 80. It solved 86% of the cases at least once (pass@3, the same as Gemini) but only 43% every time (pass^3; Gemini managed 86% in the same run). Its bad runs were ones where its own queries failed to execute against our data and it answered "can't tell yet". For a security analyst, a tool that is right on Monday and silent on Tuesday is a different product from one that is right every day. Gemini's own pass^3 on the same seven conversations ranged from 43% to 86% across three separate runs, so even the baseline needs repeats.

In this run, the same pass@3 and half the pass^3: DeepSeek's misses came and went between repeats.
The second lie was about our own work. A set of tool-call robustness fixes appeared to lift the small model by 21 points in one run. At three repeats it had no measurable effect.
On hunts, even the best model is far from reliable on our harder scenarios: Gemini 3.8 Flash passed 5 of 10 scenarios at least once, but only 2 of 10 on every repeat. We now report the mean, pass@k and pass^k side by side, and treat any single-run delta under about 10 points as noise.
A related trap: an averaged score can hide how a model fails. Claude Haiku 5.5 found as much planted evidence as Gemini on hunts, but passed only 1 of 30 trials. Its plans were 60% larger than Gemini's (a median of 13.5 steps vs 8.5) and about half of each plan's steps failed to execute, which our "executable plan" check catches. Breadth without precision looks like recall until you check whether the plan actually ran.
The leaderboard
With the harness fixed and the world frozen, we ran six models through ten autonomous hunts, three times each. Every model ran the full Scout harness; only the model changed.

Gemini 4 Argon led on score, recall and pass rate: it passed every scenario at least once, found the most planted evidence on 7 of the 10, and took about as long per hunt as the current production model. Its lead over Gemini 3.8 Flash (+6.8 score, 95% CI −3.5 to +16.7) is not yet statistically significant at ten scenarios. Claude Sonnet 5.5 sat within noise of Gemini 3.8 Flash and finished hunts about 30% faster. In a separate head-to-head on live chat, Argon also beat Gemini 3.8 Flash: 98 vs 91 over seven conversations, three repeats each, a statistically significant gap, at about 1.7× the time per conversation.
The open-weight models fell behind on hunts for two different reasons. Each failed to produce a plan at all in 7 of 30 hunts. The small self-hosted model found little (mean recall 0.18). DeepSeek, strong in chat, wrote plans with more steps that could not run. Both fail in ways a single-call benchmark would never show. Both also ran on the kind of structured-output serving Lesson 3 describes, and we have not yet separated model errors from serving errors in their failed plans.
And nobody is reliable yet on the harder scenarios: no model passed more than two of the ten on every repeat. That is the next ceiling to find.
Limitations
- Small samples. Ten hunt scenarios and seven chat conversations. Confidence intervals are wide, and we give them where it matters.
- Our own test. We wrote the scenarios, the simulated data and the scoring. We found the harness fixes on four of the ten scenarios; the other six were added after those fixes.
- No bare-harness run. We did not compare Scout with and without its harness. The harness result is one fix, measured on three models.
- Serving differs by model. Qwen3-30B ran on our own vLLM with a patched decoder, DeepSeek on a managed endpoint, and the other models on their providers' APIs. The Claude models ran the planner in JSON mode.
- Not every model ran every layer. The Claude models were not run in live chat, and Claude Opus 5.5 ran only the single-call tests.
- Cost measured for two models. From our LLM gateway's spend logs, Gemini 4 Argon cost about $1.82 per hunt against $0.83 for Gemini 3.8 Flash, and $0.50 against $0.23 per chat conversation: roughly 2.2× on both. We did not price the other models.
What this means for Huntbase customers
Scout runs on Gemini 3.8 Flash today, and the benchmark backs a hosted Gemini model: Gemini 3.8 Flash led the four-model chat tests (against DeepSeek V4.1 Flash, gpt-oss-120b and Qwen3-30B; the Claude models were not run in chat) on quality and consistency, and Gemini 4 Argon led hunts on score, recall and pass rate. In a head-to-head chat run Argon also beat 3.8 Flash (98 vs 91), though each conversation took about 1.7× as long. At about 2.2× the model cost of 3.8 Flash per hunt and per chat, we are now evaluating whether Argon's lead is worth it for Scout. More importantly, we can offer customers options backed by evidence rather than guesses.
- Hosted (default). A Gemini model for Scout and its sub-agents: 3.8 Flash today, with 4 Argon under evaluation for hunts and chat. Claude Sonnet 5.5 is within noise of Gemini 3.8 Flash on hunts and about 30% faster per hunt, so it is a credible alternative for customers who standardise on Anthropic.
- Hybrid. Open models already match hosted ones on narrow, well-specified tasks. A self-hosted model can take query authoring (90 vs 97, not a significant gap, and faster), while the agent loop and the investigation planner stay on a hosted model.
- Fully self-hosted. Possible, but not with a small model. The 30B-class model we ran on two A100s is not good enough to drive Scout's chat or autonomous hunts. A DeepSeek-class open model gets close to Gemini on chat, but is less consistent and needs a full 8-GPU H200-class node. We will validate that on our own hardware before recommending it.
And the harness work keeps paying: the evidence-grounding change found in this study lifted every model's hunt results, including the self-hosted one.
A checklist for teams evaluating agents
- Benchmark the agent, not the model. Include at least one layer where the model drives the loop and one where your harness does; the difference between them is the most informative number you will get.
- Freeze and restore the world before every repeat: data, timestamps, search indices and leftover state.
- Score only what the agent established, never words it could have copied from the prompt.
- Kill silent fallbacks and assert which model actually answered.
- Run k ≥ 3 and report pass^k next to the mean. Treat single-run deltas as rumours.
- Treat serving config as a variable: decoding settings, structured-output mode and tool-call parsing, re-tested across every agent after each change.
- When every model scores the same, look for a ceiling in the harness or the environment (or in your scoring) before concluding the model doesn't matter.
So, is it the model or the harness? In our data, both: one well-placed harness change was worth about half the gap between our production model and a small self-hosted one. Across all six models, the best-to-worst spread was more than three times that lift. The harness decides how high any model can go, and the model decides how close to that ceiling it gets. The work that paid off most was finding where the harness was hiding the model.
Why Huntbase
This study was not a one-off project. It shows how we build Scout.
- Model-agnostic by design. Every Scout agent reaches its model through one OpenAI-compatible gateway, and each agent's model is configuration, not code. That is what let us put six models, hosted and self-hosted, through the same harness, and it is what lets our customers choose.
- The harness is the product. Most of our engineering goes into the planner, per-platform query authoring, validators, query repair and the execution pipeline. This study showed why that pays: one harness change lifted every model we tested it on, including the self-hosted one. That change, evidence grounding, is on its way into Scout.
- Measured, not guessed. The benchmark lives alongside Scout's code, starts every run from a frozen copy of a full Huntbase stack, and is built to re-run whenever a new model ships or the harness changes. When we recommend a model, the recommendation comes with numbers: score, recall, reliability across repeats and time per hunt.
- Your sources, your deployment. Scout hunts across the platforms a team already uses, in each platform's own query language. Whether Scout's models run hosted, hybrid or on infrastructure you control, we can tell you what that choice costs in hunt quality, because we measured it.
If your team hunts across cloud, identity, endpoint and SIEM data and wants an AI hunter it can hold to the same standard, we would like to talk.