Benchmarking Jev, Clef Flash and GPT-6 Luna for Tool Calling

Every turn an agent takes starts with the same decision: call a tool, and which one, or answer the user. Today a large model makes that decision by reading the whole conversation plus every tool's schema. If a small, fast decision model could make it reliably, Ragwalla could offer the large model fewer tools, skip it on turns where the answer is obvious, or route the turn somewhere cheaper.

Before building any of that, we wanted to know whether a decision model can make the call at all, and, more importantly, whether it knows when it can. This post covers what we measured, how, and what we found.

The test

We ran τ-bench (v2.0) against Ragwalla agents. τ-bench simulates customer-service conversations in two domains: retail (orders, returns, exchanges) and airline (bookings, changes, cancellations). Each domain comes with a policy document and a set of tools, and a simulated customer with a goal. We used 50 tasks per domain, 3 trials each: 300 conversations per run.

• The agent was a Ragwalla assistant-backed agent running Claude Sonnet 5.5, with τ-bench's own agent prompt and τ-bench's tools served to it over MCP.
• The customer was also Sonnet 5.5. The judges that score the conversations were Claude Opus 5.5.
• The decision model ran in shadow mode. Beside every model call that offered tools, Ragwalla asked the decision model the same question as a choice: which of these tools should the agent call next, or should it reply (__reply__)? Nothing acted on the answer. Ragwalla recorded it next to what Sonnet actually did.

The decision model saw the same conversation Sonnet saw, including the policy and tool results, and each tool's name and description. It did not see the tools' argument schemas. Each question offered about 16 options.

We tested three decision models, one run each, with everything else held constant:

• Clef-flash (@cf/cloudflare/clef-flash), a Workers AI decision model.
• Jev (typesafe/jev), Typesafe's decision model.
• gpt-6-luna, a general chat model. Ragwalla now lets any chat model answer decisions through structured output, so we could compare a general model against purpose-built ones on the same footing.

What would count as a pass

We set the rule before the first run. A decision model passes if:

1. there is a confidence threshold above which it agrees with the large model 99% of the time, and that threshold still covers at least 30% of all decisions; and
2. its confidence is calibrated: in every confidence bin, its actual agreement is within 5 points of its stated confidence.

The first condition is what would make the decision model useful. The second is what would make it safe: a system that acts on confidence needs confidence that means what it says.

Agreement here means matching Sonnet, not being right. Sonnet passes about 88% of these conversations, so it is a strong reference but not ground truth.

Results

All three runs completed with no decision-model failures: 3,195 shadowed calls for Clef-flash, 3,176 for Jev and 3,124 for gpt-6-luna.

	Clef-flash	Jev	gpt-6-luna
agreement with Sonnet	42.3%	74.7%	71.8%
when Sonnet called a tool	54.6%	67.5%	62.4%
when Sonnet replied	27.5%	83.3%	82.9%
Sonnet's choice in the top 2	62.4%	87.6%	79.1%
Sonnet's choice in the top 3	70.5%	92.0%	79.9%
99% agreement first reached at	confidence ≥ 0.94, on 0.2% of calls	confidence 1.0, on 10.8% of calls	never
expected calibration error	12.2 pts	5.3 pts	23.6 pts
largest calibration gap	33.1 pts	16.0 pts	41.9 pts
median latency	1,358 ms	164 ms	1,167 ms
input tokens per call	6,128	5,715	5,710
verdict	fail	fail	fail

None of the three passes. They fail in different ways.

Each line sweeps a confidence threshold from high to low. Across is the share of decisions at or above the threshold, and up is how often those decisions agreed with Sonnet. Passing means reaching the shaded corner.

Clef-flash

Clef-flash agrees with Sonnet on 42% of calls. Most of the gap is replies: when Sonnet answered the customer directly, Clef-flash usually picked a tool instead (27.5% agreement on replies). Its confidence is also poorly calibrated, so there is no threshold worth acting on. It reaches 99% agreement only above 0.94 confidence, and only 0.2% of decisions are that confident.

Jev

Jev is the strongest of the three. It agrees on 75% of calls and matches Sonnet's replies 83% of the time. Its calibration error is 5.3 points, under half of Clef-flash's.

It fails the rule on coverage, not on accuracy. When Jev reports a confidence of exactly 1.0 it agreed with Sonnet every time: 344 calls, 10.8% of all decisions. Just below that, at 0.99 and above, it is wrong about once in 34 calls. So the only slice that meets the 99% bar is about a third of the 30% the rule asks for.

Its misses follow a pattern:

• It skips the look-ups the policy requires before acting. It is weakest on search_direct_flight (26%), get_product_details (38%) and get_order_details (56%), which it often skips in favour of replying.
• Its confident misses are jumps ahead. It picks the goal action, such as transfer_to_human_agents, return_delivered_order_items or modify_pending_order_items, while Sonnet is still replying or looking something up.

Sonnet's choice is in Jev's top 3 92% of the time, counting a tie against Jev. That is good, but not good enough to hide the remaining tools from the large model: a top-3 shortlist would still drop the right tool on about one call in 12.

gpt-6-luna

As a picker, gpt-6-luna is close to Jev: 71.8% agreement against 74.7%, and the same accuracy on replies. It is weaker on airline (66.3%) than on retail (76.1%). Its most common miss is choosing transfer_to_human_agents while Sonnet is still replying: 108 calls, 102 of them in airline.

Its confidence is where it falls apart. A chat model gives no calibrated confidence of its own, so Ragwalla measures one from the model's token probabilities. The answer to a choice question is a single token, the option's index. Ragwalla reads the probability of each index from the logprobs returned for that token.

For gpt-6-luna that meant running with reasoning turned off, because OpenAI returns logprobs for this model only then, and reading the top 5 alternatives, the most it allows. Under those conditions it puts nearly all the probability on whichever index it writes:

• 78% of its answers have a confidence of exactly 1.0, and those agree with Sonnet only 78.7% of the time.
• That holds for its reply picks (80.1%) and its tool picks (77.0%) alike.

In each panel, the solid part of a bar agreed with Sonnet and the lighter part disagreed.

No threshold separates its right answers from its wrong ones. For the same reason its second and third choices carry almost no information: they add 7 and 8 points of agreement, where Jev's add 13 and 17.

gpt-6-luna is also about 7× slower than Jev, and Clef-flash about 8× (a median of 1.4 seconds). In shadow mode that doesn't matter, because the call runs beside the large model's. A decision that gates the large model's call would add that latency to every turn.

This is a result about token probabilities as a confidence signal as much as about gpt-6-luna. It is why Ragwalla labels every decision's confidence with its source. calibrated comes from a decision model trained to report it; logprobs is measured from a chat model's tokens. Anything that acts on a confidence should require calibrated.

What we can conclude

• No decision model we tested can take the next-tool decision off the large model today. All three fail the rule.
• Picking and knowing are different skills. Jev and gpt-6-luna pick almost equally well. Only Jev's confidence carries information.
• There is one narrow place where acting looks safe. That is Jev at exactly 1.0 confidence: about 11% of decisions, with no misses on these tasks. It is a starting point, not a feature. Eleven percent of 3,176 calls on two domains is too small and too narrow a sample to act on in production.
• The errors are systematic, so they can be fixed. The misses cluster on the same tools and the same situations: skipping a required look-up, or jumping ahead to the goal action.

Limits of this test

• Agreement is not correctness. Matching Sonnet counts as right even when Sonnet was wrong, and a different, valid choice counts as wrong. Sonnet passed between 82.7% and 90.7% of conversations per domain across these runs.
• The decision models saw less than Sonnet did. They got tool names and descriptions, with descriptions cut to 300 characters, but no argument schemas. Sonnet got full schemas.
• Top 2 and top 3 count ties against the model. When options share a probability, they all take the lowest rank among them. That matters for gpt-6-luna, which gave Sonnet's choice a probability of exactly 0 in 626 of its 882 misses.
• Parallel calls are scored on the first call. When Sonnet called several tools at once, only the first one counts. This lowers agreement by about 0.7 points.
• One agent model, two domains, one benchmark. Customer-service tasks with a written policy are a favourable setting for a decision model. Open-ended agents with dozens of tools would be harder.
• gpt-6-luna was measured under the only conditions where it returns logprobs: reasoning off, top 5 alternatives. With reasoning on it might pick better, but Ragwalla would have no measured confidence to report.

None of these changed the measurements above. Each run reported here ran on one build from start to finish.