If you use Qwen3.8 27B to manage, to orchestrate, do do shallow research, and then make it ask claude you will notice that Qwen can get out of Haiku, Sonnet and Opus almost as good answers... I was researching how good the model asnswers where, I can't get to work with anything but Fable, but Qwen can get good work from any of them.
My theory is that if you make Qwen understand you, Qwen can elaborate as much as needed to get any of those models get good results, because it's filling in details.
So using Qwen to direct claude also get's you amazing results way way cheaper...
Update: let me elaborate more, I read the HN entry about the team of 5 Qwens being close to fable quality... so I embarked to add to my harness (https://github.com/alainnothere/privibe because we all have one right?) and once I had added the orchestrate tool I had to now measure somehow it it's able to improve quality, so I claude and then Qwen to run a experiment to call my harness passing the model to use and ask them to do something, 3 rounds I think and see what quality came out of it... and as per the grading... qwen and claude agreed that the quality of the response comming out the harness using the orchestrate was VERY CLOSE, with opus and fable providing more "in my opinion you should" which matches my experience on why they are better.
One more update: I can't get copilot models to work decently, it may be my way of working... but I can get Qwen to follow what I try to do... if Qwen get's me he can then fire copilot cli and get amazing results by using copilot... bonus points seems like copilot cli is cheaper with the AIC (cost of the requests) so you can make the copilot subscription longer
I do the opposite of this, use Qwen3.8-27B as subagents, Opus 5.5 as orchestrator and planner, never implementer, and it cuts my token usage significantly. So it is using Claude to direct Qwen. Never thought of the other way around, will give this a try soon as possible.
Smart agent to orchestrate cheaper agents for labour.
Dumber agents to be very explicit about asking smarter agents for requests that require deep information.
This is a thing - the SOTA models really light up the more information you can provide them, and humans are too lazy / incapable of doing that in every prompt.
Entirely depends on what you are trying to do.
Edit: you can also use a smart agent to ask another smart agent, obviously, which is good for personal workflows.
Doesn't that muddle intent of the original prompt? If the smarter model didn't understand your intent correctly, the dumber one sure won't and it rephrasing and expanding the prompt will just amplify it's misunderstanding, no?
It could make sense to use a cheap model to decide if you need to run a long, high effort or otherwise costly effort. Then you can take many decisions at low cost, but think hard when it’s time to make an important decision.
You wouldnt normally use a dumb model as a personal agent to get a smarter model to do work, but you may make use of that mechanism in some kind of automation process.
But there are definitely cases.
GPT Pro is an absolutely spectacular model, that almost cannot be used without 'prompt help'. Or rather, it can only be effectively leveraged by very intricate prompting, which requires the help of another agent.
I typically use Claude Code to write prompts for GPT Pro , and cut and paste the prompt in, and let GPT Pro work for 20 minutes, then take the response back to Claude Code and work with Opus to consider the results.
GPT Pro is the pro setting on OAI's consume chat model. It is only available online or API.
It's branded as a separate model, but really it's like an 'ultrathinking' mode of OAI's model.
The details are not public but it's parallel test time compute and multi search, aka the model reasons along different vectors and compares answers.
If 'feels like' a smarter model, even though I'm pretty sure it's a synthetic variation of Astra.
It is by far the 'smartest model' if you were to benchmark it, and many of OAIs released were based on this.
There are few or no benchmark scores on it because it's not suitable for harness testing or coding, it's almost designed to be 'one shot' or 'few shot'.
It takes 8-25 to answer, and if you give it a sufficiently sophisticated prompt, it will think for 1 hour.
It requires OAI's 'pro' account aka the $200/month one.
It's the best open secret in AI, I can't recommend it enough.
Oh wow, as someone who prides themselves on my depth of knowledge on the OpenAI docs, im ashamed I didn't know about this! I figured that Pro was a deprecated feature post 5.6 family release. Didn't realize Mode:Pro was something you could do with 6 and 6.1, thanks!
Generally, you want a different smart agent as they have different strong sides (e.g. call Sol for a review from Opus)
You may use https://github.com/pingdotgg/t3code as a wrapper above claide/codex/pi/... harnesses, or a proxy below (forgot which one, as agent what theo uses for balancing multiple claude subs)
I highly doubt that claim, has it been tested rigorously? I understand mixing different models to save money and time by using cheaper and faster models for certain tasks. But different models just to increase accuracy seems implausible.
Mixing models almost assuredly does increase accuracy in many cases - I do it all day every day and I can only say from experience though not true objective benchmarks.
I use codex/claude code all day every day and built an MCP that allow them to use each other as sub-agents.
I will not allow claude code to touch my code with any design changes without codex auditing it, and visa versa.
I believe this is the case merely because they are different models that will provide varied perspectives, and smart enough to identify issues on each other's work.
I'm not sure if there's research to back this up but I think it's fairly obvious, and also FYI diminishing marginal return - I'm not sure if a '3rd perspective' would be worth it in most cases.
It’s not obvious and I can tell you my experience is the opposite. That is why research is needed, your anecdote does not move the field one inch one way or another.
Something I’ve added is a step where the agent asks itself how confident it is in the solution/implementation - if it isn’t “High” it then has to ask a smarter agent to review.
Not sure it’s the best approach but it has improved results.
Does anyone have any evidence that having the most powerful model being the orchestrator, and other sub agents being progressively less powerful, is optimal? Is this just common sense or it has been measured somehow?
For me it could be that this scheme may make the code itself poorly written, verbose or faulty
I think whether it's bottom-up or top-down, the core principle is still that you use cheap models for grunt work and use frontier models for expertise and direction.
In my setup, the orchestrator also drives a lot of the design and planning, and the less capable models get inputs from the more capable models by responding back up the chain. The more capable models also review everything that is produced by the cheaper models.
I've also had systems that invert this and allow cheaper models to call out to more capable models when they need to, but in my observation I found them quite bad at knowing even when to call out and I find wrapping the cheap models a bit more reliable in that problems tend to get caught before they bubble up to the user.
It sure saves tokens and even the 100$ subscriptions provide massive amounts of dev power. Also fair to note - mixing models is perhaps a good idea anyway as this way the entropy in them mixes and this supposedly improves result quality.
https://www.nature.com/articles/s42256-026-01268-y/ This is probably what you're looking for. TLDR; weaker orchestrator + strong workers scored higher than the reverse. Caveat with that, old paper: "High-capability models: GPT-5, Claude Sonnet 4.5, Gemini-2.5 Pro; low-capability: GPT-5 nano, Claude Sonnet 3.7, Gemini-2.0 Flash."
With that said, there is evidence that we recently crossed over a line for 'best practices' when it comes to systems of multiple agents. Comparing the shapes of 'n agents working collaboratively' vs '1 agent spawning n agents to handle distinct work', Opus 5.5[1] and Opus 5[4] performs better with the former, Sonnet 5.5[2] and Opus 4.8[3] performs better with the ladder.
Thanks for sharing this! I didn't see GPT 6.1 Sol on OpenCode Go, are you using that through some other service? For Muse Spark 1.3 Contributor, I see on openrouter that it is not ZDR. Given Meta's track record of privacy abuses, are you concerned at all about that?
My goal was/is to have a local model as a "stable" part of the system, or at least a part of it that you have full control over, and have it reach out to the cloud for some Opus. In practice, to get results, I operate as you say.
At this point if I could do anything with zero effort or time, I would get Opus 5.5 to make a grand scheme, handoff to the local model, and give it ability to reach out to Opus again.
I do something like this by using old rule-based bot tech as the first layer you hit, and if it hits any situation it does not understand deterministically, it shells out for help to an external "smarter" agent.
That way you can do old school intent matching and rule-based bots that work super well, and only shell out for deeper work.
I just fell Claude "there's an LLM server on this port serving this model, use it as a subagent"
Claude will write some scripts to call the LLM API and use it like any other tool.
I haven't found a pressing need for anything more complex than that. My multi-agent setups tend to be a pile of bespoke scripts per project anyway, an MCP server is only gonna get in the way. But I suppose this is because I self-host the models, if you're using OpenRouter or whatever it may need more complexity than a curl call to an unauthenticated port
So I use Google Gemini to plan or review; otherwise same loop so brain dumping how I get Gemini to Qwen/Gemma/Meta's model I forget the codename for; local agent reads the AI Studio log locally via Google Drive where they store the logs by default in a JSON format
So with the gdrive app for my OS it just looks like a local file path. Had the local agent learn to parse it.
I still prompt in AI studio direct as needed; local agents see the log change but cannot push back to Gemini their result.
Few reasons:
AI Studio is available where ever via browser, no VS Code or my usual vim setup to miss. I can switch between projects there as an idea hits and the agent switches up the local repo path and session history.
AI studio can search web, etc, etc already so my agent is less complicated with addons in general
Also using AI Studio between local agent turns has been my "human in the middle" approach; to have an explicit gap in the process and check each turn.
I only have a 24GB 3090 most models also take longer locally and I never hit the free tier limits with Gemini :D
Though the latest Meta model is twice as fast at 30B for me as other similar size models. 1-2 hour code gen and QA of build errors is now 30-50 minutes to parse latest changes from Gemini and get a successful build. The Meta model is pretty great at "fill in the middle" from a well structured plan.
Overall this loop feels like a normal workday pace. I freelance so I have a lot more say over that.
Custom MCP with a variety of functions, I have different Qwen profiles with a variety of subagent numbers and cache pools, my default is 192k pool with 64k initial context that can grow 32k at a time up to 128k, allowing 3x64k, 2x96k or 1x128k and 1x64k subagents. Qwen uses Pi harness. Model I use specifically is ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF
What hardware and inference engine do you use? On my Radeon R9700, at least with llama.cpp and some others, the GPU provides as many tok/sec with a single stream as with concurrent ones, so it's just as fast and supports longer contexts to swap contexts to RAM on turn boundaries.
I've experienced the same. Are you saying you're using different context sizes with parallel invocations or just that it's possible to have longer contexts? I've been setting my config to ensure I don't blow out context server side but have not tried to manage it client side.
Why not using a proxy in claudecode to forward all the claude-* model to anthropic as a standard proxy forwarder and route other models to other providers (openrouter, vllm, ..)?
This way Claude's model can get all the context from other models as all in the same harness.
To use other models than claude-* you can use agents files.
Ps: as you won't use your proxy to distill or alter claudecode requests to anthropic you should be compliant with anthropic rules.
you can easily run qwen locally in a pi harness (or similar) and just tell claude code to delegate all implementation work to it. claude will even setup the harness with the local model
> My theory is that if you make Qwen understand you, Qwen can elaborate as much as needed to get any of those models get good results, because it's filling in details.
How is this different than just telling the model you’re using to fill in the gaps itself?
If you’re not manually reviewing the intermediate hop, how does that add to the “understanding you”?
there is no difference... each case may be different, I have a local qwen gpu setup... and is fast... but not as fast as claude.
Fable is amazing and if it cost the same as my qwen setup I'll be running fable and never had qwen in first place.
But Fable is crazy expensive, Qwen is not as fast but cheaper... Haiku is fucking dumb IN MY HANDS... BUT my local qwen is quick enough to be briefed, get the idea and delegate.
I work with Fable mostly delegating to opus/sonnet and it's cheaper than just fable, I can't stand working with opus or the other models.
Why Qwen directing the others?
For the things claude refuse, Qwen can make very specific questions and still get an answer without triggering the thing.
I am using Haiku 5 and it’s amazing for the cost. I almost always use it instead of Opus5.5 for simpler tasks. In my experience it’s much better than the Qwen models!
I think that his idea is that a lot of preparation / setup work is boiler plate / brain dead stuff that can be done by a shallower model.
It's like, find the relevant scientific papers by using Google search and the scientific paper MCPs, summarizing stuff, etc. Then the big model can come in and actually think on the artifacts
Oh man I’m been playing with little -> big vs big -> little sub-tasking a bunch.
My sense so far is it’s very much about problem domain “taste” and there is no one correct answer (that is token cost efficient).
Does the smartest agent need the full flavor of the context of the problem to make real progress (i.e what’s been tried, how you’ve influenced the approach) or does the smartest agent really need a crisp problem definition and context room to really work.
> Oh man I’m been playing with little -> big vs big -> little sub-tasking a bunch.
I too wonder the same...
The real-world provides us many examples of such big -> little, or smart -> not-as smart workflows. For instance, we don't typically ask juniors or mid-level SDEs to design solutions to exceedingly challenging problems for a reason. However there are _some_ areas that those with less experience can tackle that is relatively risk-free if their work is reviewed by a more experienced engineer, which leads me to a conclude that it's task dependent.
I suspect that a lot of the fudging around with MCPs and tools can be delegated to smaller agents. Astra doesn't need to look at 300 websearch results directly or to extract text out of raw HTML
Absolutely no matter whether you generally go big small or small big, you should always be like “ask haiku / Luna to check all X” or “see if there are other ways to Y”
Anthropic has a native solution to this which is that you can use `/advisor` to adjust config to enable using Sonnet but set Opus as an advisor or use Haiku and then set Sonnet or Opus as advisor that the weaker model will automatically call when it thinks something warrants it:
Everyone is discussing the tech, but you do realize that they made mapping data to real people easier using AI? This is essentially AI-assisted privacy violation, 4 times cheaper.
People, please try to understand what you are building. Don't just build it as asked!
Kind of offtopic, but I was kind of horrified by the rather blasé presentation of massive privacy invasion at scale, too. But there's clearly no stopping it technically, it will have to be handled by legislation, and the only hope of that happening is some kind of huge scandal.
Yes and fission reactions can be used both for energy & destruction, but that doesn't mean that the applications of working in defense or powerplants are morally equal.
> Everyone is discussing the tech, but you do realize that they made mapping data to real people easier using AI?
That's Hacker News for you. They have chronic tunnel vision and are unable to look at implications that go beyond the technical. Particular blind spot for techno-social consequences.
What are you talking about, these kind of tech have been available in industry way before (eg hypertargeting / profiling in adtech), and is entirely unrelated to LLMs or Jev.
> This is foundational for any company wishing to do outreach, personalization, or analytics on customers. You can’t know, for instance, the lifetime value of a customer without being able to map all their purchases to a single identity.
Found similar results, but the economics of the models involved have changed so quickly that it becomes so task specific on what the right thing to do was.
With an assemblage of models like Jev + Opus or Jev + Qwen, Jev would often do the right thing when deciding to escalate to another model - I was impressed that it would usually make the right call, especially if you gave it good awareness of what the other model capabilities were. "ESCALATE to opus if you need good vision awareness".
When Haiku 5.5 came out though, for my tasks, the cost differences disappeared almost immediately. Haiku was good enough / fast enough for so many things that introducing a decider layer didn't offer huge wins. Jev's "millisecond" level performance was nice but would get dwarfed by other latencies in my chain ( web browsing ), that it didn't make a measurable difference in my tasks.
This made me think... Is anyone using small models to generate candidates and let a more expensive model pick one? I.e. genetic programming on neural networks? Perhaps a decision model could be used for the picking?
Not hard to test. I wired a “judge” into my existing Jev setup and… it works. I was able to trim my prompt down to the basics without “engineering” and have my entire test suite pass. Best I could get before was 1 failure, with an extremely brittle prompt. Like, adding a comma where grammatically appropriate broke 10 more tests.
This sort of stuff has me believing that decision model controlled/prioritized context is the future of context management beyond the raw AGENTS.md file.
Why give all the context to an agent, when the context can also be curated, updated and made concise automatically for each new problem.
I too feel like context management has a lot of room for improvement, but the problem right now is that dynamic context management will most likely tank your cache hit rate. We need caching that's not just based on prefix-matching for this to make sense.
My bandaid fix for this is having a large number of sessions or "agents" and being ok with them being inactive at times, but they maintain their context. This has to be in some kind of system with appropriately sized "departments" to justify that agent's existence, or to justify it splitting into two, or creating sub-agents, etc. Yes, there is a cost when you cold start an agent after a week, but for now I'd rather pull that whole 400k+ context back without layers and layers of /compact. And then hopefully you keep that agent going for long enough to benefit from any cache-hit discounts.
Not really a good analogy, but it feels like having an image you like but every time you view it, it runs it through another loop of jpg compression. Use it when it's most effective, otherwise let it preserve it's details.
And as I see it that is impossible with current autoregressive global attention models because each part depends on the one before. Maybe if we can cache the local attention in blocks and only recompute the global parts.
It's honestly surprising how little work gets put into building harnesses which carefully steer the LLM through sophisticated context curation. Probably due to the belief that we shouldn't care much because AGI is just around the corner.
Related (but shameless) plug : I've actually been hacking a project over the past two weeks to address this very issue. I'd like to create an SDK which embeds the evaluation of LLMs into code seamlessly, and does "behind the curtains" context curation, to really enable "hybrid" code/LLM agentic applications. (if you're curious => https://github.com/VKCR/semiotique)
Yes × 1000...
Jev is the classic type of scientific discovery, any human can see it's effectiveness without actually trying it. We were only deprived the discovery because of the incredible noises created by the AGI/ASI ferver....
We have enjoyed a lot of privacy because of a lack of ability to violate it.
I mean that there was always the structural incentive to completely ignore privacy. It just wasnt technically feasible.
I wouldn’t assume any structural limit to what a government, corporation, etc wants to know about you. The day reading someone’s mind is possible is the day you realize the world doesn’t recognize any sort of right to private thought.
Does the public have an incentive to read the government's mind?
Or maybe it's more about capability.
XKCD 1390: I mean, it's not like we could just demand to see the code that's governing our lives. What right do we have to poke around in Facebook's private affairs like that?
It's illustrative that there are even errors in how the humans have formulated the task and documented the errors.
They propose:
Valentino → Female Child
Yet this is obviously a boy name, else it would be Valentina. Is this model going to return confidence values? No, they want decisions. This is an inherent problem in AI with reductive logic for these tasks.
Likewise, dates. Is 17/08/22:
- 20170822 (Chinese write the year first, and in speech and chat they will say '17' not '2017'.)
- 20220817 (Everywhere But North America.)
Is 04/06/2009:
- 20090604 (Non-North American)
- 20090406 (American)
Without knowing the cultural context, the location, the document type, the document author, etc, you could be getting it wrong.
It’s an interesting problem. And i have following thoughts to it?
- Are all cases even decidable?
- Shouldn’t one false positive rank higher than a hundred rank negative.
-Shouldnt’t the AI just abstain from decisions to avoid false positives.
- Is it smarter to seperate this in a lot of small sub problems and to define this as a rule set of small yes no decicions, why does this need to be solved agentically?
- You should definitely clean up deterministically first.
I feel like if you really want to solve this problem, you would approach it differently. This seems like a wrong approach and a bad article on top?
> -Shouldnt’t the AI just abstain from decisions to avoid false positives.
Yes, and I think regulation should be quite strong in this regard.
> - Is it smarter to seperate this in a lot of small sub problems and to define this as a rule set of small yes no decicions, why does this need to be solved agentically?
For larger contexts I worry about agentic confabulation, but humans unquestionably look at the gestalt when doing matching. I don't want idiot clankers making enforced decisions because they have misunderstood one field.
> - You should definitely clean up deterministically first.
If you can't do field validation on entry then I'm not sure deterministic rules work better, they may be worse, although easier to reason about.
But surely there are some deterministic reasons to clean up.
I personally wouldn’t lean towards LLMs on identity resolution. I would like give it the dataset and then ask me questions about specs? I really struggle to find the LLM part reasonable here. I mean potentially some typos, or nicknames, but those could be potentially resolved deterministically also.
I would probably have reviewer running that suggests new rules that I could implement. To me this seems more like compiler work than LLM work.
That is not what they say. There are errors in the data, the data does not have the real name it has “female child” in instead.
Perhaps there’s something else in the post that you’re talking about but please do take the time to actually read what’s there, I’m only telling you what’s written in the text around the problem you think you’ve found.
>Is this model going to return confidence values? No, they want decisions.
The system doesn't return confidence values because that's not what they are designing. The models, however, do and to me it's interesting that Jev's measure of confidence is pretty well calibrated to its correctness.
>Without knowing the cultural context, the location, the document type, the document author, etc, you could be getting it wrong.
And? There's no perfect process, just ones that asymptotically approach correctness at exponentially increasing cost. They own the system and get to decide what works for them.
Exactly, there is no perfect process for such simplistic binary decision making. But we are seeing pipelines for tasks, like document processing, where it is becoming accepted that the machine is right.
Rather than binary classification we should be locating document fragments in a high dimensional feature space, and with all those fragments coming to a confidence bounded guess.
Additionally, when someone submits documents that can't be recognised as valid, the response should not be INVALID, it should be UNDECIDED. In that sense, passing to Opus is great, but passing a tiny fragment of the problem is a bad way to do it.
Excellent read. I use the same hybrid approach for a lot of security engineering work and it’s working well after initial calibrations, for example for our own harness we rebuilt auto mode exactly like this to judge tool calls and intent.
I had the same idea, I extended opencode-warden to use a local decision model to validate tool inputs and outputs. It seems to work... ok? d1 is fast enough to run locally on an old 1080 GTX. Not sure if I can recommend relying on it.
Let me make sure I'm getting this: They asked the models for answers but also for confidence score.
Jev's confidence score was more reliable. Which means that they were able to use Jev despite it having lower accuracy, because of the confidence was below a cut off, they would escalate to an expensive model (Opus)?
Whereas Luna, even though it was more accurate than Jev, its confidence was way off, so they couldn't reliably know when to fall back?
There is an error in your interpretation of calibration graphs and the conclusion that Jev is more calibrated than Luna is likely incorrect. Or at least the graphs aren't enough to conclude either way.
I don't have the actual data but based on the graphs I estimate the expected calibration error to be 2-3% lower for Luna on both the tasks.
Its true that Luna seems to give lots of high confidence values but it's also correct more. Higher number of mistakes at higher confidence isn't itself undesirable if its pass rate is also high. It'd be good if they reported actual ECE.
>Most of Jev's mistakes are closer to 50% confidence, which is what you'd expect out of a well-calibrated model.
Ideally, "system one" intelligence is good both at quick reactions and at knowing when to hold off and wait for "system two." Sure enough, that works with Jev.
I don't usually get excited about the high-level, "whoooa man" marketing-speak, but the parallel is striking.
Where is the Jev that can replace 20 Opus sessions asking each other for help in a coordinated way? I want to shift towards a local model that "asks for help" in an effective way, but it still seems a bit out of reach. I mean, you could build your own 27B or 35B or A3B or bumblebee and spend a week or two carefully forming it into something useful, but it feels like that would be too specific and it would be thrown out when something like qwen4 is released. Maybe not, I don't know. Opus still exists as it is, in a "subsidized" form that's hard to step down from.
Jev is pretty cool for augmenting llms in a lot of ways. We use it in our own agent orchestrator[1] to help route ideas/prompts to running agent sessions. I find when I have a lot of parallel agents running its hard to reason about which agent I should be talking to and Jev makes that call for me.
I wonder if something like this is the undisclosed new approach from Anthropic that presumably has massive improvements by combining multiple models. Leveraging decision models as much as possible seems logical for extra speed and token efficiency.
I think OpenAI and Anthropic have both been doing similar "routing" approaches for a while. Wasn't OpenAI one of the first to have a small model start responding to you for the first several tokens so that you got an instant response while the big model became available?
Similarly, Grok has been routing to Opus internally for a little while now I think.
There’s a METR audit document which apparently describes various unreleased models. This is discussed in a YT video by Wes Roth. I have no firsthand knowledge about it.
The most glaring downside with practically using decision models is if you need the model to not only decide something but give a reason, for every case, you end up not saving money or time, but just adding an extra step.
Give me a decision model that can do both, more expensive than today’s decision models but still cheaper than full on frontier model, and it’d open up so many more use cases, IMO
Would it make sense to add a second decisions after the first with predefined options for why the model chose what it chose? I guess this could work for some cases
It would make more sense to have the decision model check for each possible reason whether it applies in the current case and then have the resulting consequences determined by a bog-standard if-then-else program.
Why isn't Jev on Bedrock asap... I did a demo for using it as an orchestrator but the problem is upper management won't approve it unless its on Bedrock which is a real bummer
Or a pigeon pecking at a checkerboard with randomly scattered bird feed.
On a serious note: all the heat and light is about AI for automating coding tasks, but a significant amount of management is just theater about deliverables. So much of that could be replaced by better product delivery pipelines that can accurately generate the "when can I expect this release?" without a layer of management that considers sitting in zoom calls all day work.
I agree. What I’ve encountered though is even building enough cases for a calibration data set is hard. And so we farm out to smarter expensive models like Astra to label cases and use that as a calibration data set for the now cheaper LLM judge actually running as our judge
Is this standard ? Or am I just too lazy to manually label enough data to calibrate ?
Yep. It's actually kind of mind blowing how well "reasoning" works at de-biasing and pushing out of global minima, in hindsight, when you see how jagged things get again without.
These are the results of a little performance tuning/permutation testing run against context format and question wording we did. Obviously there is a lot of noise here but I was actually kinda surprised that there was a clear "best way to ask the question" even for relatively low N and low complexity/ambiguity of the task.
(This is a ~30s Ai-driven gif presentation of the results, the full grid shows up about 15s in): https://imgur.com/a/wGFZvGw
These models in particular have some pretty dubiously generalized and diverse training datasets. You can see this for yourself BTW by looking at the datasets SOTA Jev-like models are being trained against and evaluated on. No knock against the teams btw, I just think it's rather demystifying and makes me quite optimistic at how much better these models are going to get over the next few years: https://labs.baseten.co/articles/mammoth-1-and-elephant-1-fr...
The bar charts in this post really warped my head. If you’re interested in demonstrating confidence calibration you shouldn’t visualize the number of mistakes in a bar chart. Use a classifier calibration curve or something, but don’t just show me the counts in 10 percentage point buckets.
This is the Model Router architecture, from another angle. The problem is always, how does a dumb model know it needs to escalate to a smart model?
Some problems are obviously hard known-unknowns (eg “solve Navier Stokes”). The problem is unknown-unknowns or knowledge gaps, where the model doesn’t know they need help.
It's not a problem you can get right, because you can't know if you need to escalate until after you escalate (and what you escalated to disagrees with what your first choice would have been). But that's the point: how cautious you make the model is the knob that lets your trade cost for correctness.
I don't believe the answer is ever "how certain is the model?" It's "how much is at stake?" Or "how hard of a problem does this appear to be?" It's not about teaching the model to have uncertainty, but rather to be avoidant of things that are sufficiently important to get right.
We've been doing this for a while. Fine tuning the decision model helps a lot too. The problem is you still need a pretty low confidence threshold (e.g 10%), but often times that reduces the work by 70% with 0% quality loss (in our benchmarks)
Like a comment on another thread, 0.25x would be the much better way to say this. It's 25% of the original value. It's also consistent: If 3x means 300% the original value, then 0.25x is 25% of the original value.
If you say 4x lower, that changes the semantics for lower than x vs higher than x, which is absurd.
If you are looking at huge categorizing and parsing work, using a local decision model and passing it to Haiku or Luna can really lower your token cost.
their cost numbers give you the hard set escalation rate even though they don't list it, (2.94 - 0.06) / 5.39 is ~53% going to Opus vs ~22% on the easy set, same thresholds
Seems to me that the important factor is that Jev has well-calibrated confidence for this task. Which secondary expensive model it defers to is probably less critical in this technique as long as that model has higher performance than Jev.
i'm so numb to these headlines nowadays... i guess i've been disappointed by how effective these approaches end up being for actual work but i feel like it's easy to produce these headlines with "proof" that doesnt prove anything besides the sandboxed scenario it was run in
I use swival.dev where a decision model (Cloudflare's Clef running locally is free and works great) is used for automatic model selection, but also to complete a task, just like the LLM. It gives confidence scores on the reliability of the LLM output, is used instead of the LLM, even mid-task when appropriate, etc.
And yes, that saves money since less tokens are sent to a remote LLM.
Memdoor author here which is a coding agent that run Jev inside. I believe this is becoming mainstream in any coding agent for routing calls and reduce input sizes. First candidates are edit tools for reading file or jgrep for greping large files folder etc.. The Token saving is mostly on the read side.
If we compare coding agents with Jev on vs off we get those numbers (metered with Memdoor):
- read-heavy tasks (coding questions , logs tail etc..): -63% to -65% input tokens, 40-45% fewer model calls, on GLM 5.3 half the wall time. Reproduced a month apart on two models (DeepSeek V4.1 Flash, GLM 5.3).
- a file edit: -23%
Two things match the post. Calibration is what makes it safe: I apply a decision at confidence >= 0.75 and fail below this threshold or on timeout or on an unknown tool family. The dollar saving scales with the price of the model being saved: on a $0.04/M model Jev costs what it saves so on a $0.38/M model it's negligible but on Claude model for example it can be outstanding.
Where it did not help: giving Jev more context about the repo. I found same thing as the post.
If you use Qwen3.8 27B to manage, to orchestrate, do do shallow research, and then make it ask claude you will notice that Qwen can get out of Haiku, Sonnet and Opus almost as good answers... I was researching how good the model asnswers where, I can't get to work with anything but Fable, but Qwen can get good work from any of them. My theory is that if you make Qwen understand you, Qwen can elaborate as much as needed to get any of those models get good results, because it's filling in details. So using Qwen to direct claude also get's you amazing results way way cheaper...
Update: let me elaborate more, I read the HN entry about the team of 5 Qwens being close to fable quality... so I embarked to add to my harness (https://github.com/alainnothere/privibe because we all have one right?) and once I had added the orchestrate tool I had to now measure somehow it it's able to improve quality, so I claude and then Qwen to run a experiment to call my harness passing the model to use and ask them to do something, 3 rounds I think and see what quality came out of it... and as per the grading... qwen and claude agreed that the quality of the response comming out the harness using the orchestrate was VERY CLOSE, with opus and fable providing more "in my opinion you should" which matches my experience on why they are better.
One more update: I can't get copilot models to work decently, it may be my way of working... but I can get Qwen to follow what I try to do... if Qwen get's me he can then fire copilot cli and get amazing results by using copilot... bonus points seems like copilot cli is cheaper with the AIC (cost of the requests) so you can make the copilot subscription longer
I do the opposite of this, use Qwen3.8-27B as subagents, Opus 5.5 as orchestrator and planner, never implementer, and it cuts my token usage significantly. So it is using Claude to direct Qwen. Never thought of the other way around, will give this a try soon as possible.
Both of those orchestrations will work for sure.
Smart agent to orchestrate cheaper agents for labour.
Dumber agents to be very explicit about asking smarter agents for requests that require deep information.
This is a thing - the SOTA models really light up the more information you can provide them, and humans are too lazy / incapable of doing that in every prompt.
Entirely depends on what you are trying to do.
Edit: you can also use a smart agent to ask another smart agent, obviously, which is good for personal workflows.
Doesn't that muddle intent of the original prompt? If the smarter model didn't understand your intent correctly, the dumber one sure won't and it rephrasing and expanding the prompt will just amplify it's misunderstanding, no?
It could make sense to use a cheap model to decide if you need to run a long, high effort or otherwise costly effort. Then you can take many decisions at low cost, but think hard when it’s time to make an important decision.
You wouldnt normally use a dumb model as a personal agent to get a smarter model to do work, but you may make use of that mechanism in some kind of automation process.
But there are definitely cases.
GPT Pro is an absolutely spectacular model, that almost cannot be used without 'prompt help'. Or rather, it can only be effectively leveraged by very intricate prompting, which requires the help of another agent.
I typically use Claude Code to write prompts for GPT Pro , and cut and paste the prompt in, and let GPT Pro work for 20 minutes, then take the response back to Claude Code and work with Opus to consider the results.
Opus is 'dumber' (nominally) then GPT Pro.
What is GPT Pro?
GPT Pro is the pro setting on OAI's consume chat model. It is only available online or API.
It's branded as a separate model, but really it's like an 'ultrathinking' mode of OAI's model.
The details are not public but it's parallel test time compute and multi search, aka the model reasons along different vectors and compares answers.
If 'feels like' a smarter model, even though I'm pretty sure it's a synthetic variation of Astra.
It is by far the 'smartest model' if you were to benchmark it, and many of OAIs released were based on this.
There are few or no benchmark scores on it because it's not suitable for harness testing or coding, it's almost designed to be 'one shot' or 'few shot'.
It takes 8-25 to answer, and if you give it a sufficiently sophisticated prompt, it will think for 1 hour.
It requires OAI's 'pro' account aka the $200/month one.
It's the best open secret in AI, I can't recommend it enough.
Oh wow, as someone who prides themselves on my depth of knowledge on the OpenAI docs, im ashamed I didn't know about this! I figured that Pro was a deprecated feature post 5.6 family release. Didn't realize Mode:Pro was something you could do with 6 and 6.1, thanks!
>you can also use a smart agent to ask another smart agent
What's the best method to have a 'smart' agent just.. ask itslef?
Generally, you want a different smart agent as they have different strong sides (e.g. call Sol for a review from Opus)
You may use https://github.com/pingdotgg/t3code as a wrapper above claide/codex/pi/... harnesses, or a proxy below (forgot which one, as agent what theo uses for balancing multiple claude subs)
I highly doubt that claim, has it been tested rigorously? I understand mixing different models to save money and time by using cheaper and faster models for certain tasks. But different models just to increase accuracy seems implausible.
Mixing models almost assuredly does increase accuracy in many cases - I do it all day every day and I can only say from experience though not true objective benchmarks.
I use codex/claude code all day every day and built an MCP that allow them to use each other as sub-agents.
I will not allow claude code to touch my code with any design changes without codex auditing it, and visa versa.
I believe this is the case merely because they are different models that will provide varied perspectives, and smart enough to identify issues on each other's work.
I'm not sure if there's research to back this up but I think it's fairly obvious, and also FYI diminishing marginal return - I'm not sure if a '3rd perspective' would be worth it in most cases.
It’s not obvious and I can tell you my experience is the opposite. That is why research is needed, your anecdote does not move the field one inch one way or another.
Something I’ve added is a step where the agent asks itself how confident it is in the solution/implementation - if it isn’t “High” it then has to ask a smarter agent to review.
Not sure it’s the best approach but it has improved results.
Just ask. The agents already know how. To boost it, have it read the exact docs that OAI/Anthropic provide.
Asking Claude to make prompts is like a Math teacher stuck working at a cafe about Math problem, the just light up.
They are very good at at it.
This is how I do it too. I run three levels:
As long as you have an appropriate project and sandbox for Muse Spark it's the real secret sauce for my setup and it's dirt cheap on OpenCode Go.GPT 6.1 Sol is surprisingly efficient too, I've been impressed with it.
My orchestration skill setup is https://github.com/beefsack/processed-beef/
Just one thing
Does anyone have any evidence that having the most powerful model being the orchestrator, and other sub agents being progressively less powerful, is optimal? Is this just common sense or it has been measured somehow?
For me it could be that this scheme may make the code itself poorly written, verbose or faulty
I think whether it's bottom-up or top-down, the core principle is still that you use cheap models for grunt work and use frontier models for expertise and direction.
In my setup, the orchestrator also drives a lot of the design and planning, and the less capable models get inputs from the more capable models by responding back up the chain. The more capable models also review everything that is produced by the cheaper models.
I've also had systems that invert this and allow cheaper models to call out to more capable models when they need to, but in my observation I found them quite bad at knowing even when to call out and I find wrapping the cheap models a bit more reliable in that problems tend to get caught before they bubble up to the user.
It sure saves tokens and even the 100$ subscriptions provide massive amounts of dev power. Also fair to note - mixing models is perhaps a good idea anyway as this way the entropy in them mixes and this supposedly improves result quality.
https://www.nature.com/articles/s42256-026-01268-y/ This is probably what you're looking for. TLDR; weaker orchestrator + strong workers scored higher than the reverse. Caveat with that, old paper: "High-capability models: GPT-5, Claude Sonnet 4.5, Gemini-2.5 Pro; low-capability: GPT-5 nano, Claude Sonnet 3.7, Gemini-2.0 Flash."
With that said, there is evidence that we recently crossed over a line for 'best practices' when it comes to systems of multiple agents. Comparing the shapes of 'n agents working collaboratively' vs '1 agent spawning n agents to handle distinct work', Opus 5.5[1] and Opus 5[4] performs better with the former, Sonnet 5.5[2] and Opus 4.8[3] performs better with the ladder.
[1] https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba50242199...
[2] https://assets.claude.com/870c8f525702625d2c62fc6dd04c857e32...
[3] https://assets.claude.com/0f0c97ad20d8005706296bd92aa1c27c6b...
[4] https://assets.claude.com/ceaf5c7ff2783855203fde8208ec311252...
Thanks for sharing this! I didn't see GPT 6.1 Sol on OpenCode Go, are you using that through some other service? For Muse Spark 1.3 Contributor, I see on openrouter that it is not ZDR. Given Meta's track record of privacy abuses, are you concerned at all about that?
And which agent checks the work that the Worker does? If it's garbage then isn't that just extra tokens consumed by the Orchestrator or Feature lead?
I'd be interested to know what the difference in cost is in this setup on building a complex feature vs using Opus directly.
My goal was/is to have a local model as a "stable" part of the system, or at least a part of it that you have full control over, and have it reach out to the cloud for some Opus. In practice, to get results, I operate as you say.
At this point if I could do anything with zero effort or time, I would get Opus 5.5 to make a grand scheme, handoff to the local model, and give it ability to reach out to Opus again.
I do something like this by using old rule-based bot tech as the first layer you hit, and if it hits any situation it does not understand deterministically, it shells out for help to an external "smarter" agent.
That way you can do old school intent matching and rule-based bots that work super well, and only shell out for deeper work.
What do you use to get Opus to call Qwen?
I just fell Claude "there's an LLM server on this port serving this model, use it as a subagent"
Claude will write some scripts to call the LLM API and use it like any other tool.
I haven't found a pressing need for anything more complex than that. My multi-agent setups tend to be a pile of bespoke scripts per project anyway, an MCP server is only gonna get in the way. But I suppose this is because I self-host the models, if you're using OpenRouter or whatever it may need more complexity than a curl call to an unauthenticated port
Nah, OpenRouter is the same API as self hosting, but you need to provide your API key. It's as easy as a URL swap
So I use Google Gemini to plan or review; otherwise same loop so brain dumping how I get Gemini to Qwen/Gemma/Meta's model I forget the codename for; local agent reads the AI Studio log locally via Google Drive where they store the logs by default in a JSON format
So with the gdrive app for my OS it just looks like a local file path. Had the local agent learn to parse it.
I still prompt in AI studio direct as needed; local agents see the log change but cannot push back to Gemini their result.
Few reasons:
AI Studio is available where ever via browser, no VS Code or my usual vim setup to miss. I can switch between projects there as an idea hits and the agent switches up the local repo path and session history.
AI studio can search web, etc, etc already so my agent is less complicated with addons in general
Also using AI Studio between local agent turns has been my "human in the middle" approach; to have an explicit gap in the process and check each turn.
I only have a 24GB 3090 most models also take longer locally and I never hit the free tier limits with Gemini :D
Though the latest Meta model is twice as fast at 30B for me as other similar size models. 1-2 hour code gen and QA of build errors is now 30-50 minutes to parse latest changes from Gemini and get a successful build. The Meta model is pretty great at "fill in the middle" from a well structured plan.
Overall this loop feels like a normal workday pace. I freelance so I have a lot more say over that.
Custom MCP with a variety of functions, I have different Qwen profiles with a variety of subagent numbers and cache pools, my default is 192k pool with 64k initial context that can grow 32k at a time up to 128k, allowing 3x64k, 2x96k or 1x128k and 1x64k subagents. Qwen uses Pi harness. Model I use specifically is ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF
What hardware and inference engine do you use? On my Radeon R9700, at least with llama.cpp and some others, the GPU provides as many tok/sec with a single stream as with concurrent ones, so it's just as fast and supports longer contexts to swap contexts to RAM on turn boundaries.
I've experienced the same. Are you saying you're using different context sizes with parallel invocations or just that it's possible to have longer contexts? I've been setting my config to ensure I don't blow out context server side but have not tried to manage it client side.
RX 7900 XT (20GB VRAM) with a custom ninfer engine I built through long Claude Code sessions
Why not using a proxy in claudecode to forward all the claude-* model to anthropic as a standard proxy forwarder and route other models to other providers (openrouter, vllm, ..)? This way Claude's model can get all the context from other models as all in the same harness. To use other models than claude-* you can use agents files.
Ps: as you won't use your proxy to distill or alter claudecode requests to anthropic you should be compliant with anthropic rules.
Probably depends if you're trying to treat it as "subagent(s)" or as two different models working on the same stream of thought/context.
Both are, I think, valid options. Depends on the workflow you're looking for.
Try this out
https://tysonberry-wogan.com/command-and-control-by-tyson/
Install the llm cli and tell opus to use it to prompt qwen.
you can easily run qwen locally in a pi harness (or similar) and just tell claude code to delegate all implementation work to it. claude will even setup the harness with the local model
You can have a setup like this with OpenCode agents, where you can define custom top-level agents, subagents, and assign specific models to each.
https://opencode.ai/docs/agents/
There are a lot of ways to do this if you stop using Big Ai harnesses that make it difficult to use models besides theirs.
ohmypi will spawn subagents across different providers out of the box. Opus will spawn subagents running on my local ds4 installation just fine
> My theory is that if you make Qwen understand you, Qwen can elaborate as much as needed to get any of those models get good results, because it's filling in details.
How is this different than just telling the model you’re using to fill in the gaps itself?
If you’re not manually reviewing the intermediate hop, how does that add to the “understanding you”?
there is no difference... each case may be different, I have a local qwen gpu setup... and is fast... but not as fast as claude. Fable is amazing and if it cost the same as my qwen setup I'll be running fable and never had qwen in first place. But Fable is crazy expensive, Qwen is not as fast but cheaper... Haiku is fucking dumb IN MY HANDS... BUT my local qwen is quick enough to be briefed, get the idea and delegate. I work with Fable mostly delegating to opus/sonnet and it's cheaper than just fable, I can't stand working with opus or the other models.
Why Qwen directing the others? For the things claude refuse, Qwen can make very specific questions and still get an answer without triggering the thing.
I am using Haiku 5 and it’s amazing for the cost. I almost always use it instead of Opus5.5 for simpler tasks. In my experience it’s much better than the Qwen models!
Haiku 5.5 xhigh/max is surprisingly capable; it just uses too much context.
I think that his idea is that a lot of preparation / setup work is boiler plate / brain dead stuff that can be done by a shallower model.
It's like, find the relevant scientific papers by using Google search and the scientific paper MCPs, summarizing stuff, etc. Then the big model can come in and actually think on the artifacts
Big model can easily delegate to luna to do this task.
I ask task x -> big model thinks and delegate to luna heavy dumb work.
Oh man I’m been playing with little -> big vs big -> little sub-tasking a bunch.
My sense so far is it’s very much about problem domain “taste” and there is no one correct answer (that is token cost efficient).
Does the smartest agent need the full flavor of the context of the problem to make real progress (i.e what’s been tried, how you’ve influenced the approach) or does the smartest agent really need a crisp problem definition and context room to really work.
> Oh man I’m been playing with little -> big vs big -> little sub-tasking a bunch.
I too wonder the same...
The real-world provides us many examples of such big -> little, or smart -> not-as smart workflows. For instance, we don't typically ask juniors or mid-level SDEs to design solutions to exceedingly challenging problems for a reason. However there are _some_ areas that those with less experience can tackle that is relatively risk-free if their work is reviewed by a more experienced engineer, which leads me to a conclude that it's task dependent.
I suspect that a lot of the fudging around with MCPs and tools can be delegated to smaller agents. Astra doesn't need to look at 300 websearch results directly or to extract text out of raw HTML
Absolutely no matter whether you generally go big small or small big, you should always be like “ask haiku / Luna to check all X” or “see if there are other ways to Y”
Anthropic has a native solution to this which is that you can use `/advisor` to adjust config to enable using Sonnet but set Opus as an advisor or use Haiku and then set Sonnet or Opus as advisor that the weaker model will automatically call when it thinks something warrants it:
https://code.claude.com/docs/en/advisor
It allegedly will automatically judge when to call the stronger model, not sure how much instructions in prompts or AGENTS.md can steer it.
The "oh my pi" (omp) harness has this also.
Thoughts on gemma?
Add this to the things I never imagined when thinking about what a world with artificial intelligence might look like.
Everyone is discussing the tech, but you do realize that they made mapping data to real people easier using AI? This is essentially AI-assisted privacy violation, 4 times cheaper.
People, please try to understand what you are building. Don't just build it as asked!
Kind of offtopic, but I was kind of horrified by the rather blasé presentation of massive privacy invasion at scale, too. But there's clearly no stopping it technically, it will have to be handled by legislation, and the only hope of that happening is some kind of huge scandal.
You also need that for things like merging paper hospital records when you digitise them.
Yes and fission reactions can be used both for energy & destruction, but that doesn't mean that the applications of working in defense or powerplants are morally equal.
> Everyone is discussing the tech, but you do realize that they made mapping data to real people easier using AI?
That's Hacker News for you. They have chronic tunnel vision and are unable to look at implications that go beyond the technical. Particular blind spot for techno-social consequences.
Hacker News is you, too.
It's just named entity linking. A classic NLP problem.
What are you talking about, these kind of tech have been available in industry way before (eg hypertargeting / profiling in adtech), and is entirely unrelated to LLMs or Jev.
I am talking about the article, which is applying LLMs and Jev to make those existing toxic patterns more effective and efficient.
> This is foundational for any company wishing to do outreach, personalization, or analytics on customers. You can’t know, for instance, the lifetime value of a customer without being able to map all their purchases to a single identity.
yuck
Found similar results, but the economics of the models involved have changed so quickly that it becomes so task specific on what the right thing to do was.
With an assemblage of models like Jev + Opus or Jev + Qwen, Jev would often do the right thing when deciding to escalate to another model - I was impressed that it would usually make the right call, especially if you gave it good awareness of what the other model capabilities were. "ESCALATE to opus if you need good vision awareness".
When Haiku 5.5 came out though, for my tasks, the cost differences disappeared almost immediately. Haiku was good enough / fast enough for so many things that introducing a decider layer didn't offer huge wins. Jev's "millisecond" level performance was nice but would get dwarfed by other latencies in my chain ( web browsing ), that it didn't make a measurable difference in my tasks.
This made me think... Is anyone using small models to generate candidates and let a more expensive model pick one? I.e. genetic programming on neural networks? Perhaps a decision model could be used for the picking?
This is very cool. But also, it’s a benchmark/result for a very specific experiment/task: identity resolution
Despite the title, the article is not making claims of, or demonstrating general usage improvements or general coding agent improvements
Not hard to test. I wired a “judge” into my existing Jev setup and… it works. I was able to trim my prompt down to the basics without “engineering” and have my entire test suite pass. Best I could get before was 1 failure, with an extremely brittle prompt. Like, adding a comma where grammatically appropriate broke 10 more tests.
Agreed, though many kinds of tasks are likely to benefit from this pattern, YMMV.
This sort of stuff has me believing that decision model controlled/prioritized context is the future of context management beyond the raw AGENTS.md file.
Why give all the context to an agent, when the context can also be curated, updated and made concise automatically for each new problem.
I too feel like context management has a lot of room for improvement, but the problem right now is that dynamic context management will most likely tank your cache hit rate. We need caching that's not just based on prefix-matching for this to make sense.
My bandaid fix for this is having a large number of sessions or "agents" and being ok with them being inactive at times, but they maintain their context. This has to be in some kind of system with appropriately sized "departments" to justify that agent's existence, or to justify it splitting into two, or creating sub-agents, etc. Yes, there is a cost when you cold start an agent after a week, but for now I'd rather pull that whole 400k+ context back without layers and layers of /compact. And then hopefully you keep that agent going for long enough to benefit from any cache-hit discounts.
Not really a good analogy, but it feels like having an image you like but every time you view it, it runs it through another loop of jpg compression. Use it when it's most effective, otherwise let it preserve it's details.
And as I see it that is impossible with current autoregressive global attention models because each part depends on the one before. Maybe if we can cache the local attention in blocks and only recompute the global parts.
Completely agree.
It's honestly surprising how little work gets put into building harnesses which carefully steer the LLM through sophisticated context curation. Probably due to the belief that we shouldn't care much because AGI is just around the corner.
Related (but shameless) plug : I've actually been hacking a project over the past two weeks to address this very issue. I'd like to create an SDK which embeds the evaluation of LLMs into code seamlessly, and does "behind the curtains" context curation, to really enable "hybrid" code/LLM agentic applications. (if you're curious => https://github.com/VKCR/semiotique)
Yes × 1000... Jev is the classic type of scientific discovery, any human can see it's effectiveness without actually trying it. We were only deprived the discovery because of the incredible noises created by the AGI/ASI ferver....
most people only need if statements
wait till they found out about that nanosecond latency “conditional model”
turns out I’ve been an “RLIF” training expert my whole career
Amusingly the biggest revolution of this tech wave will probably be the automatic if writing.
I like cheap intelligence but I don't like ubiquitous identity resolution. Perhaps my values are internally incoherent.
We have enjoyed a lot of privacy because of a lack of ability to violate it.
I mean that there was always the structural incentive to completely ignore privacy. It just wasnt technically feasible.
I wouldn’t assume any structural limit to what a government, corporation, etc wants to know about you. The day reading someone’s mind is possible is the day you realize the world doesn’t recognize any sort of right to private thought.
Does the public have an incentive to read the government's mind?
Or maybe it's more about capability.
XKCD 1390: I mean, it's not like we could just demand to see the code that's governing our lives. What right do we have to poke around in Facebook's private affairs like that?
It's illustrative that there are even errors in how the humans have formulated the task and documented the errors.
They propose:
Valentino → Female Child
Yet this is obviously a boy name, else it would be Valentina. Is this model going to return confidence values? No, they want decisions. This is an inherent problem in AI with reductive logic for these tasks.
Likewise, dates. Is 17/08/22:
- 20170822 (Chinese write the year first, and in speech and chat they will say '17' not '2017'.)
- 20220817 (Everywhere But North America.)
Is 04/06/2009:
- 20090604 (Non-North American)
- 20090406 (American)
Without knowing the cultural context, the location, the document type, the document author, etc, you could be getting it wrong.
Germans write dates like 17.02.2026
It’s an interesting problem. And i have following thoughts to it?
- Are all cases even decidable?
- Shouldn’t one false positive rank higher than a hundred rank negative.
-Shouldnt’t the AI just abstain from decisions to avoid false positives.
- Is it smarter to seperate this in a lot of small sub problems and to define this as a rule set of small yes no decicions, why does this need to be solved agentically?
- You should definitely clean up deterministically first.
I feel like if you really want to solve this problem, you would approach it differently. This seems like a wrong approach and a bad article on top?
> - Are all cases even decidable?
No, and I think undecideable is quite frequent.
> -Shouldnt’t the AI just abstain from decisions to avoid false positives.
Yes, and I think regulation should be quite strong in this regard.
> - Is it smarter to seperate this in a lot of small sub problems and to define this as a rule set of small yes no decicions, why does this need to be solved agentically?
For larger contexts I worry about agentic confabulation, but humans unquestionably look at the gestalt when doing matching. I don't want idiot clankers making enforced decisions because they have misunderstood one field.
> - You should definitely clean up deterministically first.
If you can't do field validation on entry then I'm not sure deterministic rules work better, they may be worse, although easier to reason about.
>>Shouldnt’t the AI just abstain from decisions to avoid false positives.
>Yes, and I think regulation should be quite strong in this regard.
See also: The other Sean Byrne doesn't exist
https://news.ycombinator.com/item?id=49307592
As well as some of the harrowing comments in the thread.
But surely there are some deterministic reasons to clean up.
I personally wouldn’t lean towards LLMs on identity resolution. I would like give it the dataset and then ask me questions about specs? I really struggle to find the LLM part reasonable here. I mean potentially some typos, or nicknames, but those could be potentially resolved deterministically also.
I would probably have reviewer running that suggests new rules that I could implement. To me this seems more like compiler work than LLM work.
It’s a surname. (Edit you can see this in the first row they use it as an example surname)
It can also be a surname, but in the example they infer female child (don't know why they say child).
Cf Lies we believe about Names, Addresses and Dates.
That is not what they say. There are errors in the data, the data does not have the real name it has “female child” in instead.
Perhaps there’s something else in the post that you’re talking about but please do take the time to actually read what’s there, I’m only telling you what’s written in the text around the problem you think you’ve found.
>Is this model going to return confidence values? No, they want decisions.
The system doesn't return confidence values because that's not what they are designing. The models, however, do and to me it's interesting that Jev's measure of confidence is pretty well calibrated to its correctness.
>Without knowing the cultural context, the location, the document type, the document author, etc, you could be getting it wrong.
And? There's no perfect process, just ones that asymptotically approach correctness at exponentially increasing cost. They own the system and get to decide what works for them.
Exactly, there is no perfect process for such simplistic binary decision making. But we are seeing pipelines for tasks, like document processing, where it is becoming accepted that the machine is right.
Rather than binary classification we should be locating document fragments in a high dimensional feature space, and with all those fragments coming to a confidence bounded guess.
Additionally, when someone submits documents that can't be recognised as valid, the response should not be INVALID, it should be UNDECIDED. In that sense, passing to Opus is great, but passing a tiny fragment of the problem is a bad way to do it.
Excellent read. I use the same hybrid approach for a lot of security engineering work and it’s working well after initial calibrations, for example for our own harness we rebuilt auto mode exactly like this to judge tool calls and intent.
I had the same idea, I extended opencode-warden to use a local decision model to validate tool inputs and outputs. It seems to work... ok? d1 is fast enough to run locally on an old 1080 GTX. Not sure if I can recommend relying on it.
Let me make sure I'm getting this: They asked the models for answers but also for confidence score.
Jev's confidence score was more reliable. Which means that they were able to use Jev despite it having lower accuracy, because of the confidence was below a cut off, they would escalate to an expensive model (Opus)?
Whereas Luna, even though it was more accurate than Jev, its confidence was way off, so they couldn't reliably know when to fall back?
It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so.
- Mark Twain
There is an error in your interpretation of calibration graphs and the conclusion that Jev is more calibrated than Luna is likely incorrect. Or at least the graphs aren't enough to conclude either way.
I don't have the actual data but based on the graphs I estimate the expected calibration error to be 2-3% lower for Luna on both the tasks.
Its true that Luna seems to give lots of high confidence values but it's also correct more. Higher number of mistakes at higher confidence isn't itself undesirable if its pass rate is also high. It'd be good if they reported actual ECE.
>Most of Jev's mistakes are closer to 50% confidence, which is what you'd expect out of a well-calibrated model.
This statement is also incorrect
Any time I see these reports its usually practically unusable. Is this one anything different? I am curious in Jev case, that model fascinates me.
Ideally, "system one" intelligence is good both at quick reactions and at knowing when to hold off and wait for "system two." Sure enough, that works with Jev.
I don't usually get excited about the high-level, "whoooa man" marketing-speak, but the parallel is striking.
Where is the Jev that can replace 20 Opus sessions asking each other for help in a coordinated way? I want to shift towards a local model that "asks for help" in an effective way, but it still seems a bit out of reach. I mean, you could build your own 27B or 35B or A3B or bumblebee and spend a week or two carefully forming it into something useful, but it feels like that would be too specific and it would be thrown out when something like qwen4 is released. Maybe not, I don't know. Opus still exists as it is, in a "subsidized" form that's hard to step down from.
Jev is pretty cool for augmenting llms in a lot of ways. We use it in our own agent orchestrator[1] to help route ideas/prompts to running agent sessions. I find when I have a lot of parallel agents running its hard to reason about which agent I should be talking to and Jev makes that call for me.
[1] https://github.com/codealong/codealong
I wonder if something like this is the undisclosed new approach from Anthropic that presumably has massive improvements by combining multiple models. Leveraging decision models as much as possible seems logical for extra speed and token efficiency.
I think OpenAI and Anthropic have both been doing similar "routing" approaches for a while. Wasn't OpenAI one of the first to have a small model start responding to you for the first several tokens so that you got an instant response while the big model became available?
Similarly, Grok has been routing to Opus internally for a little while now I think.
> undisclosed new approach from Anthropic
Any source on that?
There’s a METR audit document which apparently describes various unreleased models. This is discussed in a YT video by Wes Roth. I have no firsthand knowledge about it.
they do have the 'advisor' model setting now which also plays well with this
The most glaring downside with practically using decision models is if you need the model to not only decide something but give a reason, for every case, you end up not saving money or time, but just adding an extra step.
Give me a decision model that can do both, more expensive than today’s decision models but still cheaper than full on frontier model, and it’d open up so many more use cases, IMO
Would it make sense to add a second decisions after the first with predefined options for why the model chose what it chose? I guess this could work for some cases
It would make more sense to have the decision model check for each possible reason whether it applies in the current case and then have the resulting consequences determined by a bog-standard if-then-else program.
Why isn't Jev on Bedrock asap... I did a demo for using it as an orchestrator but the problem is upper management won't approve it unless its on Bedrock which is a real bummer
Upper management don't like the idea they could be replaced by Jev.
Or a pigeon pecking at a checkerboard with randomly scattered bird feed.
On a serious note: all the heat and light is about AI for automating coding tasks, but a significant amount of management is just theater about deliverables. So much of that could be replaced by better product delivery pipelines that can accurately generate the "when can I expect this release?" without a layer of management that considers sitting in zoom calls all day work.
But maybe I'm just a little bit too jaded.
> Calibration matters
I agree. What I’ve encountered though is even building enough cases for a calibration data set is hard. And so we farm out to smarter expensive models like Astra to label cases and use that as a calibration data set for the now cheaper LLM judge actually running as our judge
Is this standard ? Or am I just too lazy to manually label enough data to calibrate ?
What's interesting and annoying is that the results only matter for these specific releases of the models, with the specific parameters used.
That isn't obvious, unless you've been doing ML for a while.
Yep. It's actually kind of mind blowing how well "reasoning" works at de-biasing and pushing out of global minima, in hindsight, when you see how jagged things get again without.
These are the results of a little performance tuning/permutation testing run against context format and question wording we did. Obviously there is a lot of noise here but I was actually kinda surprised that there was a clear "best way to ask the question" even for relatively low N and low complexity/ambiguity of the task.
(This is a ~30s Ai-driven gif presentation of the results, the full grid shows up about 15s in): https://imgur.com/a/wGFZvGw
These models in particular have some pretty dubiously generalized and diverse training datasets. You can see this for yourself BTW by looking at the datasets SOTA Jev-like models are being trained against and evaluated on. No knock against the teams btw, I just think it's rather demystifying and makes me quite optimistic at how much better these models are going to get over the next few years: https://labs.baseten.co/articles/mammoth-1-and-elephant-1-fr...
Im also planning to combine them.
I have haiku hooked up for guardrails in my llm proxy. With the prompts routed to opus or sonnet after they pass the check.
I'll replace haiku with Jev and my costs should definitely drop.
The bar charts in this post really warped my head. If you’re interested in demonstrating confidence calibration you shouldn’t visualize the number of mistakes in a bar chart. Use a classifier calibration curve or something, but don’t just show me the counts in 10 percentage point buckets.
As an aside I'm amazed at just how fast the word Jev became generic. Possibly the fastest ever in corporate history?
Liked the read. I wish there were some code examples for how jev/Luna were actually done as I try wrapping my head around their new paradigm.
This is the Model Router architecture, from another angle. The problem is always, how does a dumb model know it needs to escalate to a smart model?
Some problems are obviously hard known-unknowns (eg “solve Navier Stokes”). The problem is unknown-unknowns or knowledge gaps, where the model doesn’t know they need help.
It's not a problem you can get right, because you can't know if you need to escalate until after you escalate (and what you escalated to disagrees with what your first choice would have been). But that's the point: how cautious you make the model is the knob that lets your trade cost for correctness.
I don't believe the answer is ever "how certain is the model?" It's "how much is at stake?" Or "how hard of a problem does this appear to be?" It's not about teaching the model to have uncertainty, but rather to be avoidant of things that are sufficiently important to get right.
Yeah, it’s a precision/recall tradeoff. And as such you calibrate by the cost of each FP vs FN.
You will depend on a super intelligent AI whose purpose is to take money from you
> We tested whether fast, cheap decision models like Jev and GPT-6 Luna
GPT-6 Luna is a small LLM, not a decision model :P
No but it's the only OpenAI model with a decisions API https://developers.openai.com/api/docs/guides/decisions
Small LLM + no reasoning + fixed JSON-only output = decision model
There's actually a decision model version of GPT-6 Luna... You are right to be confused though. OpenAI and their never ending naming games. https://openrouter.ai/openai/gpt-6-luna-decisions
We've been doing this for a while. Fine tuning the decision model helps a lot too. The problem is you still need a pretty low confidence threshold (e.g 10%), but often times that reduces the work by 70% with 0% quality loss (in our benchmarks)
I got very good results from JevK5: https://github.com/allebee/jevk5
It's what I use locally. For my workload, it was right up there with Jev.
4x lower is so mathematically awkward.
That reads fine to me as a native-ish English speaker. What would read less awkward for you?
Like a comment on another thread, 0.25x would be the much better way to say this. It's 25% of the original value. It's also consistent: If 3x means 300% the original value, then 0.25x is 25% of the original value.
If you say 4x lower, that changes the semantics for lower than x vs higher than x, which is absurd.
> If you say 4x lower, that changes the semantics for lower than x vs higher than x, which is absurd.
where does it say in the rulebook that human languages have to have perfectly symmetric pairs of words?
Whatever happens, surely we can unite against the Crazed Barbarians who use percentages. :)
So like, -3x?
More like 0.25x.
Thanks for making targeted advertising and surveillance state stronger. We love you for that.
Hope my sarcasm is making it across.
I originally read this as “a Jew can ask opus for help gets accuracy at 4x lower cost” and was surprised
If you are looking at huge categorizing and parsing work, using a local decision model and passing it to Haiku or Luna can really lower your token cost.
The application creeps me out
One of the most immediate and practical uses for AI for most (all?) businesses is CRM enrichment; this is a great article, thanks for sharing.
Been doing this on Discourse for Spam and locale detection, mostly doing d3-turbo with Qwen 3.8 Flash Next Help.
their cost numbers give you the hard set escalation rate even though they don't list it, (2.94 - 0.06) / 5.39 is ~53% going to Opus vs ~22% on the easy set, same thresholds
What about Sonnet? Haiku? Does it need Opus specifically?
Seems to me that the important factor is that Jev has well-calibrated confidence for this task. Which secondary expensive model it defers to is probably less critical in this technique as long as that model has higher performance than Jev.
Yeah exactly - very interesting that Luna is calibrated so poorly despite working better on easier problems.
i'm so numb to these headlines nowadays... i guess i've been disappointed by how effective these approaches end up being for actual work but i feel like it's easy to produce these headlines with "proof" that doesnt prove anything besides the sandboxed scenario it was run in
Isn’t that by nature a deterministic problem?
Seeing the title made me think: Jev used to be quite popular. Is anyone else still using it heavily these days?
Jev is interesting because it adds "muscle memory" to LLMs. I think we're approaching some fusion of systems that approaches the human brain here.
I use swival.dev where a decision model (Cloudflare's Clef running locally is free and works great) is used for automatic model selection, but also to complete a task, just like the LLM. It gives confidence scores on the reliability of the LLM output, is used instead of the LLM, even mid-task when appropriate, etc.
And yes, that saves money since less tokens are sent to a remote LLM.
Memdoor author here which is a coding agent that run Jev inside. I believe this is becoming mainstream in any coding agent for routing calls and reduce input sizes. First candidates are edit tools for reading file or jgrep for greping large files folder etc.. The Token saving is mostly on the read side.
If we compare coding agents with Jev on vs off we get those numbers (metered with Memdoor): - read-heavy tasks (coding questions , logs tail etc..): -63% to -65% input tokens, 40-45% fewer model calls, on GLM 5.3 half the wall time. Reproduced a month apart on two models (DeepSeek V4.1 Flash, GLM 5.3). - a file edit: -23%
Two things match the post. Calibration is what makes it safe: I apply a decision at confidence >= 0.75 and fail below this threshold or on timeout or on an unknown tool family. The dollar saving scales with the price of the model being saved: on a $0.04/M model Jev costs what it saves so on a $0.38/M model it's negligible but on Claude model for example it can be outstanding.
Where it did not help: giving Jev more context about the repo. I found same thing as the post.
If you are interested i have put figures here : https://github.com/guregodevo/memdoor/blob/main/docs/feature...
My guy you sound like a bot, stop it.
I wish my bots talk like this. BTW if you have anything relevant to say about Jev i'm listening