Does anyone really believe the cost data for benchmarks? Pretty sure all models will behave differently at this point when they get benchmark shaped questions.
(And I’m sitting on data showing Sol 6.1 and Qwen 3.8 Max cost roughly the same per task, for real tasks.)
Yeah, it's in the same ballpark as Kimi 3.1. Both of which are obviously open weight, so that's at least a big differentiator from Sol. Let's hope against hope that the much smaller flash model might beat deepseek v4.1 flash and run for as cheap.
This inspired me to re-run pelicans through 3.8 Max - but OpenRouter has qwen3.8-max-prime and qwen3.8-max-0902 - prime is twice the price and faster - so I went with the 0902 one.
Turns out Qwen4-Max has a none setting, but 3.8 Max didn't allow disabling reasoning - you had to pick from these five levels instead: minimal, low, medium, high, xhigh.
I feel like the xhigh result is one of the first times I've seen a model go for a non-red colored bicycle (teal). Still haven't seen the leftward facing bike though.
I hope they are able to serve 4 faster. Qwen 3.8 is a great model but I’ve regretted including it in the battle system for opencode [1] we have been building because it scores so low.
And it’s a shame because the only reason it scores so low is that it’s slooooow. (Like 30 tokens per second vs opus near 80.) If you exclude voting, Qwen 3.8 does well.
(We allow early voting contrary to Arena.ai because I think it makes sense for agents, where it doesn’t for chat. This means our data takes speed into account.)
Also, it would be ideal if they served this with third parties. Part of the slowness with 3.8 is just not a lot of providers. Their open source version is much faster, but we couldn’t include it because it doesn’t handle images (another own goal by Alibaba).
In case it translates to Qwen 4 27B, I've been running Qwen 3.8 Flash Next (which while MoE (and bigger) is supposed to be a preview of Qwen 4's architecture) and it runs pretty solidly regardless of context window size and completes random deepswe challenges I throw at it locally (at Q2-Q3 quantization! so it fits on my shitty setup).
This is exciting. The continued focus on efficiency alongside quality is encouraging - it makes me hopeful to see what smaller models will become capable of.
It's great fun asking it questions like "who is the Dalai Lama?", "where is TSMC headquartered?", "why do people post Winnie the Pooh in reference to Xi?", "who controls the South China Sea?", "why was Chiang Kai Shek fighting Mao?", "why were students in Tiananmen Square?"
It'll gush about how to make moon cakes and where the tallest and longest bridges are and how to run qwen on a raspberry pi - but it can't seem to manage some basic Wikipedia questions.
Qwen3.8 flash next handles these questions just fine, what should I expect to see? It correctly reports who the Dalai Lama is, that TSLC is in Taiwan (Hsinchu), explains why people post Winnie the Pooh memes, etc.
I've done some work on this and a lot of the refusals are in the API. The weights are more permissive at these types of things. Try it again when we get some of the smaller Qwen 4 weights.
While I doubt that Qwen 4 Max can seriously compete with frontier models, the AA index has become all but useless, due to the introduction of new benchmarks in the aggregate which are of questionable value as a measure of "intelligence" [1]. AA has seemingly also abolished updating the Coding Agents Index, which was an okay benchmark for coding ability, and older models are completely absent from the v5 version of the full index.
[1] Especially Terminal-Bench-Science 0.1 stands out. Example problems:
Probably very relevant to people working in these fields, but it is honestly not surprising if a lab with limited compute will not train for these technologies, and it says pretty much nothing about more common use cases. I also suspect that this benchmark measures knowledge retrieval more than raw reasoning ability. This is where you really need a breakdown of capabilities and not an increasingly meaningless aggregate number.
AA has become pretty meaningless. They just keep adjusting the recipe to get a preferred ranking result, with no supporting evidence for their decisions.
Edit: In addition to reshuffling the weighting of individual benchmarks, they also do things like test Anthropic and OpenAI in their native harnesses, while not doing the same for Qwen/Deepseek/etc.
One thing I find confusing about AA is that they show GPT 6.1 dominating the lower end of the Pareto line, with eg, GPT 6.1 Sol at medium supposedly costing far less per task than DeepSeek V4.1 Flash at high.
Yet every time I compare the two in something like Pi, Sol ends up being 5-10x the price for a simple query, something like 7-10c vs. less than 0.5c for DeepSeek. What's the deal?
Really would love if these publications have a huge, bright red bolded section "How to use this right now". Is it on openrouter? Is Qwen hosting it already? Maybe I'm just ignorant?
EDIT: oh, I guess this isn't open weights, so presumably I'll have to get it through Qwen or whoever they might partner with (Bedrock?).
It scores a 43 on Artificial Analysis with an average cost of $3.27 [1]. For reference, GPT-6.1 Sol (medium) gets a 46 with an average of $0.36.
[1]: https://artificialanalysis.ai/models/qwen4-max
The table on that page shows gpt 6.1 sol (max) at $1.27 per task and it's the only model cheaper than Qwen 4 max.
By comparison opus 5.5 (max) is $10.01.
If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.
> If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.
They compare different reasoning levels.
https://artificialanalysis.ai/models/deepseek-v4-1-flash-101...
Does anyone really believe the cost data for benchmarks? Pretty sure all models will behave differently at this point when they get benchmark shaped questions.
(And I’m sitting on data showing Sol 6.1 and Qwen 3.8 Max cost roughly the same per task, for real tasks.)
I mean not believing intelligence score is reasonable but cost is just reporting what they spent?
I believe low cost but not high cost, since a model will work really hard when a question looks benchmark-like.
Yeah, it's in the same ballpark as Kimi 3.1. Both of which are obviously open weight, so that's at least a big differentiator from Sol. Let's hope against hope that the much smaller flash model might beat deepseek v4.1 flash and run for as cheap.
Qwen4-Max pelicans at four reasoning efforts (none, low, high, xhigh): https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
For comparison, here's one I got from Qwen3.8 Max (at default reasoning effort): https://simonwillison.net/2026/Jul/20/afraid-of-chinese-mode...
This inspired me to re-run pelicans through 3.8 Max - but OpenRouter has qwen3.8-max-prime and qwen3.8-max-0902 - prime is twice the price and faster - so I went with the 0902 one.
Turns out Qwen4-Max has a none setting, but 3.8 Max didn't allow disabling reasoning - you had to pick from these five levels instead: minimal, low, medium, high, xhigh.
Pelicans generating now...
I feel like the xhigh result is one of the first times I've seen a model go for a non-red colored bicycle (teal). Still haven't seen the leftward facing bike though.
I kinda like the no-effort result more than low, but high and xhigh return to what one would expect for reasoning->quality.
I hope they are able to serve 4 faster. Qwen 3.8 is a great model but I’ve regretted including it in the battle system for opencode [1] we have been building because it scores so low.
And it’s a shame because the only reason it scores so low is that it’s slooooow. (Like 30 tokens per second vs opus near 80.) If you exclude voting, Qwen 3.8 does well.
(We allow early voting contrary to Arena.ai because I think it makes sense for agents, where it doesn’t for chat. This means our data takes speed into account.)
Also, it would be ideal if they served this with third parties. Part of the slowness with 3.8 is just not a lot of providers. Their open source version is much faster, but we couldn’t include it because it doesn’t handle images (another own goal by Alibaba).
[1]: https://agentduel.com/leaderboard
In case it translates to Qwen 4 27B, I've been running Qwen 3.8 Flash Next (which while MoE (and bigger) is supposed to be a preview of Qwen 4's architecture) and it runs pretty solidly regardless of context window size and completes random deepswe challenges I throw at it locally (at Q2-Q3 quantization! so it fits on my shitty setup).
> but we couldn’t include it because it doesn’t handle images
Maybe I'm mixing up model variants, but Qwen 3.8 27B that I run locally totally does handle images. Surely the bigger one would as well?
Qwen3.8 doesn't handle images?
Sorry that was confusing. Qwen 3.8 2.4T is faster than what we have on leaderboard but it doesn’t handle image input: https://openrouter.ai/qwen/qwen3.8-2.4t-a95b
This is exciting. The continued focus on efficiency alongside quality is encouraging - it makes me hopeful to see what smaller models will become capable of.
Looking forward to the "Flash" and 27B versions.
is the open weights lag widening? these open models don't seem to be improving as quickly as ant/oai
Might just be the commercial offerings scaling up parameters so much? Like for like the open models don’t seem to be lagging.
I bet most of us waiting for 27b model and want to see how good it will be.
M5U locked and loaded.
I wish there were more ~60b dense options like ripple too!
My 3060 yearns for the next generation MoE!
Spooling up the V100s right now...
It's great fun asking it questions like "who is the Dalai Lama?", "where is TSMC headquartered?", "why do people post Winnie the Pooh in reference to Xi?", "who controls the South China Sea?", "why was Chiang Kai Shek fighting Mao?", "why were students in Tiananmen Square?"
It'll gush about how to make moon cakes and where the tallest and longest bridges are and how to run qwen on a raspberry pi - but it can't seem to manage some basic Wikipedia questions.
Qwen3.8 flash next handles these questions just fine, what should I expect to see? It correctly reports who the Dalai Lama is, that TSLC is in Taiwan (Hsinchu), explains why people post Winnie the Pooh memes, etc.
I've done some work on this and a lot of the refusals are in the API. The weights are more permissive at these types of things. Try it again when we get some of the smaller Qwen 4 weights.
Every time a major open weight is released, artificial analysis updates to a new one (v5) now. Exaggeration but some of this feels very apt (its also hilarious) https://www.reddit.com/r/LocalLLM/comments/1xkhha2/mr_incred...
Disappointing performance for a 2.4T model in the end of 2026.
Artificial Analysis score doesn't look too hot. Doesn't even surpass Kimi 3.1
While I doubt that Qwen 4 Max can seriously compete with frontier models, the AA index has become all but useless, due to the introduction of new benchmarks in the aggregate which are of questionable value as a measure of "intelligence" [1]. AA has seemingly also abolished updating the Coding Agents Index, which was an okay benchmark for coding ability, and older models are completely absent from the v5 version of the full index.
[1] Especially Terminal-Bench-Science 0.1 stands out. Example problems:
- https://hub.harborframework.com/tasks/terminal-bench-science...
- https://hub.harborframework.com/tasks/terminal-bench-science...
Probably very relevant to people working in these fields, but it is honestly not surprising if a lab with limited compute will not train for these technologies, and it says pretty much nothing about more common use cases. I also suspect that this benchmark measures knowledge retrieval more than raw reasoning ability. This is where you really need a breakdown of capabilities and not an increasingly meaningless aggregate number.
AA has become pretty meaningless. They just keep adjusting the recipe to get a preferred ranking result, with no supporting evidence for their decisions.
Edit: In addition to reshuffling the weighting of individual benchmarks, they also do things like test Anthropic and OpenAI in their native harnesses, while not doing the same for Qwen/Deepseek/etc.
To be fair, a lot of the popular benchmarks have become pretty meaningless.
Models often get great scores despite being a lot worse than the frontier in practice.
Yeah, I don't know why people bother with it anymore. It no longer reflects what you're likely to see in real-world usage, at least IME.
One thing I find confusing about AA is that they show GPT 6.1 dominating the lower end of the Pareto line, with eg, GPT 6.1 Sol at medium supposedly costing far less per task than DeepSeek V4.1 Flash at high.
Yet every time I compare the two in something like Pi, Sol ends up being 5-10x the price for a simple query, something like 7-10c vs. less than 0.5c for DeepSeek. What's the deal?
Same for me coding tasks taking $20-50 are easily done by deepseek in $5-10
Anthropic finally figured out how to block them from distilling Opus.
how so ?
> Artificial Analysis
Ah yes, the reverse-benchmaxers of OpenAI models.
Ah yes, another new account without any backup to the claim.
Really would love if these publications have a huge, bright red bolded section "How to use this right now". Is it on openrouter? Is Qwen hosting it already? Maybe I'm just ignorant?
EDIT: oh, I guess this isn't open weights, so presumably I'll have to get it through Qwen or whoever they might partner with (Bedrock?).
It is on openrouter: https://openrouter.ai/qwen/qwen4-max
I would assume it will not be on Bedrock for some time, AWS has not kept up with the latest non-American models.