This benchmark is too simplified to be meaningful. The choice of a framework is rarely based on speed alone, it factors in other elements like the user's domain knowledge, long term maintainability and the framework's track record itself w.r.t security and development of features.
In this regard alone, it will make a lot of sense to spend extra $2 and 2 mins to trade off for a long-term maintainable codebase and code you can reason about than just something made quickly just to get it deployed fast.
In my personal use case, that would be Svelte 5 for me and I'm sure it will be something else for everyone.
I have most of my apps in it without much trouble.
One of my recent ones had a very complicated form and state management.
AI really struggled with HTMX in that use case.
Like the form might work but the page would take 30s to load. Or some state would randomly drop or a toggle wouldn't work after changing a dependency.
If I rewrote that in react it would have one shot it without a doubt. Some things are just better models for different problems.
For the record I eventually got enough tests in place to keep the agent honest and the HTMX version is now a lot faster anything I could have got out of react but yeah, it was an adventure.
I judge languages by how they look. JavaScript is a wall of const/async/await these days, which is ugly. Functionality aside (the APIs have gotten pretty decent), the language has gotten uglier. It used to be nice and nuanced even with boilerplate. Closures and prototype inheritance is neat. Walls of const are not. Does every enclosed function need a million consts for every expression?
Also, it’s a language where everything happens somewhere else. Not here and now, somewhere else on some other event. That sucks for the small stuff.
The point of using a framework is to reduce mental load at expense of customizability. The moment you need to change something in the framework - this mental model falls apart.
Obvious solutions are to vendor your dependencies (compile the framework from owned source) or to drop to the lower level of abstraction.
LLMs should have no problem using browser/node APIs directly since they're very well used, well documented, and not changing as much as higher level software does.
> My goal here is not to write the end-all-be-all benchmark for “I’m a vibe-coder, which framework should I choose?” The point of this post is to demonstrate that it’s possible in the era of AI coding to still make decisions based on evidence rather than vibes.
But it’s obvious that the article will be understood as a useful benchmark, so author probably should have addressed that point earlier and more thoroughly.
Taking the authors pov for a moment, they just showed you how to put all your criticisms to work in making a newer better benchmark :)
It scores well here because Astro is for “content-driven websites” and that’s what the benchmark app was. I’d be curious how different the results would be if the app was highly interactive, with nontrivial realtime/async stuff going on etc.
Can confirm Astro is very very pleasant to use, and the escape hatches for when you do need something a bit more heavy hitting feel very well designed.
Solid work, but it would be really cool if we can also test Pokemon motions and how the game responds in real-time. Is an agent able to think about reactions, directional movements and attack patterns of each Pokemon is where the real intelligence test lies.
Traditional languages may have their limitations, but [C++, Rust, JavaScript, TypeScript etc] can still do pretty well.
my experience is that one-shotting a simple front end works great. Then you want to add features and move things around and each turn starts getting slower and the single page with JS and CSS and HTML starts to turn into a 3000 line monster and your context window starts to creak under the strain of even simple changes, because the LLM has to keep more and more of the application in its head just to make relatively small changes without breaking something else. Either you should have started with some kind of framework with hygienic rules on how to separate concerns or at minimum a good set of coding standards that force the LLM to start building clean components from scratch. The benefit isn't necessarily cleaner code for its own sake, it's that the LLM can work on one part of the application without having to understand the whole thing every time. That being said, once you have a 3000 line prototype it's not that complex to simply ask the LLM to refactor it into a set of orthogonal components. As long as it was building unit tests using playwright or something this can turn your slop into framework code cleanly. The interesting question is whether you saved anything by putting off that structure in the first place.
I run into this problem turning claude artifacts into things I might want to maintain longer term.
And a little research turns out that anthropic itself makes available a *react* based artifact builder so you don't end up with SLOP from the get go. You could probably modify the template to work with your framework of choice.
The vanilla advantage is also there in a pure client-side pwa. Arguably it’s even bigger, because the bundle size ends up way smaller and the agent needs to hold less in context.
Plus all the other vanilla advantages: fewer dependencies, lower maintenance, less retraining, etc…
I'm reading that agents can now mash up video games that were written in assembly. At that point I have to wonder if the abstractions are adding anything.
Of course, there is the question of "the human can still understand what is going on", but it's not clear the new frameworks are optimizing for that...
I mean maybe I'm too grumpy, but never in my life did I think I'd have to fight against a strong contingent of people who really don't see the value of abstraction. Taking crazy pills must feel like this.
They are trapped by ego-driven motivated reasoning in epistemic closure. Painful consequences are perceived as evidence that their decisions don't cause pain and that the pain must stop if they do more of what is causing it.
When I was running a small dev agency with a sprinkling of web projects, Angular was useful because it every project looked more or less the same. The rigidity (and verbosity) it imposed was an advantage in this setting, as it helped with the cognitive load when switching between projects. Apart from that, I wasn’t a fan either.
I mean it just seems like even in the agentic era, you should know when to use the right tool for the right job. The site shown seems perfect for htmx style websites, so idk why you would use an SPA. Still very cool experiment, appreciate the data and human writing!
Just use plain HTML for God's sake. HN demonstrates very well that you can do plenty without some Javascript monstrosity. At the very least, make your your websites degrades gracefully when JS is turned off.
"do plenty" or just use React for everything and have a single thing where you don't need to worry about the limitations or having to remake the same stuff the framework provides?
Unfortunately the author is not only correct about front end development vibes, but due to this state of affairs, I ceased to recognize browser based UIs as being software development at all. Too much browser tech is broken to call it software. As for agents, good for them!
This benchmark is too simplified to be meaningful. The choice of a framework is rarely based on speed alone, it factors in other elements like the user's domain knowledge, long term maintainability and the framework's track record itself w.r.t security and development of features.
In this regard alone, it will make a lot of sense to spend extra $2 and 2 mins to trade off for a long-term maintainable codebase and code you can reason about than just something made quickly just to get it deployed fast.
In my personal use case, that would be Svelte 5 for me and I'm sure it will be something else for everyone.
HTMX is interesting.
I have most of my apps in it without much trouble.
One of my recent ones had a very complicated form and state management.
AI really struggled with HTMX in that use case.
Like the form might work but the page would take 30s to load. Or some state would randomly drop or a toggle wouldn't work after changing a dependency.
If I rewrote that in react it would have one shot it without a doubt. Some things are just better models for different problems.
For the record I eventually got enough tests in place to keep the agent honest and the HTMX version is now a lot faster anything I could have got out of react but yeah, it was an adventure.
I judge languages by how they look. JavaScript is a wall of const/async/await these days, which is ugly. Functionality aside (the APIs have gotten pretty decent), the language has gotten uglier. It used to be nice and nuanced even with boilerplate. Closures and prototype inheritance is neat. Walls of const are not. Does every enclosed function need a million consts for every expression?
Also, it’s a language where everything happens somewhere else. Not here and now, somewhere else on some other event. That sucks for the small stuff.
The point of using a framework is to reduce mental load at expense of customizability. The moment you need to change something in the framework - this mental model falls apart.
Obvious solutions are to vendor your dependencies (compile the framework from owned source) or to drop to the lower level of abstraction.
LLMs should have no problem using browser/node APIs directly since they're very well used, well documented, and not changing as much as higher level software does.
This is buried at the end.
> My goal here is not to write the end-all-be-all benchmark for “I’m a vibe-coder, which framework should I choose?” The point of this post is to demonstrate that it’s possible in the era of AI coding to still make decisions based on evidence rather than vibes.
But it’s obvious that the article will be understood as a useful benchmark, so author probably should have addressed that point earlier and more thoroughly.
Taking the authors pov for a moment, they just showed you how to put all your criticisms to work in making a newer better benchmark :)
Astro comes out on top with the best time, #2 lowest spend and zero client side JS. I know what I'm using for my next project.
> I know what I'm using for my next project.
Django?
It scores well here because Astro is for “content-driven websites” and that’s what the benchmark app was. I’d be curious how different the results would be if the app was highly interactive, with nontrivial realtime/async stuff going on etc.
Can confirm Astro is very very pleasant to use, and the escape hatches for when you do need something a bit more heavy hitting feel very well designed.
next project? pun of the year?
Solid work, but it would be really cool if we can also test Pokemon motions and how the game responds in real-time. Is an agent able to think about reactions, directional movements and attack patterns of each Pokemon is where the real intelligence test lies.
Traditional languages may have their limitations, but [C++, Rust, JavaScript, TypeScript etc] can still do pretty well.
> in the agentic era, when developers are no longer typing code by hand
I hope I'll be the last person on Earth writing code by hand.
Plenty of people will be coding by hand for years to come. They just wont be employed for doing so.
Neocities ride hard on the old-internet hand-crafted HTML aesthetic. Maybe we could start a similar club?
my experience is that one-shotting a simple front end works great. Then you want to add features and move things around and each turn starts getting slower and the single page with JS and CSS and HTML starts to turn into a 3000 line monster and your context window starts to creak under the strain of even simple changes, because the LLM has to keep more and more of the application in its head just to make relatively small changes without breaking something else. Either you should have started with some kind of framework with hygienic rules on how to separate concerns or at minimum a good set of coding standards that force the LLM to start building clean components from scratch. The benefit isn't necessarily cleaner code for its own sake, it's that the LLM can work on one part of the application without having to understand the whole thing every time. That being said, once you have a 3000 line prototype it's not that complex to simply ask the LLM to refactor it into a set of orthogonal components. As long as it was building unit tests using playwright or something this can turn your slop into framework code cleanly. The interesting question is whether you saved anything by putting off that structure in the first place.
I run into this problem turning claude artifacts into things I might want to maintain longer term.
And a little research turns out that anthropic itself makes available a *react* based artifact builder so you don't end up with SLOP from the get go. You could probably modify the template to work with your framework of choice.
https://github.com/anthropics/skills/blob/main/skills/web-ar...
Java script frameworks are just extra boiler plate that isn't needed in the age of ai-coding.
Server side page generation with a sprinkle of js will give you the same results with GBS less of packages and comple time
The vanilla advantage is also there in a pure client-side pwa. Arguably it’s even bigger, because the bundle size ends up way smaller and the agent needs to hold less in context.
Plus all the other vanilla advantages: fewer dependencies, lower maintenance, less retraining, etc…
I'm reading that agents can now mash up video games that were written in assembly. At that point I have to wonder if the abstractions are adding anything.
Of course, there is the question of "the human can still understand what is going on", but it's not clear the new frameworks are optimizing for that...
[flagged]
LOL
I mean maybe I'm too grumpy, but never in my life did I think I'd have to fight against a strong contingent of people who really don't see the value of abstraction. Taking crazy pills must feel like this.
Your mistake is trying to fight people. You don't have to do that. You can just let them suffer from the consequences of their own decisions.
They are trapped by ego-driven motivated reasoning in epistemic closure. Painful consequences are perceived as evidence that their decisions don't cause pain and that the pain must stop if they do more of what is causing it.
Interesting, I’m just sad svelte didnt behave better because of the LLM reading migration guide on every answer
Interesting article. Kind of seems like you'd want a more complex task, there's not much differentiation here.
Angular sucks, I always knew it, and now we have some data that it’s broken even for agents. I’m not surprised
When I was running a small dev agency with a sprinkling of web projects, Angular was useful because it every project looked more or less the same. The rigidity (and verbosity) it imposed was an advantage in this setting, as it helped with the cognitive load when switching between projects. Apart from that, I wasn’t a fan either.
> The Pokémon app is small and doesn’t represent a large, complex real-world application.
If you aim to make small apps that don’t represent complex real-world applications you should not use Angular.
But blanket statements like yours are what I hate the most in software industry.
I suspect that results would be different if the app was a bunch of boring forms with complex validation rules, which is often encountered in jobslop.
Not necessarily that Angular would win that, but I imagine that it would at least significantly shuffle the order.
Nice to see Nuxt holding its own. Astro or Nuxt are my gotos for client work and personal projects.
I mean it just seems like even in the agentic era, you should know when to use the right tool for the right job. The site shown seems perfect for htmx style websites, so idk why you would use an SPA. Still very cool experiment, appreciate the data and human writing!
Glad to see Astro doing well here. It's by far my favorite way to build sites of all kinds, both content rich MPAs, and react heavy web apps.
It is fairly obvious that it is all going to be some kind of React/JSX with tailwind.
It's definitely all tail wind
where is jquery? :(
On 64% of websites.
Shhh we don’t talk about it
htmx for the win, as always
Just use plain HTML for God's sake. HN demonstrates very well that you can do plenty without some Javascript monstrosity. At the very least, make your your websites degrades gracefully when JS is turned off.
HN uses JavaScript though, not just plain HTML.
HN is perfectly usable without JavaScript enabled.
The only feature that appears to require JavaScript is search.
And it certainly doesn't do "plenty."
"do plenty" or just use React for everything and have a single thing where you don't need to worry about the limitations or having to remake the same stuff the framework provides?
and what % of users have JS turned off? 0.00001%?
Unfortunately the author is not only correct about front end development vibes, but due to this state of affairs, I ceased to recognize browser based UIs as being software development at all. Too much browser tech is broken to call it software. As for agents, good for them!
[flagged]
[flagged]