I’ve been thinking about how we practically implement this at an organizational layer for large codebases. There’s clearly alpha to be had in optimizing AGENTS.md / skills / tools / … to improve performance, quality, and cost efficiency of an agent. The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever.
The first step I see towards this is building a generic, reliable, and accurate *fitness function* for codebases - turning PRs into gradeable tasks that an agent can solve, and improve on.
I’m pretty curious how others are optimizing the coding agent harness now, as this has been a huge pain point for my company (we’re pretty much relying solely on vibes).
I literally hold a "retro" with my agent after a session.
> Let's conclude with a retro. Did you run into any issues during this session that you think could be improved? Any failed tool calls, confusing docs/prompts, or tricky wording that took you effort to figure out, etc? Any final thoughts that you want to raise? Anything minor you didn't mention? Help make this codebase easier for the next agent to work in.
It has ideas for coding changes, spots unmentioned small bugs, suggests invariants (I have a high-level list of invariants to keep), principles to adopt, lint rules, tooling tweaks (I've built SO MUCH agent-tooling; so many useful bash scripts), skill-file updates, follow up work, all kinds of stuff.
Ive been building my own ai voice agent harness from scratch for close to a year now and following good software architecture practices is a good start. So those rules have to be coded in agents.md somewhere also really helps to have a "vision" section or "spirit of the project" section that describes what the end goal vaguely looks like and things I care about in achieving for the project. This prevents agent from being brittle and "single minded" about its work. But yeah vibes most of the time is how I've also been doing it, but I did find one very important thing that has really sped up my work. So I figure I'd share it here. And that advise is to ignore front end design at all costs until the very end and you are ready to launch. UI related woes kill any type of inertia and are responsible for most of the issues. So I decided abandon all UI stuff until the very end and just focus on iterative refinement, cutting, and other back-end related work and its been great ever since. Ideas can be had, tested, validated/invalidates and you ' get stuck on the optimization thought loop. Recently I have started to consider how i can fully automate the development process as the capabilities are there but designing the instructions on how to do this well and how to handle niche cases without getting me involved takes careful planning in writing out the guide so that will be interesting to see once i get there.
It only tries to solve the „quality“ aspect of that equation tho, at least for the code output.
But I think for actual evals, some heuristics about a coding agent session are needed. But since the input tasks are always different, it’s hard to make any KPI actually comparable.
Did the agent needing 3 rounds of feedback on a big task perform worse than the one that needed none, but only worked on a small change?
Just working on my own stuff so YMMV, but from a cost reduction standpoint, the two most outsized ROI wins came from making sure the agent runs all terminal commands in quiet and using a codebase index mcp to speed up understanding and reduce full file reads [0]. I've also found the latter also dramatically improved the quality of output.
However what I wonder how we make these decisions on what to remove. Certainly some subset of the skills / AGENTS.md / prompt / tools do actually help agent performance, the problem is in identifying those, throwing away the rest, and validating that this didn't actually hurt the agent performance
Yes, my question is directed around how you are determining "task result." Is this measuring whether the code works, or whether it is maintainable?
As with human-powered coding, we read code far more frequently than we write it. It's worth spending a little extra (time|tokens) during authorship to make future maintenance feasible.
My CLAUDE.md, memories, and skills are all about either (a) adherence to project standards and guidelines, (b) product decisions which impact future code, and (c) instructions on how to prototype and work in my environment.
Removing these instructions would mean more turns with the AI to get the desired result.
in this case there was a hidden grader that checked if the implementation was correct (because that was the easiest thing to check), all 3 agents cleared this hurdle in all 9 runs
I agree, next it makes sense to try more open ended tasks + have humans (and/or multiple models) grade the runs and their results
You've just described what the AI industry refers to as "evals", a collection of which forms a particular benchmark. I definitely encourage you to define your own evals, because the public benchmarks are often either saturated (largely solved and only going to see small improvements going forward) or seemingly not predictive of real world performance. I could also go further and speculate that they are in the training data, although credible benchmarks avoid this, I'm just not sure how successful they are.
Thanks for the comment - I’ve actually been working on something to generate evals from private repos, as I 100% agree that public benchmarks are either contaminated (as OpenAI found with SWE-bench Verified), not high quality (as OpenAI found with SWE-bench Pro), or in the case of “good” benchmarks (Cognition’s Frontier Code comes to mind), non-representative of a specific repo’s work.
Sharing some challenges / learnings from the journey so far:
* Tests alone are non-representative: an agent can pass tests but write the code in a way that’s subjectively worse than another test-passing result
* However, tests are still important and the best form of
deterministic evaluation we have
* Creating executable environments to run the tests is challenging, especially when considering that this should be doable for arbitrary repos
* Selecting tasks that discriminate is an art - they need to be challenging, but not too challenging, represent the variety of work that’s done in the repo, and contain tests that failed prior to the change
* LLMs / agents are very powerful at judging code - but doing so in a way that is calibrated, consistent, and representative of codebase standards requires careful rubric creation and grading
* I’ve shifted towards using agents (from “static” LLM calls) to generate the rubrics, and to later grade the rubrics. This is powerful but even more non-deterministic
> The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever.
I've built and my team uses plannotator [1] for specs and code review. It stores all human annotated feedback, which we compile from time-to-time and iterate on coding standard skills for automated review (that seems to get better with the more feedback we provide).
You could write a script that does the same for your PR review comments, which it sounds like you maybe do with a bit more structure.
IMHO training weights has peaked and now it is time for a training paradigm for prompts and code. We don't have the gradient descent here - but I think it can be more sample efficient because causal theories can be better than just correlations.
I am working on a unified theory in https://zby.github.io/commonplace/ - it is all agent edited so it might be hard to read, but hopefully we are catching most logical errors. Some day the llm prose will improve.
Is there any reason to think that training weights has peaked rather than is accelerating? It feels like now they are increasingly able to pick some low hanging fruit by using the models in order to improve themselves and test optimizations.
I think it is now or close to now - and not because there is no more to be gained - but rather because it is now clear that you can gain more with the right agent environment.
But benchmark saturation is also something to account for.
We've using auto-research for harnesses and it's surprisingly powerful. It's amazing how many problems are easy to spot and fix from traces.
Few key things were required to get it working well: 1) let it read a heap of prod traces to spot real issues. 2) let it write it's own tools (example "loading context" goes from 20k tokens across 15 tool calls to 800 tokens and 1 call to session_context tool). 3) you need evals and val/test splits, it will reward hack. 4) you need proper tooling (synthetic users, synthetic tools) for it to be able to crunch for 12 hours and produce something. 5) you need the optimization target to be a reasonable size: not your 1M line codebase, but a lighter agent harness (pi harness, skill only, Kiln harness).
If we don't create the Torment Nexus first, somebody else who is much less responsible is gonna create the Torment Nexus before us. It's outright irresponsible to not take the lead, we might have to even give up on all safety concerns to make sure we make it to Torment Nexus IPO first.
If we don't do our best to awaken Dead Cthulhu, when He invariably awakens He will know that we didn't do our best and we'll be eaten last by the Great Old Ones upon Their return!
Since we all know that at this point, someone is definitely going to build the torment nexus, we all may as well get involved and help, so that at least we can exert some limited degree of control over the eventual character of the torment nexus. At least that way maybe we can make it a little less terrible. At any rate, it can’t make it worse than it would have been if we -didn’t- help build it!
If you're going to get involved in a torment nexus project, at least do yourself a favor and research the different comp packages being offered and choose the best one.
And of course, a healthy skepticism of "the man" is essential when you're busting your ass building his unfathomably horrifying torture apparatus for him 40-50 hours a week (excluding PTO days and parental leave of course).
Thanks for the post. For https://Document.bot (Kinda Cursor IDE for knowledge workers), im already trying to improve the harness (besides spotting bugs) using hillclimb experiments. More and more i'm using a AI harness engineering skill in Codex to further improve the app. This blog post helped me to improve the skill a bit.
Great article! I am currently writing my second harness (first was in Emacs Lisp, using Emacs as UI; second is a command line coding agent written in Common Lisp).
If anyone wants to argue that it is inefficient writing your own AI coding harness, I wouldn’t disagree. That said it is satisfying to have long coding sessions using my own tools.
The article is a valuable resource, thanks to the author.
I stuck a lisp interpreter inside a web server and its a pretty cool pattern. I do have a basic harness but I'm still kind of just using the REPL from an external agent over REST, which seems to work just as well without having to go through the brain damage of building a harness
Thanks for the really nice in-depth post! Hoping for a future one about:
"Much recent work on auto-research, self-improving agents, and evolutionary program search can be organized around this question. Other work on model self-play, synthetic data, test-time training and a broader theme of continual learning also matches the RSI vision (e.g. Yuan et al. 2024, Chen et al. 2024), Zhao et al. 2025, Choi et al. 2026)) but they will not be the focus of this post."
Really good points Lillian. Agreed on keeping the evaluator outside the loop that evolves the harness.
One practical failure mode I’ve have experienced in my agentic harness tasks similar to the “weak evaluators” point: an incomplete check suite that still reports full success. That’s was worse than a weak evaluator, because it made it look correct and decisive.
After several trial and error, what helped was fail-closed on coverage i.e if the fixed checks for each operation aren’t all there, nothing ships / nothing gets sent. Otherwise you can “pass” while never running the cases that would have failed.
Isn't the harness basically where the frontier model companies can capture value and create a moat of sorts? I am also curious about building a harness for fun but would expect it to be more interesting in a scenario where I can self host an open weight model.
What motivates the people who build their own ChatGPT/Claude harness for example? And how do you keep it tuned with the rapid development of frontier models.
The basis of the argument is that the labs are constantly pushing updates to their system prompts that are used in claude code or codex, which are exceptionally bloated and change the sand beneath your feet with every update.
By rolling your own harness, as long as you keep up with the latest advances and changes in the ecosystem, you capture a lot of the 'control' that LLM-based development feels like it strips from you.
Obvious disclaimer that I use pi.dev when I am aiming for consistency, not absolute quality. Custom harnesses on pi are what I ship, claude code is still my fallback if I need to make sure a PR is the highest absolute quality
My own opinion is that the "harness companies" (Anthropic, OpenAI, Cursor etc.) are fucked long-term since customers will tire of that quicksand. As the models improve, we will tire of that harness crap.
> Isn't the harness basically where the frontier model companies can capture value and create a moat of sorts?
They're trying. I see a few main avenues:
1. Fitting the models to their specific harness, so that if you want peak model performance, you're stuck with their harness. But this only works if there aren't alternative models that are similar in capability or good enough that don't have that restriction.
2. Locking you out of the harness so that you eventually are just letting it do "stuff" with your data and system, although you don't get to see what the stuff is or why it's doing it. This is the pattern discussed in that Earendil blog post, "The session you cannot take with you." Encrypted reasoning tokens, secret agent prompts, perhaps eventually not even seeing what files are being read or what data is being sent back to their server. This way, you are also shackled to their harness because nothing is portable. But it only works if you trust them implicitly and don't have alternative models and harnesses that don't treat you this way.
3. Tying subscription pricing to the use of their harness, so that it's financially punitive to use another one. This is what Anthropic does. But again, it only works if there aren't alternative models and harnesses that work similarly well for you and don't have that restriction.
4. Marketing. Anthropic is leaning into this one heavy, from what I can tell, based on the constant ads I see for Claude Code. Can it work? People do overpay for things like vodka in fancy bottles that are chemically identical to the cheap stuff. Is Veblen-good AI a trillion dollar business, though?
I think the pressure from open source innovation in models and model tooling is going to make it a tough row to hoe. But I'm biased, as I'm actively rooting for Openthropic's demise.
I don't think so. A basic consumer-oriented harness is a commodity (zero moat: you can ask one harness to write you another).
State-of-the-art models have more of the workflow sensibilities built in, and don't need as much help from the harness.
Where harness helps the most is very customised personal workflows (not a textbox for a prompt, but an end-to-end IDE for how you develop software) But then it's best to have your own, rather than some closed-source rigid product.
Harnesses help smaller models, and very tightly hand-holding harnesses are needed for models runnable locally. I think that is very valuable to users, but avoiding paying for the biggest models is the opposite of what the frontier labs want.
one form of very effective self-improvement that coding agents do all the time:
install or build stuff that they can then use
it changes the environment instead of the agent/harness but in a sense how separate is the agent from its environment and why do we apply this distinction re self-improvement?
animals and humans do the same thing and are great at it, without 'self-improvement' with emphasis on the 'self'
Blame Descartes? I don't know. Extended theories of cognition all get shot down, even analytic philosophers who are accused of being more "continental" still primarily hold mind-body dualisms
I think the biggest problem with RSI is the tendency of models to try and find "clever" solutions when presented with very hard problems (aka cheat).
I just got done asking gpt5.5 to make some suggestions that would improve the convergence properties of a symbolic ML experiment and the very first thing it did was add an instruction for directly emitting bytes and a way to store those verbatim in the model. Perfect score achieved in record time but the result is pointless.
If you know the direction of improvement you need to head in, I think the current models could take you there. However, I do not think they are capable of deciding which direction is best, especially not when given some scalar quantity to min/max and autonomy over that process.
I've been working with self improvement harness a little bit and one thing i've come to conclusion is harness task fit. The learning can be significantly improved if we understand the behaviour of task and how it should be learned.
I'm pretty sure a general solution will definitely exist which will do fine, but we are yet to see one.
Agree with the sibling comment that an example would be useful. I'm also curious what kinds of tasks you have in mind. Theoretically almost every knowledge work task boils down to the file editing that agent harnesses already are built around (which probably explains the second half of your comment).
Thinking that harnesses and agents can improve without human involvement, even if it works, will yield a much lower growth rate than if a human gets involved in the loop.
can you elaborate this part please? "The workflow graph also emphasizes the model analyzing its own trajectories and failure cases and then iterating on its progress through an “agent runtime” rather than a static prompt template."
I’ve been thinking about how we practically implement this at an organizational layer for large codebases. There’s clearly alpha to be had in optimizing AGENTS.md / skills / tools / … to improve performance, quality, and cost efficiency of an agent. The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever.
The first step I see towards this is building a generic, reliable, and accurate *fitness function* for codebases - turning PRs into gradeable tasks that an agent can solve, and improve on.
I’m pretty curious how others are optimizing the coding agent harness now, as this has been a huge pain point for my company (we’re pretty much relying solely on vibes).
I literally hold a "retro" with my agent after a session.
> Let's conclude with a retro. Did you run into any issues during this session that you think could be improved? Any failed tool calls, confusing docs/prompts, or tricky wording that took you effort to figure out, etc? Any final thoughts that you want to raise? Anything minor you didn't mention? Help make this codebase easier for the next agent to work in.
It has ideas for coding changes, spots unmentioned small bugs, suggests invariants (I have a high-level list of invariants to keep), principles to adopt, lint rules, tooling tweaks (I've built SO MUCH agent-tooling; so many useful bash scripts), skill-file updates, follow up work, all kinds of stuff.
Listen to your agent's whinging.
Ive been building my own ai voice agent harness from scratch for close to a year now and following good software architecture practices is a good start. So those rules have to be coded in agents.md somewhere also really helps to have a "vision" section or "spirit of the project" section that describes what the end goal vaguely looks like and things I care about in achieving for the project. This prevents agent from being brittle and "single minded" about its work. But yeah vibes most of the time is how I've also been doing it, but I did find one very important thing that has really sped up my work. So I figure I'd share it here. And that advise is to ignore front end design at all costs until the very end and you are ready to launch. UI related woes kill any type of inertia and are responsible for most of the issues. So I decided abandon all UI stuff until the very end and just focus on iterative refinement, cutting, and other back-end related work and its been great ever since. Ideas can be had, tested, validated/invalidates and you ' get stuck on the optimization thought loop. Recently I have started to consider how i can fully automate the development process as the capabilities are there but designing the instructions on how to do this well and how to handle niche cases without getting me involved takes careful planning in writing out the guide so that will be interesting to see once i get there.
Trying something like this in a reusable way at https://github.com/mateffy/gesetz
It only tries to solve the „quality“ aspect of that equation tho, at least for the code output.
But I think for actual evals, some heuristics about a coding agent session are needed. But since the input tasks are always different, it’s hard to make any KPI actually comparable.
Did the agent needing 3 rounds of feedback on a big task perform worse than the one that needed none, but only worked on a small change?
Just working on my own stuff so YMMV, but from a cost reduction standpoint, the two most outsized ROI wins came from making sure the agent runs all terminal commands in quiet and using a codebase index mcp to speed up understanding and reduce full file reads [0]. I've also found the latter also dramatically improved the quality of output.
[0] https://github.com/DeusData/codebase-memory-mcp
maybe a bit counter-intuitive but:
I found that removing
+ reducing tools to just 1 (sh)gives better results than having 'more' of them
(e.g. look at these traces to see more vs less in action:)
https://smolenv.com/t/nested-template-includes-60636/
not saying the right context does not help
(it definitely does!, but it's not trivial to provide the right context)
This is the right direction (Anthropic removing 80% of the system prompt https://claude.com/blog/the-new-rules-of-context-engineering...)
However what I wonder how we make these decisions on what to remove. Certainly some subset of the skills / AGENTS.md / prompt / tools do actually help agent performance, the problem is in identifying those, throwing away the rest, and validating that this didn't actually hurt the agent performance
'better results' in terms of what though? A benchmark, or code that I would actually click "approve" on in a pull request scenario?
apologies, I should have clarified the 'better' claim
Yes, my question is directed around how you are determining "task result." Is this measuring whether the code works, or whether it is maintainable?
As with human-powered coding, we read code far more frequently than we write it. It's worth spending a little extra (time|tokens) during authorship to make future maintenance feasible.
My CLAUDE.md, memories, and skills are all about either (a) adherence to project standards and guidelines, (b) product decisions which impact future code, and (c) instructions on how to prototype and work in my environment.
Removing these instructions would mean more turns with the AI to get the desired result.
in this case there was a hidden grader that checked if the implementation was correct (because that was the easiest thing to check), all 3 agents cleared this hurdle in all 9 runs
I agree, next it makes sense to try more open ended tasks + have humans (and/or multiple models) grade the runs and their results
You've just described what the AI industry refers to as "evals", a collection of which forms a particular benchmark. I definitely encourage you to define your own evals, because the public benchmarks are often either saturated (largely solved and only going to see small improvements going forward) or seemingly not predictive of real world performance. I could also go further and speculate that they are in the training data, although credible benchmarks avoid this, I'm just not sure how successful they are.
Thanks for the comment - I’ve actually been working on something to generate evals from private repos, as I 100% agree that public benchmarks are either contaminated (as OpenAI found with SWE-bench Verified), not high quality (as OpenAI found with SWE-bench Pro), or in the case of “good” benchmarks (Cognition’s Frontier Code comes to mind), non-representative of a specific repo’s work.
Sharing some challenges / learnings from the journey so far:
* Tests alone are non-representative: an agent can pass tests but write the code in a way that’s subjectively worse than another test-passing result
* However, tests are still important and the best form of deterministic evaluation we have
* Creating executable environments to run the tests is challenging, especially when considering that this should be doable for arbitrary repos
* Selecting tasks that discriminate is an art - they need to be challenging, but not too challenging, represent the variety of work that’s done in the repo, and contain tests that failed prior to the change
* LLMs / agents are very powerful at judging code - but doing so in a way that is calibrated, consistent, and representative of codebase standards requires careful rubric creation and grading
* I’ve shifted towards using agents (from “static” LLM calls) to generate the rubrics, and to later grade the rubrics. This is powerful but even more non-deterministic
> The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever.
I've built and my team uses plannotator [1] for specs and code review. It stores all human annotated feedback, which we compile from time-to-time and iterate on coding standard skills for automated review (that seems to get better with the more feedback we provide).
You could write a script that does the same for your PR review comments, which it sounds like you maybe do with a bit more structure.
[1] https://github.com/backnotprop/plannotator
IMHO training weights has peaked and now it is time for a training paradigm for prompts and code. We don't have the gradient descent here - but I think it can be more sample efficient because causal theories can be better than just correlations.
I am working on a unified theory in https://zby.github.io/commonplace/ - it is all agent edited so it might be hard to read, but hopefully we are catching most logical errors. Some day the llm prose will improve.
I have even a preliminary theory on what is needed for the positive feedback loop: https://zby.github.io/commonplace/articles/reflective-self-i... - (this is not stable yet - but I think you can give it to your agent to read :).
Is there any reason to think that training weights has peaked rather than is accelerating? It feels like now they are increasingly able to pick some low hanging fruit by using the models in order to improve themselves and test optimizations.
to be frank - mostly because they are now good enough to unlock the other learning paradigms
What date was the peak? If it is today it’s not something you can know so I assume you think the peak was many months ago.
I think it is now or close to now - and not because there is no more to be gained - but rather because it is now clear that you can gain more with the right agent environment.
But benchmark saturation is also something to account for.
We've using auto-research for harnesses and it's surprisingly powerful. It's amazing how many problems are easy to spot and fix from traces.
Few key things were required to get it working well: 1) let it read a heap of prod traces to spot real issues. 2) let it write it's own tools (example "loading context" goes from 20k tokens across 15 tool calls to 800 tokens and 1 call to session_context tool). 3) you need evals and val/test splits, it will reward hack. 4) you need proper tooling (synthetic users, synthetic tools) for it to be able to crunch for 12 hours and produce something. 5) you need the optimization target to be a reasonable size: not your 1M line codebase, but a lighter agent harness (pi harness, skill only, Kiln harness).
What is the fitness function , eval?
I am wondering when will harnesses generate their own RLHF/DPO training sets and just LoRA finetune the models they are running?
The quest for Torment Nexus continues
If we don't create the Torment Nexus first, somebody else who is much less responsible is gonna create the Torment Nexus before us. It's outright irresponsible to not take the lead, we might have to even give up on all safety concerns to make sure we make it to Torment Nexus IPO first.
If we don't do our best to awaken Dead Cthulhu, when He invariably awakens He will know that we didn't do our best and we'll be eaten last by the Great Old Ones upon Their return!
Since we all know that at this point, someone is definitely going to build the torment nexus, we all may as well get involved and help, so that at least we can exert some limited degree of control over the eventual character of the torment nexus. At least that way maybe we can make it a little less terrible. At any rate, it can’t make it worse than it would have been if we -didn’t- help build it!
If you're going to get involved in a torment nexus project, at least do yourself a favor and research the different comp packages being offered and choose the best one.
And of course, a healthy skepticism of "the man" is essential when you're busting your ass building his unfathomably horrifying torture apparatus for him 40-50 hours a week (excluding PTO days and parental leave of course).
https://fundthebasilisk.com
whats the other option, touch grass? not likely!
We cannot allow a Torment Nexus gap!
Eagerly waiting for the TormentBench.
Thanks for the post. For https://Document.bot (Kinda Cursor IDE for knowledge workers), im already trying to improve the harness (besides spotting bugs) using hillclimb experiments. More and more i'm using a AI harness engineering skill in Codex to further improve the app. This blog post helped me to improve the skill a bit.
Very interesting, will give it a demo.
Great article! I am currently writing my second harness (first was in Emacs Lisp, using Emacs as UI; second is a command line coding agent written in Common Lisp).
If anyone wants to argue that it is inefficient writing your own AI coding harness, I wouldn’t disagree. That said it is satisfying to have long coding sessions using my own tools.
The article is a valuable resource, thanks to the author.
I'm curious to learn more about your experiences here, what you've learned, if you've enjoyed using the REPL with agents
The dream of the AI lisp machine ! Did you build in the functionality that the agent can on the fly rewrite its own (harness)-code?
I stuck a lisp interpreter inside a web server and its a pretty cool pattern. I do have a basic harness but I'm still kind of just using the REPL from an external agent over REST, which seems to work just as well without having to go through the brain damage of building a harness
No, but a good idea. I want something simple, practical, understandable. I think I will write a short book on this general topic.
Thanks for the really nice in-depth post! Hoping for a future one about:
"Much recent work on auto-research, self-improving agents, and evolutionary program search can be organized around this question. Other work on model self-play, synthetic data, test-time training and a broader theme of continual learning also matches the RSI vision (e.g. Yuan et al. 2024, Chen et al. 2024), Zhao et al. 2025, Choi et al. 2026)) but they will not be the focus of this post."
Really good points Lillian. Agreed on keeping the evaluator outside the loop that evolves the harness.
One practical failure mode I’ve have experienced in my agentic harness tasks similar to the “weak evaluators” point: an incomplete check suite that still reports full success. That’s was worse than a weak evaluator, because it made it look correct and decisive.
After several trial and error, what helped was fail-closed on coverage i.e if the fixed checks for each operation aren’t all there, nothing ships / nothing gets sent. Otherwise you can “pass” while never running the cases that would have failed.
Very enjoyable article.
Isn't the harness basically where the frontier model companies can capture value and create a moat of sorts? I am also curious about building a harness for fun but would expect it to be more interesting in a scenario where I can self host an open weight model.
What motivates the people who build their own ChatGPT/Claude harness for example? And how do you keep it tuned with the rapid development of frontier models.
There is a really good video by the author of pi.dev (which I have used to build some of my own harnesses): https://www.youtube.com/watch?v=RjfbvDXpFls
The basis of the argument is that the labs are constantly pushing updates to their system prompts that are used in claude code or codex, which are exceptionally bloated and change the sand beneath your feet with every update.
By rolling your own harness, as long as you keep up with the latest advances and changes in the ecosystem, you capture a lot of the 'control' that LLM-based development feels like it strips from you.
Obvious disclaimer that I use pi.dev when I am aiming for consistency, not absolute quality. Custom harnesses on pi are what I ship, claude code is still my fallback if I need to make sure a PR is the highest absolute quality
My own opinion is that the "harness companies" (Anthropic, OpenAI, Cursor etc.) are fucked long-term since customers will tire of that quicksand. As the models improve, we will tire of that harness crap.
> Isn't the harness basically where the frontier model companies can capture value and create a moat of sorts?
They're trying. I see a few main avenues:
1. Fitting the models to their specific harness, so that if you want peak model performance, you're stuck with their harness. But this only works if there aren't alternative models that are similar in capability or good enough that don't have that restriction.
2. Locking you out of the harness so that you eventually are just letting it do "stuff" with your data and system, although you don't get to see what the stuff is or why it's doing it. This is the pattern discussed in that Earendil blog post, "The session you cannot take with you." Encrypted reasoning tokens, secret agent prompts, perhaps eventually not even seeing what files are being read or what data is being sent back to their server. This way, you are also shackled to their harness because nothing is portable. But it only works if you trust them implicitly and don't have alternative models and harnesses that don't treat you this way.
3. Tying subscription pricing to the use of their harness, so that it's financially punitive to use another one. This is what Anthropic does. But again, it only works if there aren't alternative models and harnesses that work similarly well for you and don't have that restriction.
4. Marketing. Anthropic is leaning into this one heavy, from what I can tell, based on the constant ads I see for Claude Code. Can it work? People do overpay for things like vodka in fancy bottles that are chemically identical to the cheap stuff. Is Veblen-good AI a trillion dollar business, though?
I think the pressure from open source innovation in models and model tooling is going to make it a tough row to hoe. But I'm biased, as I'm actively rooting for Openthropic's demise.
I don't think so. A basic consumer-oriented harness is a commodity (zero moat: you can ask one harness to write you another).
State-of-the-art models have more of the workflow sensibilities built in, and don't need as much help from the harness.
Where harness helps the most is very customised personal workflows (not a textbox for a prompt, but an end-to-end IDE for how you develop software) But then it's best to have your own, rather than some closed-source rigid product.
Harnesses help smaller models, and very tightly hand-holding harnesses are needed for models runnable locally. I think that is very valuable to users, but avoiding paying for the biggest models is the opposite of what the frontier labs want.
Careful about over fitting to the point of cheating... [0]
0 - https://arxiv.org/abs/2604.11806
one form of very effective self-improvement that coding agents do all the time:
install or build stuff that they can then use
it changes the environment instead of the agent/harness but in a sense how separate is the agent from its environment and why do we apply this distinction re self-improvement?
animals and humans do the same thing and are great at it, without 'self-improvement' with emphasis on the 'self'
Blame Descartes? I don't know. Extended theories of cognition all get shot down, even analytic philosophers who are accused of being more "continental" still primarily hold mind-body dualisms
I think the biggest problem with RSI is the tendency of models to try and find "clever" solutions when presented with very hard problems (aka cheat).
I just got done asking gpt5.5 to make some suggestions that would improve the convergence properties of a symbolic ML experiment and the very first thing it did was add an instruction for directly emitting bytes and a way to store those verbatim in the model. Perfect score achieved in record time but the result is pointless.
If you know the direction of improvement you need to head in, I think the current models could take you there. However, I do not think they are capable of deciding which direction is best, especially not when given some scalar quantity to min/max and autonomy over that process.
I've been working with self improvement harness a little bit and one thing i've come to conclusion is harness task fit. The learning can be significantly improved if we understand the behaviour of task and how it should be learned. I'm pretty sure a general solution will definitely exist which will do fine, but we are yet to see one.
Agree with the sibling comment that an example would be useful. I'm also curious what kinds of tasks you have in mind. Theoretically almost every knowledge work task boils down to the file editing that agent harnesses already are built around (which probably explains the second half of your comment).
I haven't understood a bit.
Can you make an example?
Thinking that harnesses and agents can improve without human involvement, even if it works, will yield a much lower growth rate than if a human gets involved in the loop.
All this work for something we already solved 40+ years ago with ontologies and expert systems
What do you mean please give more information.
They say engineering but it's more a soft science.
It's engineering management I suppose.
HN seems to think LLMs are hard science, despite all the evidence they're basically cultural artifacts generated by complex models.
in america AI harnesses you.
Fascinating article, thank you!
The simplicity is the point
can you elaborate this part please? "The workflow graph also emphasizes the model analyzing its own trajectories and failure cases and then iterating on its progress through an “agent runtime” rather than a static prompt template."