You should read it especially now when more and more code is written by LLM. The important thing is not the code itself but your mental model of the software you're building. Sadly we seem to be moving away from it. We're accumulating more and more code that we don't understand or haven't even read.
I was going to say that an LLM can't do this, because it loses everything at the end of the session. But... could an LLM write out its "state" or "understanding" so that you could recover that for the next session? Do any LLMs currently have that ability?
It's very common, but (like most things with LLMs) it's not as deterministic as you might imagine. A common technique for agents is to have them create a "handoff" document (usually markdown) that summarizes the previous session-- goals, important files/links, etc. There are dozens of proprietary ways of doing this, and Claude Code automates the process with its /compact command and even does auto-compaction as you reach your context limit. ChatGPT has been doing autocompaction since the beginning as it started out with a comically small context window.
The problem with auto compaction is that you aren’t given the opportunity to review its compacted understanding to confirm that it’s correct or doesn’t contain large omissions. I try to avoid letting it compact whenever possible and stick to plans that I review because it seems to get extremely dumb after an auto compaction.
Yeah, I still find Opus to be pretty unreliable once you get past around 150K tokens, so I usually run a custom hand-off command at that point that extracts specific elements to specialized documents. The command contains a "Documentation Map" with single line summaries of each of those documents to help the agent sort everything out. Like most memory systems, it works pretty well around 80% of the time. I messed around with RAG and other complex solutions, and I never got much better results than my KISS system.
This brings up a philosophical question. Are we willing to hand over the role of "theory building" to LLM if that's even possible? If yes, what will be the role of human beings?
It may destroy many foundational assumptions that humans have had for thousands of years.
In theory maybe in some sense, but if we read Naur's definition of "theory" in a more strict or philosophical way, they can't in full. An LLM can't build a theory, because it doesn't have "real" experience, it's essentially just following rules. It also can't really argue or justify its choices like a person can.
This is discussed in the "Ryle's Notion of Theory" section of the original essay.
The name "theory building" doesn't really resonate with me - I think effective design ("programming" if you will) is more about things like decomposition, factoring and representations.
The larger the project the more ways there are that you could decompose it, but only some of these are going to have good outcomes in terms of things like a concise flexible implementation, easy to read/write, debug and extend etc.
You are mentally exploring the alternatives trying to find the ideal factorization that minimizes complexity, keeps interfaces between parts simple and friction-free, and results in an implementation where the code almost reads like a high level description of the requirements, with additional levels of detail only exposed as you descend each level of the implementation.
I can't off the top of my head think of a super pithy way of expressing it, but optimizing the factorization and representations being exchanged between parts (the two go hand in hand) is the key. How do you reduce the requirements into a design with the fewest moving parts and simplest interfaces between parts. It's kind of co-evolution in a way.
In my mind design and theory are inseparable. Design is the accumulation of many design decisions. Theory explains what influenced those decisions.
Design needs theory to be intentional. It can of course be accidental (”seems to work, I guess”) or intuitive (”i know in my guts this is right but cant explain it”).
While both can end up with functional systems, if you cant vocalize the design journey the system is not very maintainable in the industrial sense (hence - theory is the vocalization of the design and the forces that influenced it).
Most of my posts have aged terribly in the age of AI (especially the ones I didn't finish...so long, extended discussion of how to use a lab notebook when debugging, we hardly knew ye. Claude fixes our bugs now) but one job that engineers still have is the collection and retention of context that AI doesn't have and can't easily get.
This is helps describe my biggest pain point when engineering a program with LLMs. They do not have the full theory of the program, which makes things difficult. Additionally, the more hands-off approach to programming (even when I try to maintain involvement as much as I can) means that I lose the clear conceptualization of that piece code. I'm still trying it to see if it can work, but it is definitely a vibe shift from making 20 micro-architectural decisions in every function.
The only reason I recommend this paper is because I encounter so many people that have a very myopic view of the software that they’re building. They are focused on individual features and how to quickly made them happen regardless of what happens to its cohesiveness. You start to talk about interfaces and contracts and they’re like a deer blinded by a car’s headlights.
I wouldn't start to think of someone as a real developer unless they've at least designed & written something of at least 10K LOC or so of complexity from scratch a few times. At least, you're not going to be able to understand these lessons and characterizations of programming in the large unless you do have at least that level of experience.
The larger and more varied projects that you have designed from scratch, the more you start to understand what programming/designing is really about.
You should read it especially now when more and more code is written by LLM. The important thing is not the code itself but your mental model of the software you're building. Sadly we seem to be moving away from it. We're accumulating more and more code that we don't understand or haven't even read.
I was going to say that an LLM can't do this, because it loses everything at the end of the session. But... could an LLM write out its "state" or "understanding" so that you could recover that for the next session? Do any LLMs currently have that ability?
It's very common, but (like most things with LLMs) it's not as deterministic as you might imagine. A common technique for agents is to have them create a "handoff" document (usually markdown) that summarizes the previous session-- goals, important files/links, etc. There are dozens of proprietary ways of doing this, and Claude Code automates the process with its /compact command and even does auto-compaction as you reach your context limit. ChatGPT has been doing autocompaction since the beginning as it started out with a comically small context window.
The problem with auto compaction is that you aren’t given the opportunity to review its compacted understanding to confirm that it’s correct or doesn’t contain large omissions. I try to avoid letting it compact whenever possible and stick to plans that I review because it seems to get extremely dumb after an auto compaction.
Yeah, I still find Opus to be pretty unreliable once you get past around 150K tokens, so I usually run a custom hand-off command at that point that extracts specific elements to specialized documents. The command contains a "Documentation Map" with single line summaries of each of those documents to help the agent sort everything out. Like most memory systems, it works pretty well around 80% of the time. I messed around with RAG and other complex solutions, and I never got much better results than my KISS system.
This brings up a philosophical question. Are we willing to hand over the role of "theory building" to LLM if that's even possible? If yes, what will be the role of human beings?
It may destroy many foundational assumptions that humans have had for thousands of years.
In theory maybe in some sense, but if we read Naur's definition of "theory" in a more strict or philosophical way, they can't in full. An LLM can't build a theory, because it doesn't have "real" experience, it's essentially just following rules. It also can't really argue or justify its choices like a person can.
This is discussed in the "Ryle's Notion of Theory" section of the original essay.
The name "theory building" doesn't really resonate with me - I think effective design ("programming" if you will) is more about things like decomposition, factoring and representations.
The larger the project the more ways there are that you could decompose it, but only some of these are going to have good outcomes in terms of things like a concise flexible implementation, easy to read/write, debug and extend etc.
You are mentally exploring the alternatives trying to find the ideal factorization that minimizes complexity, keeps interfaces between parts simple and friction-free, and results in an implementation where the code almost reads like a high level description of the requirements, with additional levels of detail only exposed as you descend each level of the implementation.
I can't off the top of my head think of a super pithy way of expressing it, but optimizing the factorization and representations being exchanged between parts (the two go hand in hand) is the key. How do you reduce the requirements into a design with the fewest moving parts and simplest interfaces between parts. It's kind of co-evolution in a way.
I think you did use an appropriate word: design.
In my mind design and theory are inseparable. Design is the accumulation of many design decisions. Theory explains what influenced those decisions.
Design needs theory to be intentional. It can of course be accidental (”seems to work, I guess”) or intuitive (”i know in my guts this is right but cant explain it”).
While both can end up with functional systems, if you cant vocalize the design journey the system is not very maintainable in the industrial sense (hence - theory is the vocalization of the design and the forces that influenced it).
I wrote a series of blog posts about this a few years ago: https://creating.software/essays/theory_of_a_program/, some of the few I ever actually finished, lol.
Most of my posts have aged terribly in the age of AI (especially the ones I didn't finish...so long, extended discussion of how to use a lab notebook when debugging, we hardly knew ye. Claude fixes our bugs now) but one job that engineers still have is the collection and retention of context that AI doesn't have and can't easily get.
This is helps describe my biggest pain point when engineering a program with LLMs. They do not have the full theory of the program, which makes things difficult. Additionally, the more hands-off approach to programming (even when I try to maintain involvement as much as I can) means that I lose the clear conceptualization of that piece code. I'm still trying it to see if it can work, but it is definitely a vibe shift from making 20 micro-architectural decisions in every function.
By the curry-howard correspondence, this is literally true.
The only reason I recommend this paper is because I encounter so many people that have a very myopic view of the software that they’re building. They are focused on individual features and how to quickly made them happen regardless of what happens to its cohesiveness. You start to talk about interfaces and contracts and they’re like a deer blinded by a car’s headlights.
I wouldn't start to think of someone as a real developer unless they've at least designed & written something of at least 10K LOC or so of complexity from scratch a few times. At least, you're not going to be able to understand these lessons and characterizations of programming in the large unless you do have at least that level of experience.
The larger and more varied projects that you have designed from scratch, the more you start to understand what programming/designing is really about.