This is the thesis behind the "Information Theory, Inference, and Learning Algorithms" course that was taught at Cambridge University.
> Why unify information theory and machine learning? Because they are
two sides of the same coin. In the 1960s, a single field, cybernetics, was
populated by information theorists, computer scientists, and neuroscientists,
all studying common problems. Information theory and machine learning still
belong together. Brains are the ultimate compression and communication
systems. And the state-of-the-art algorithms for both data compression and
error-correcting codes use the same tools as machine learning.
I had a long ranting comment I deleted. I just don't like this trend of people presenting work in a way that makes you think some combo of 1) they discovered from scratch themselves 2) it's new 3) they didn't try to cite or acknowledge where they learned it/point to good sources 4) they don't really care about trying to teach something deeply, they want shiny stuff that makes them seem deep.
This post references specific parts/calculations, but you'd never know it was not news if you didn't know better.
The author of the post uses standard terminology like entropy coding and arithmetic coding, and cited a paper "in 2023, Google DeepMind released a paper arguing that language modeling and compression are two views of the same thing" which discusses it further.
This blog post is great. Well explained, and clearly took a lot of effort.
I don't interpret it as them claiming to have to discovered it independently.
You expect every blog post to find the earliest relevant paper to cite, just so one could look at the year (without reading said paper - which would have made clear that the connection isn’t recent) to assess novelty? I don’t think that’s reasonable.
It’s a blog post. If it was, say, a peer reviewed paper by Hinton or LeCunn that fails to cite Schmidhuber, that would be reasonable criticism in my opinion. (Spoiler: they fail to cite him)
Why would blog posts not be subject to such criticism?
Either the author knew of prior work that argues the same thing and they ignored it, or they didn't know. And if one writes a 1000+ word article premised on this idea, wouldn't one be presumed to know at least in which century the idea originated from?
Arguably these kind of blog posts should be more subject to such criticisms, because the blog posts purport to "teach" the general public about a concept in an authoritative tone (or at least the author seems to pose as knowledgeable in the subject), while for academic papers, everyone who actually reads the paper knows where the ideas came from anyway and it's mainly an issue of attribution (and maybe about fairly distributing the citation count...)
If the blog were about calculus, and stated that an elegant proof of the Fundamental Theorem of Calculus could be found in such and such undergraduate textbook, would you be upset that the citation wasn't to either Newton's or Leibniz' work?
The two contributioms that come to mind are the Hutter cash prize for the best compressor of the English Wikipedia and the work on PPM compression-based
text classification by the late Prof. Ian Witten's group at Waikato (NZ) [1,2].
The model that best compresses the input string was likely generated by the distribution from which the compression model was 'trained'
"Understanding" is clearly linked to compression. Taking observations and coming up with a more compact representation that explains them, analogous to coming up with a compact set of axioms that generate facts, or a small Turing machine or short program that generates a list of strings.
Intelligence is a broader concept but definitely involves understanding how a system/envoronment works and making predictions about its unfolding, especially actionable ones that allow you to steer that state towards some goal states.
The post says this is all part of gzip and LLMs, what are you saying? I’ve been using gzip my entire life. I read between the lines “this is common knowledge” throughout the piece. Throwing in some names and dates only makes this super clear story harder to read (and more like studying then the playful exploration this post was intended as).
A short paragraph at the end on the origin of these ideas can be an easy way to dispel the misconception of potential beginner readers that the insights are novel.
I'm of two minds here. The pro is that the "you could have invented this" walkthrough from first principles is more engaging than "and then so and so introduced this term in 1972 and the definition is such and such". This style is a reaction to that boring and dry teaching style and tries to push towards what eg Feynman pointed at in the Brazil critique.
The con is that you don't get to understand and see any of the history of the ideas or even the ballpark when it was discovered, you attribute it to the blog mentally and you don't know what is how new or old and can't reference it properly when talking to others.
Often the most straightforward way to walk through an idea while teaching it is not the same order that the ideas were developed, and might not even use the same set of ideas in building up to it, so it can be tricky to get both the best explanation of the idea and the historical context in at the same time without making things more confusing.
I like to see ideas presented as the evolved. Each solution is developed as a perceived reaction to the shortcomings of the previous. This becomes a contrast and comparison as to why one idea is appropriate for a particular context.
I'm speaking in general terms and I have not studied much on this subject so I don't have specific suggestions for this post. Please refer to earlier posts in this comment chain for a general idea of what citations would be useful.
They did not come up with the ideas themselves, so they got them somewhere. Follow the source and all the citations show up. It must be a modern thing where online blogging randos pretend they are all geniuses.
Better too assume they are just not aware. Technology is multi-layered cake of development. I have no doubt the only reason I know a lot of details is that I lived their development.
When standing on the shoulders of giants it's hard to tell what is below them.
You're reading this the wrong way I think, citations aren't given because its obviously a pedagogical article about well established stuff. Much like you wouldn't give citations in a blog post explaining calculus.
One could give citations regarding calculus it's pretty interesting. Since it was done twice by both Newton and Liebniz. There must have been cultural developments in the 1660's that demanded calculus be invented.
If you followed the data compression scene in the 80s and early 90s, there were plenty of reinventions of LZ-ish and Huffman-ish algorithms (I also coded my own variant...), and people even tried to patent some of them, so at least for the basics I think it is something that many can discover independently; of course in these times, it's more likely they didn't.
The first sentence says she came across it when reading about compression. I didn't read that as her claiming to have discovered the idea or that it was a new idea, I read it as "today I learned". I think somebody who was unfamiliar with how compression algorithms or language models work would find this an approachable and interesting introduction. Not everybody studied information theory.
Perhaps tangential to your point, but I often write blog posts (although finish and publish far fewer than I start) where I write about something as it has occurred to me, informed by things I've absorbed no doubt, but without specific research. In such cases I explicitly avoid searching out prior work as a) seeing that something is well discussed and explored can take away the motivation to explore (in the same way reading puzzle solutions before starting might), and b) to avoid having green shoots of ideas shaped by the current of existing consensus. Now that doesn't mean I don't come back after doing my own thinking to see what the more well developed literature of people cleverer than me, who've thought far longer than me think; I just don't want to snuff out my own exploration at the start.
As I say most of these I never publish as I'm mainly using writing as a vehicle for thought, but when I do I'm never sure how to flag them. I don't want (imaginary, lets be honest) readers thinking I'm deluded into thinking I've found something new. I want to come up with a tag I can put on them which adds a pithy disclaimer card at the top or something so I feel more comfortable publishing them.
I'm glad to see someone feels similarly. There is nothing wrong with ignorance, but there's no excuse mistaking learning for invention. Especially from someone bearing the title "Developer Educator"
I don't think it's the case here, but worth noting too that LLM-written blog posts adopt this tone seemingly by default.
Never the least bit of surprise, wonder, doubt, or frustration to get in the way of the steady staccato beat of metaphors, conclusions... and three-item lists.
> I'm glad to see someone feels similarly. There is nothing wrong with ignorance, but there's no excuse mistaking learning for invention. Especially from someone bearing the title "Developer Educator"
>> a Developer Educator at ngrok with a passion for nerd-sniping developers.
And yet to this day, in AI threads, so many people act shocked and surprised if you dare follow the obvious implication and claim that understanding is a form of lossy compression.
Of course it is but again "X is just Y" is often used to mislead. A brain is just neurons! A computer is just transistors! An LLM just predicts the next token! It's just like a parrot! It's just like a blurry jpeg of the internet! Kinda yes, but what do you use this for? It's a bad intuition pump is it leads people to conclude demonstrably false things about capabilities.
Shorter description isn't understanding, let alone of it is lossy.
When you shorten a description in a lossy way, you are deciding a priori that some differences in the object don't matter, and it's not because you understand the object, but because it serves your goal of shortening the description.
You can compress syntax, losslessly even, with zero understanding of its semantics. Zero understanding not only imbued into the compressor/decompressor, but even the designer of the compressor doesn't require understanding the semantics. Actually, even of the syntax.
A compression program can compress a book written in a language that the author of the program doesn't understand, on a topic he knows little about.
Finding common characters and building a list of words is a low level type of understanding. Doing it better does actually start directly representing syntax patterns and that's a less-low level of understanding.
I think "losslessly even" is the wrong way to think about it. Lossless compression often requires less understanding than high quality lossy compression. If you can do a lossy compression that correctly decides what details are unimportant, that's a good sign of understanding.
> I think "losslessly even" is the wrong way to think about it. Lossless compression often requires less understanding than high quality lossy compression. If you can do a lossy compression that correctly decides what details are unimportant, that's a good sign of understanding.
This is the crux and reminds me of things like mp3 that exploit the nature of human hearing being limited to a frequency range.
We model the data. The model, hopefully, captures something real in the data. If it does, then it's fair to say that we understand the data better.
But it's frankly a philosophical question what's real or not. No model is going to capture absolutely everything about the thing it models - at that point, it would be the thing. The best we can hope for is that it captures everything we care about.
And no experiment or metric can tell you if you care about the right things. At best it can tell us if we care about a thing given other things we care about. "No cares in, no cares out".
To make it a little more concrete: you could compress a string from back to front. You could build an LLM to help you do that. If you care about file size, that's almost certainly a bad idea, the forward LLM will be better for that purpose. But are there purposes for which the backward LLM might be better? I think that's not so hard to imagine. Often we wonder about "what came before".
I'm not aware of a better definition of "understanding" that would allow me to tell whether some system "understands" some other system. Do you happen to know one?
See: A. M. Turing (1950) Computing Machinery and Intelligence. Mind 49: 433-460.
I mean, my interpretation is that the question Turing tried to answer is equivalent to "How can we determine whether machines understand humans/human thought?"
This only works when both systems can talk about pretty much arbitrary things, but if you want a more general method for less complex systems, perhaps having one system simulate another system is sufficient. (Which is also another Turing invention)
There are many people who would claim that passing the Turing test is insufficient to show "understanding" (compare for example the Chinese Room thought experiment).
Yes but it's (kind of?) a definition as you asked for.
At this point, I am unaware of a better definition. I know the Chinese Room argument (and I disagree with it), but I'm not aware whether the proponents of that argument have a better definition of understanding other than "well, the Turing Test isn't enough"...
---
PS: Interestingly the issue of compression is highly relevant regarding the Chinese Room argument -- the essential element in the Chinese Room argument is that the information is not compressed...
I was speaking in the context of humans. When someone teaches you, the content coming from the teacher is very compressed. One decompress it when they can generalize and apply it. So understanding is compressed, but is not the act of compressing. I mean it is not compressed from a larger data or made by compressing a larger data. The larger data it represents never existed. It is like the definition of a fractal...
I find it bothersome that language works this way. You can spend your whole life discovering things that are well known by the rest of the world. But the minute that you mention to a large group that you “discovered” it, suddenly you’re taking credit for discovering it for all of mankind.
I'm glad you're pointing this out because not only are these old insights, but I'm also pretty sure I've seen variations of this blog post years ago on even HN already.
The author acting as if they discovered this independently had me feel the exact same way. Kinda irritating and almost ... disrespectful? Not sure of the right words to describe it tbh
> The ts_zip utility can compress (and hopefully decompress) text files using a Large Language Model. The compression ratio is much higher than with other compression tools.
It's not only an old idea it's been totally done already.
As much as I want to, I sadly don't think Information Theory makes sense in this setting, and I really wanted to believe this. When Shannon made his theory of information, he was always dealing with informational representations on the abstraction level of bits. At Bell Labs, a lot of the work was on the compression of data for transfer over telephone wires. Entropy coding, later codexes like algorithmic coding, and all compression on this level assumes that you have a bit-based X, and you compress it.
However, in deep neural networks, you are dealing with compression on different levels of abstraction. How do you decide what shared features a peacock and a palm tree have? At what scale should they be represented? How do you deal with invariance under affine transforms? Do you want to open the box of invariance under non-affine transforms?
When you start looking at what it would mean to compress feature representations, you immediately get to the question of data. You realize that Shannon simply was given a form of a very low abstraction data and that information theory came out to handle data at this level, but it's not suited for the data representations of many higher level modalities.
If you read Society of Mind by Marvin Minsky, which has aged well to about 80%, you can get the hint of the kind of abstractions that humans make and what would be needed to represent them, this is not representable in bits, you need to go to higher level shared features, and then you open all of the questions above as well as credit assignment, mutual information approximation, Fischer information between bayesians, etc.
> How do you decide what shared features a peacock and a palm tree have? At what scale should they be represented? How do you deal with invariance under affine transforms? Do you want to open the box of invariance under non-affine transforms?
The whole point is that the representation is learned. When you talk about various levels of abstraction, you're missing that all of these levels are representable with words and the relationships between them. That is verbatim what LLMs are optimized for. Interestingly, when you take an embedding, you do see that some transformations in embedding space actually hold which is quite interesting (e.g. tree + many ~ forest)
1) I am talking about representations beyond language models and language embeddings. If you take for example image, video, audio, 3D-spatial DICOM or combinations like VLMs. If you ask a language model to make an image of a Begonia ferox leaf without training it with images as well, it will not be able to represent this.
2) Language is already a higher-order lossy compressed abstraction made by humans to communicate fast and fill out the left out information with a learned prior. If you train a model on language only, it will not have the opportunity to have a non-compressed representation to make its own abstraction from.
3) If you are LLM-pilled and believe that we will be able to reach arbitrary levels of precise informational representation using language only, and that all abstractions that we may ever want can live on every single embedding layer in an LLM, your argument is fair.
1) If you ask a language model to make an image of a Begonia ferox leaf without training it with images as well
is this not true also of humans?
2) The idea that there is a non-compressed prior is already philosophical no? It seems to just be a rehash of whether an example of a "Chair" is a representation of a perfect Idea of a chair, or whether accidental things put together read as a chair. Its just dualism or not, Plato vs aristotle but for LLMs vs human learning.
If we do happen to just form an abstraction from less examples than an LLM, then its a question of efficiency in the learning algo and the capability to fill the gaps faster/better.
Obvs a world model and multi modal learning would close the gap between LLM latent space and what the human mind encodings might look like. But you can formulate a reality where humans do not have access to a non compressed representation (or even say that doesnt exist) and we are just more efficient at abstraction and recollection
3) There are multiple answers to this. One is that language can just be a prong in the learning modes available. Another is that language is many times redundant and includes supplemental information and nuances that despite being a higher order abstraction has enough additional information to reconstruct entire messages even after the lossy transmission.
Btw I am pretty against LLMs as path forward, but I have been interested in their ability to solve problems with Convergent solutions and their learning model as a lossy compression algorithm where instead of saving the actual data, you save a very good guesser of the next word. So if a text is part of its traning algo then you get almost a 1:1 (like a zip dictionary) but if its not, then he gives you something that plausibly could be in an imaginary Zip with that title.
Ah Sir David MacKay. I so respect him. Great explainer and speaker. He had built this text entry tool called Dasher [0] - that I'd heard him introduce at Princeton around 2006. It was basically an early language model that predicted which characters are more likely than others, given what you've already types and it would adjust the sizes of the available next characters based on their probabilities.
I also went to a couple of his (fantastic) undergraduate courses, and have a huge amount of respect for him.
That said, I think it's worth mentioning that Climate Change Without the Hot Air has aged pretty badly, and I'd be reluctant to recommend it to people who don't already have the background to understand what's aged well and what hasn't.
The high level approach of making high level numerical estimates makes sense, but it dismisses solar energy in about a page due to assumed high costs. It turns out that even if you're David Mackay you can still be caught out by exponentials :) I notice now that the version you link has some inline updates pointing out how off the assumptions in this section were, but it seems to me that's not enough; you probably need to redo the entire analysis based on what we know today rather than trying to make purely local adjustments.
On the other hand the point at biofuels are even more inefficient, and therefore a dead end even before you consider broader environmental impacts, are well made and something that is sadly not yet widely reflected in policy.
> biofuels are even more inefficient, and therefore a dead end
Only if energy density doesn't matter. But it really does, though. Battery powered electric trucking? Dead end. Battery powered aviation? Dead end. Battery powered shipping? Dead end.
[edit] Maybe there's some sustainable way to convert solar energy into sufficiently energy dense fuels that isn't biological, but so far it seems like seed oils or algae are probably the least bad?
> It was basically an early language model that predicted which characters are more likely than others, given what you've already types and it would adjust the sizes of the available next characters based on their probabilities.
"adjust the sizes of the available next characters": you mean assigning less bits to characters the most likely to come next, and more bits to the least likely characters to come next?
So it's not totally unlike adaptive Huffman coding but instead of updating the tree on already visited characters, here the tree with the weights of the possible following character is predictive?
It is infact a visual implementation of arithmetic coding, and he used it in his lectures to teach arithmetic coding in the lecture following the one on huffman coding. The video lectures posted earlier are highly recommended, best course I took as an undergrad
I suspect you're being snarky :-) but this is a really interesting question, and one that has had a lot of research done. I'm not current (I stopped following folks doing this research closely around 2019) but what we 'didn't' know about how brains work was still huge. Signaling levels, enzymes, the connectome, quantum effects, it is a really deep question.
That said, once we do get a working idea of how it works, and can perhaps synthesize a brain artificially with proteins, it will inform us on the next steps for silicon realization of that.
I think the point is that tremendous complexity can arise from relatively simple mechanisms. That is what life is, at many levels. I’m not at all convinced that the current LLM approach will yield something we can broadly call consciousness but saying that it’s a simple concept and therefore won’t support consciousness is a specious argument imo.
I completely agree, tremendous complexity can arise from simple mechanisms. Gleick's Chaos is a really good introduction to that. I was talking about the article though, and the mechanisms currently used for training and inference in LLMs. Those mechanisms are mathematically precise (unlike Chaotic attractors) and as the author points out, achieve the same function as compressors do in a strict bit pattern minimization role. Sometimes tensor math is pretty complex, like the FFT and DCTs on JPEG compression, but with the same inputs you get the same results. And while a JPEG will never decompress to a different image than the one that was compressed in the first place, LLMs do not 'infer' token streams that haven't been trained in their training process. The big difference here is that if you imagine a JPEG compressor that compresses 100 different images into one 'chunk', you can see how to provoke it to produce any one of the images it previously compressed. And with a bit of creativity you can have it express different images in different parts of the resulting composite. FWIW I looked at patenting something like this for digital cameras to give them more "shots" space for a given amount of SD storage.[1]
Given the way that models work in 'inference' mode (vs 'training' mode) you can't forward bias the result into the correct result when there are multiple forward results that have identical weights. It's the root cause of hallucinations, and you've lost information in the training phase that you can't then use to discriminate between the 'right' answer and an equally valid 'wrong' answer.
[1] FWIW I could never recover enough state to insure that the image it regenerated was all of the same image you took. So you might get the street but one of the houses might be a house that was in a different picture you took. That kind of bug. Mostly arising out of the same kind of problem you have with using hashes to find documents, when you get a hash collision two documents have the same hash, so you don't know which one to return.
I do not understand this intuition that "true consciousness has to be random". The things that make me me are highly deterministic!
> LLMs do not 'infer' token streams that haven't been trained in their training process
While we're at it, this is simply untrue (in-context learning) unless you generalize "token streams" so radically that it could be readily analogized to humans as well.
This self replicating proteins can do a lot of stuff that a program running inside a computer cannot do. Like producing secretions, generate electricity, generate light, generate force etc etc....
Compression and error correction also go hand in hand: in compressed data every bit carries more information and therefore errors are more detrimental. This is one of the results that Shannon phrased exactly in terms of entropy.
My PhD supervisor had a beautiful example. Take an English message and ‘compress it’ by removing vowels. It’s still intelligible because English has redundancy:
rrrs cn mk mssgs nrdbl
Similarly, an uncompressed message with errors (swapped characters) is also intelligible because of the redundancy of English:
erwurs lan nake wesaagis unfeatable
But now we do both: we compress the message AND add errors. The result should be much harder (if not impossible) to read:
…for years. Because it is so apparent if you actually try to look at the problem and what is being solved by it.
The extraction of features from a corpus, the features significant to certain solution, is always and since day zero - compression. As this is the definition of compression - efficient and potentially lossless feature extraction.
And the Hutter Prize for AI which measures how good AI is by measuring how well it compresses data is over 20 years old now just to really drive the point home.
It's basic information theory, which has been around since the end of WWII. It's a common topic today because some of its subtle insights are becoming increasingly relevant in our current era of AI, as we learn to understand these black boxes.
it was vaguely in my understanding of information & intelligence with compression; it was also brought up in several of the initial trials against AI companies where they discussed how the AI is akin to compression.
The page source appears to contain all the actual text within <p> tags, but structured in a completely illogical way. With JavaScript disabled, there are a bunch of shaded bars where the text should appear, which look like placeholders for something that hasn't loaded yet even though it was there from the beginning. The <p> tags don't even seem to show up in the DOM. (I didn't check closely, but maybe they're embedded in an inline script.)
This is actively user-hostile. The site is going out of its way to interfere with the most basic possible function of HTML, i.e., the presentation of minimally marked-up plain text. The needless complexity is especially ironic in the context of an article about compression.
The article is using probability where it really means proportion and prediction where it means evaluation. The mathematical equivalency is both much less surprising and less revealing once reframed.
If we consider the first example with the arithmetic code, the initial presupposition that only the characters A, B, and C appear in the string already reduces the entropy from 56 ascii bits to 14 bits (A vs Not A and B vs Not B for each character). If you further consider that you only need to distinguish B vs Not B if it's not A, then you can just represent As with a single zero bit and only represent the non-As as two bits (the first of which will necessarily always be a 1 bit). This gets you to 10 bits without even having the proportions of the string. Of course this would be a poor convention if there were say only a single A; in that worst case scenario you would need 13 bits, but simply knowing which character appears the most, without knowing by how much, 11 bits is the worst case scenario for a length 7 string with 3 potential characters. The last bit can be made implicit if you further choose the second conditional appropriately - i.e. if instead of B vs Not B we chose C vs Not C, our last bit would be zero and could simply be dropped meaning both 10 and a single 1 bit encode C - allowing you to encode the example string in just 9 bits and an arbitrary string of that length in 10, again regardless of proportions. That improvement over the arithmetic encoding result in the example is just a case of us cramming a little extra information into the encoding algorithm.
Arithmetic encoding is more clean and more easily extensible, it makes more sense to use than this custom encoding of 7 trits to binary but the point is the "probability" the article mentions is a superficial quality of life feature, not the secret sauce that is the actual key to compression.
Prediction is compression, but I am not sure if it is true the other way around.
It's obvious that an accurate predictor enables encoding only the data that the predictor gets wrong.
But a compressor can encode patterns that defy prediction by looking at the data as a whole. It doesn't have to look at everything in sequence as it arrives.
Applying transformations prior to entropy encoding often isn't just 'rearranging into an easier to compresss format' the transformation can be doing the job of peeking into the future. That makes the encoding a whole lot easier, but it is much harder to call it prediction.
Indeed. If you're going for a catchy generalization, at least write it correctly. Most compression is history, and only extrapolates under the assumption that "nothing changes".
there are dictionary compressors (decent compression, most common, fast), and statistical compressors (better compression, slower). Statistical compressors are much closer to LLMs in that an llm is learning statistics about the data too.
And yes, compression is history, that's what statistics are all about. Statistics can only measure the past to make a prediction about the future. And LLMs work in the same way. The context is the history, and given that history, it predicts the next token. An LLM can, almost trivially, be dropped into something like the PPM statistical compressor (it's just replacing one implementation of a markov chain with another).
Anything can only represent past measurements. Statistics is not an exception. But they don't make a prediction about the future. That comes from a model you have, and it often is implicit: "the linear trend from the last 12 months will hold in the next month" or whatever. So compression isn't by definition prediction. The other way around doesn't have to hold either, but in the case of LLMs it does.
I'll add Minimum Description Length to the mix. Under certain definitions and conditions, it equals the Bayesian Information Criterion plus an extra term, which I consider a very interesting result in this "two faces of the same coin" perspective.
This is exactly why I think they are one and the same. It's relatively trivial to just plop a (lossy) machine learned markov chain instead of one learned (perfectly) from the data into PPM. With zero changes to the rest of the algorithm.
Nope; there is a bit more nuance and the distinction is important.
Compression is functionally equivalent to prediction when the data distribution is exactly representative of all future problems. The story changes drastically if you want generalization -- because the test distribution could be arbitrarily different, even if it had the same support! Eg: you observe a rare edge case in your training data and (lossy) compression could simply ignore it. But if you wanted generalization in that particular part of the space -- either because an adversary was testing you, or for design freedom where you choose to build in that specific corner -- then you don't just want data compression, but good prediction performance on a test distribution which peaks in that corner.
Assuming that the training data distribution is exactly the distribution you will ever care for is implicitly doing a lot of the heavy lifting in the claim that compression = prediction, and I'm peeved at how much this statement is unthinkingly repeated like a manifesto.
There is nothing natural about the training data distribution, especially if the data generation process is exploratory while the downstream usage will be exploitative.
How does that invalidate the "compression is prediction"? If the future data is different and you failed to generalize, then the failure to predict means you got worse at compressing and have to spend more bits storing the new information. Conversely, if the future data is the same as that you've seen previously, you could predict it very well, and compress better as a result.
A maximally efficient compressor for the existing data distribution is not in general (and often will not be) maximally efficient for future data. The former may only be enabled by convenient local optima of the input distribution that a compressor accounting for the latter could not take advantage of.
For instance, consider the distribution of strings drawn from the language '0+'. Now consider the same for the language '[01]+'. A compressor looking at only the strings of the first language within those of the second can do a much better job if it does not have to account for future data.
This also relates distantly to the idea of overfitting in machine learning.
It might not be optimal, but it's not wrong to call it the best available guess. That's basically assuming Occam's razor / Solomonoff induction. Hutter published a bunch of work about what it means to have an "optimal" compressor and famously spent the past couple decades running a compression contest on the idea that it'd lead to insights in AI.
There is a distinction between a compressor for a fixed dataset and one for an unknown population from which we have a sample. The optimal compressor for the sample may be the single best guess for the population, but that's not what Solomonoff induction does. It begins with a prior that allows all possible programs, and it never assigns all probability to the single optimal compressor, so it has no problem with the all-zeroes example.
But the Hutter prize (of which I'm a big fan) is for ever-more-optimal compressors, and in fact many of the solutions don't generalise to other input data without stripping out various tricks.
I think it's better to think of the hutter prize as a challenge to see how close people can get to efficiently approximating AIXI without the exponential cost of AIXItl. The fact that winners don't generalize well is just that they're not great approximations, because the behavior is different in the limit.
The winner generalize really well. They're just incredibly slow.
Most of the neural compressors are trying to squeeze every last bit out by sacrificing runtime. You can speed them up a fair bit and they're pretty good. fast-cmix with a small priming corpus will probably be about twice as compact as something like lzma or brotli.
I’m not exactly sure what distinction you are trying to make, but a compressor that generalizes from currently observed day to potential future data _is_ a better compressor, as it will have better predictions as it observes more of the source data distribution. Maybe you are making a point about whether compressors can “learn” after compressing the input data, but this also isn’t a fundamental property of compressors. Also compressors do have to deal with non-stationary data sources, so it also isn’t correct to say a compressor only models stationary distributions.
But your definition of compression as prediction is tautological. How would you operationalize that into a measurable metric of compression?
The crucial idea is that once you do that and start optimizing for compression, at some point more compression on the training dataset can lead to worse predictions on the futures you care for. Which is very incompatible with the poster statement.
> because the test distribution could be arbitrarily different,
In that case all bets are off for all machine learning algorithms anyway. If the changes are not arbitrarily different then some of the lost performance can be recovered.
You talk about generalization a lot, but perhaps you are not aware that the technical term is defined under the assumption that test and train distribution are the same.
The record setting file compressors are all very generalizable. This comes about because you score compression by adding the program size (including any training data or dictionary) to the compressed data. If they didn't score it this way you could just ship a dictionary of the test data and your program just prints that. So of course they combine it and effectively measure the Kolmorogov complexity.
This means that training data isn't that helpful to top of the line compressors. eg. Fabrice Bellards nncp that's currently the record holding compressor for enwik9 is an LLM that learns on the fly. It's 628KB as shipped (LLM code complexity is high) and it wins on data sets gigabytes in size due to it's ability to create the training data on the fly. No pre-trained LLM comes close when you measure it as above (program size + compressed result).
So it's really not worth that much concern. There's a reason we all say AI is compression and we dismiss comments like the above. They don't pan out. No one's shipping significant dictionaries or pre-trained data. You want to win competitive compression? You'd better have raw code that learns on the fly and is naturally generalizable.
Statistical compression works by learning statistics about the data as compression progresses. This is a(n admittedly very good) implementation of that, but the idea of learning about the data is as old as time (or computers, whichever came last). Pretty much all compressors do this, wither by building a huffmann tree on blocks and shipping the tree allowing the data to be decoded, or by incrementally updating statistics after each token, and using the updated statistics for the next one.
You're just assuming a flawed implementation of a predictor and complain that it's flawed. That doesn't invalidate the general premise that predictors are compressors.
Predictors don't have to give you some corpus-average probability. They can give probabilities adjusted per symbol. The probability could even be exactly 1 for some symbol when the predictor is absolutely certain (and then it costs 0 bits to encode).
If you had a magic oracle predictor that already knew all the data you wanted to send, you'd have perfect compression.
The point you make is observed in compression too, though, once you start optimizing for certain use cases. When LAME encodes sound into an mp3 it doesn't give equal weight to all frequencies. It throws away data outside the range of human hearing to leave more room for what's in-range.
I'd approach this distinction differently. Prediction from compression is valid within the distribution of the compressed data. Which brings it much closer to LLMs in this case (can an LLM talk about a topic it has never seen in training? unlikely if it cannot be derived from other training data)
> Which brings it much closer to LLMs in this case (can an LLM talk about a topic it has never seen in training? unlikely if it cannot be derived from other training data)
This is a great experiment. Do some world building offline about Smergle the Keen finding the Hoopla of Kynashiusous. Then ask ChatGPT what reagents would have been more effective in breaking through the borantiastic door than squeeble stomach juices.
Of course we know the answer - it will either drop into a 'I have only been trained on data before blah blah blah' or 'I cannot talk about current world leaders' or 'I have not been trained on your hackneyed and unpublished romantasy saga, why don't you tell me about it?'
On the other hand, if it is aware of the general concepts of your concept, and has bridging concepts for related topics it has been trained upon, it might try to apply the general concepts to a specific but different scenario.
If your compression algrotihm is deep enough (think LLM), it will capture a lot of abstraction, making it compress well even in future cases that differ from the passed but fit the scheme in some other way.
But not cases that don't fit the scheme, and the future always has some of those. We are continually surprised by what comes next, and have been notoriously bad at predictions that aren't clear trends (until those trends change).
This perspective is a useful source of intuition against the “LLMs can’t have new ideas, they’re just next-token-predictors” style arguments. What if you shift your perspective to thinking of training as optimization over a vast parametrized family of compression algorithms? Well, it suddenly looks a lot more plausible that “new” “ideas” can emerge from that process!
How I see it, is if the human brain does lossy compression/prediction of the natural world that learns from its "training set" (sensory inputs) and we have been able to come up with new ideas, then it seems like AI would be able to as well.
Therefore Einstein couldn't have had the idea of relativity because it wasn't in his training data.
Joking aside, the reality is that a pre-Einstein in 1850 couldn't have had the idea of relativity because Maxwell wasn't in his training data.
And a pre-Maxwell couldn't have had the idea of electromagnetism in 1820 because Faraday wasn't in his training data.
What we call "new ideas" are actually just deduced from the available training data, which includes all the previous ideas and all experimental results. AI can do the exact same thing (I assert). Perhaps not today's AI, but some AI, probably not too far in the future.
Imagine you're curve-fitting a bunch of data points on, say, the orbital motion of planets and asteroids. You get tons and tons of data on these orbital motions, and then put it into a huge black-box optimization algorithm that compresses the heck out of it. It compresess and compresses until it can't find a more compact representation, no matter how much more effort it applies. The output is a function, where you can provide an initial condition, and it gives you the predicted future orbital position at any requested time.
Of course, one thing you get out of this is a great curve-fit for your existing data, which you can interpolate to find the position of any observed planet at any desired time.
But could this function also succeed in predicting the orbital motion of objects that aren't in the dataset? If I spot a new comet, and put it into the compressed function, would I get an accurate prediction of its orbital motion, even though that object wasn't in the training data?
The answer is "it depends, but probably yes". Newton's laws of orbital motion turn out to be simple compared to the size of the training data. So if the black-box compression has done a good job, it might output that function, or a close approximation of it. With a sufficient quantity of sufficiently accurate data, it might even improve on it; random errors can't be compressed, but where the deviations between observations and Newton's law turn out not to be random but rather the influence of an unobserved gravity source, or general relativity, the black-box algorithm will likely capture that as well.
A lot of people seem to think of the training process as curve-fitting data (the "stochastic parrot" model), but I think of it more as "solving an inverse problem to approximate the unknown source that generated the training data". Machine learning has proven to be quite good at solving inverse problems, and this is just a very abstract one of them.
(A forward-problem is something like solving for the electric fields from a set of charged particles; an inverse problem, https://en.wikipedia.org/wiki/Inverse_problem, is one where you have data on the electric fields at various points and want to reconstruct the arrangement of charged particles that produced it. Or more generally, you have sampled data on the output of an unknown process, and want to reconstruct the process that produced the data).
The inverse-problem-solving happens at the ML training step. The language model itself, that comes out of that, is solving the forward-problem: it has a generative-process baked in and now it's generating new data from it. But if the training process has done a good job of compression, it will certainly be able to generate valid new ideas that aren't in the training set, because the inverse model has solved for the underlying features of the real process that generated the training data, and those features can generate additional valid outputs that it wasn't trained on.
> A lot of people seem to think of the training process as curve-fitting data (the "stochastic parrot" model), but I think of it more as "solving an inverse problem to approximate the unknown source that generated the training data"
I just wanted to confirm your underlying point here: training a model isn’t about finding a function that fits the observed data (even though that’s the outcome) but instead finding an approximation of the unknown source that generated the source data in the first place.
In the case of LLMs an approximation not of a human but of the sum of humanity that produced the training copora?
Yes, exactly. And it's the compression that makes this happen. If it was just curve-fitting then the LLM really would act like a stochastic parrot, and it wouldn't generalize; also the model would also need to be much much bigger. As it is, the models are very big but they're still tiny compared to the dataset that they compress, so the compression process forces them to approximate a solution to the inverse problem.
And yeah, that means the base model training process is solving the inverse problem of finding an approximation for the processes that generate all human records and also anything else in the dataset with a compressible pattern to discover (weather data, etc). So it's not surprisng that some kind of world model emerges out of this.
Once MP3s were invented, I had the idea for the Apple IPOD; but obviously I didn't have a giant manufacturing wing, the ability to make small hard drives, or anything else.
I don't think Apple invented the ipod anymore than I invented it; LLMs likely would have also come to the same conclusion about an ipod like device.
Original ideas either dont exist or have a functionally irrelevent definition in comparison with inputing tokens to LLMs to get novel ideas out.
Prediction is literally what's allowed computers to make amazingly creative chess and go moves that a human would never have thought of.
More generally prediction allows you to path find towards a solution to reach some goal that no human might ever see.
If someone states "LLMs can't have new ideas because they only predict" you have to laugh. Prediction literally enables new ideas as you use those predictions to path find to a goal.
There's another element to this that I almost never see discussed. Ideas are not facts. Neither LLMs nor humans can generate new knowledge, as opposed to ideas, by thinking alone. Physical investigation and experimentation is necessary.
The exception being pure mathematics since it exists solely in the realm of ideas. I'm willing to call that knowledge, but it's still a distinction, the old analytic/synthetic dichotomy of Kant.
That conception of knowledge is interesting, but I think using the label 'knowledge' for it is very problematic, it's too far from common definitions. The fact that you have to carve out an exception for mathematics already shows there's a problem. Because if maths, shouldn't thought experiments also produce new knowledge? You're excluding special and general relativity. It seems to me that what the concept actually describes is "information about the world".
> The fact that you have to carve out an exception for mathematics already shows there's a problem.
I don't see how that's an exceptional case. Mathematical theories may not correspond to physical objects, but they still have to be tested for consistency, logical validity etc. As for relativity, one could argue that Einstein's theories were indeed not "knowledge" until their predictions were confirmed (like, say, the gravitational lensing thing). The Michelson–Morley experiment preceded Einstein, but it had to be checked that the theory was consistent with that, and so on and so forth.
Oh, since the topic of semantics compression via LLMs came up, here is some interesting research result that I had found earlier this year that I posted here and failed to explain properly, with a benchmark as well for you to try on your own if you want.
Essentially, copypaste the codeblock in the Markdown into any LLM chat, and it will return with the benchmark results. Very easy benchmark to run.
Essentially, semantic compression refers to reducing the size of a set of data while retaining its full semantic meaning. The useful application of that is of course, with prompt compression to save context. I know a lot of people essentially sends their prompt to another LLM to compress into JSON first before they send it out, and this came out of an experiment to see the best method to accomplish that task, and the idea is that the compressed and uncompressed prompts will return the same result if sent to another LLM.
What that block of Chinese text is essentially a kind of "meta-prompt" that causes the LLM to reflect on itself as well as the method of how to compress information into the highest possible density form, and the reason it is in Chinese is because it is the language with the highest semantic density that I know of. You can ask an LLM to explain what the text in the block means to have an explanation of what everything means and why it works, but overall it tends to greatly increase the efficiency of semantic compression task of turning prose to JSON across the board on pretty much every LLM that I've tested it on.
That's basically the explanation of it, I thought it was a crazy discovery when I found it a couple of months ago, but now I just think it is pretty neat.
This is a lot less surprising when you learn how non-LZ compressors work, that is, by modeling a probability distribution and using those probabilities to encode information in the minimum number of bits required to transmit the data. A less obvious conclusion is that LZ compressors do this to implicitly, the length of each symbol they could emit (literal or match, etc.) can be converted to the probability distribution the LZ compressor induces, since the number of bits to encode the symbol is related to its probability by the information content.
A common design in compressors is to use LZ as a first step, but to then represent the constant data and/or offset-length pairs from LZ using an entropy coder.
Deflate (as used in gzip) uses a Huffman coder. LZMA (as used by xz) uses a predictive range coder. Zstandard can use either Huffman or FSE. Some high-speed compressors like LZ4 skip the entropy coding stage entirely at the expense of compression ratio.
Bzip2 is an interesting aversion of this pattern - it uses the Burrows-Wheeler transform as a first pass instead of LZ. Unfortunately, this is one of the major reasons why it's so slow.
If doesn't correspond cleanly. I can see why you draw the link, because LZ compression will replace words with symbols but BPE is a non-contextual entropy encoding while LZ is contextual and adaptive and that makes it very different. I think BPE actually has more in common with Huffman encoding.
Small world. I just did a podcast on this same topic, but coming at it from a different direction, ie. me and my neighbor trying to beat the hutter prize for compression.
Hutter Prize being where you are paid if you can compress wikipedia small enough. LLMs do very well at that, if, big if, you ignore the cost of initial weights.
A cool Claude Shannon story:
Shannon wanted to measure how much information is actually contained in ordinary
English text. His 1948 theory said such a number must exist, but he had no way to
calculate it, because the patterns in English reach across dozens of letters and no
equation or frequency table captures all of them at once.
So instead of calculating it, he ran an experiment on a person.
He took a passage from a novel that the subject had not read, and covered it with a
card so only the text already guessed was visible. He asked the subject to name
the first letter. If the guess was wrong, he asked again, and kept asking until the
subject named the correct letter. He wrote down how many guesses it had taken,
revealed the letter, and moved the card one position to the right. Then he repeated
the process for the next letter, and the next, through the whole passage.
What this produced was not a sequence of letters but a sequence of numbers — one
number per letter, recording how many guesses that letter required. Most of the
numbers were 1, because someone fluent in English, seeing the preceding text,
usually names the next letter correctly on the first attempt.
Shannon then argued that this sequence of numbers contains exactly as much
information as the original passage.
I stumbled across a connection between LLMs and compression when researching N-dim polytope emergence in neural networks. Toy Models of Superposition (Anthropic, 2022) suggests that gradient descent can independently discover efficient geometric packing arrangements for sparse features. LVQ compression uses regular lattice structures, including some based on 4D lattices.
I found this interesting and wonder whether LLMs have a higher density ceiling, since training and inference don't rely on a fixed lattice and can instead learn their own representational geometry.
If a string produced from random noise gets compressed (because it has invariably some repetitions in it if long enough), is there any prediction? Even getting the probability distributions right doesn't get to any way to reliably to predict the next symbol out of the sample string. Any functions fitted etc. will be incorrect, too.
Cool visuals and breakdown. I wrote something in early 2025 about how LLMs seem to be an emergent behavior of lossy compression, but did not have the knowledge or verbiage at the time to get this detailed. In retrospect my writing seems naive and I'm happy to have found this and the Google paper linked inside. To be a fly on the wall in some of the labs, man.
Another thought that came from the same post is that, insofar as we see LLMs as human-style intelligence, they're more like stream of consciousness devices. Essentially incessant talking and buying enough time until you get to a usable answer. I think I associate some subset of intelligence with what you don't say, which is impossible with the SOC-style outputs, so this is something I think about a fair bit.
What could maybe differentiate current gen models from next gen is the ability to call tools modeled within the layers themselves, not externally. I think as far as I understand it, model trainers expect the model to do this itself in a way we don't understand or control, like a version of the bitter lesson. But I posit we can model many determinate tools as NNs themselves and figure out how to get the internal states of the LLM to make use of them during inference, e.g. calculators, indexes, citations. Just an enthusiast though, so grain of salt and all.
I keep on seeing this claim, especially from popular creators such as 3Blue1Brown. How is this not borderline vacuous?
I'm not a LessWrong^TM rationalist guy, but one really good thought experiment I always keep in the back of my mind from them is Solomonoff induction. AIT people take it as a framework to work with - it's pretty cool, I agree. But I (and some other people, such as certain AI execs at Amazon - according to my interpretation of their public interviews) think it just highlights the trap - given an arbitrarily powerful oracle, you can get compression down pat. Like, if you assume the source is generatable with a turing machine, and you write a function to brute force over all turing machines, then whoa, your compression works. You will necessarily find the optimal compression at some point because your search function is literally searching over all possible turing machines that could've generated the input sequence, anyways (because the input sequence was generated by a turing machine)
These are the kinds of results you can get if you don't have any actual constraints on what the compressor can do.
(Of course, again - this is not the point of solomonoff induction - it's to use this as a base truth, to then layer parsimony on top of that. There are infinite number of turing machines that could match your prefix, parsimony filters, throw some bayesian inference on top of that, and you get Solomonoff induction. They constrain it afterwards. But I think to that intuition as a base whenever people claim new results.).
But I see in casual conversation, people constantly making claims like, "LLM's are so good because they compress a model of the world". What is that model then? Scott Aaronson has made points like this before - your "model" could just be a massive lookup table, so you can't just claim "compression" and win - the compressor must be reasonably small, too.
I don't object to the notion that LLM's have some notion of world models more sophisticated than memorization. That's proven by actual interventional experiments, such as the ones that actual interperability researchers do. But mere compression is vacuously powerful. "Vacuous" not in the sense that "oh, you might be suboptimal and be a little more complex", vacuous as in "the philosophical point you were trying to make is vacuous because you make a vacuously powerful statement".
(I'm not a total fan of intervention either, as an end-all gospel as some people use, but it's far, far better than not having it).
If you want to best predict what state comes next from a space of possibilities, you have to figure out how probable each next state is and pick the most probable one.
If you want to compress something, you have to figure out how probable each next next state is and assign the smallest code to the most probable state.
These two processes are essentially the same, and the resulting structure of a system which regulates either process will be similar, approaching the same structure at high confidence.
‘ I was reading about compression recently when I stumbled upon something crazy: that compressors and LLMs are, at their core, trying to solve the exact same problem.’
This reads as written by someone who just happened to understand what LLMs do, so I totally fail to understand how anything further said can have any real credibility…
As a matter of fact the best compression by Fabrice Bellard’s models have been achieved with NNs long before LLMs.
And also MP3 and MPEG in general are very apparent neural networks, yet not deep as in modern VLMs
If you have a validator for if a JPEG is a valid proof in mathematics, and can generate lots of "plausible" JPEGs, then you can arrive at hew results in mathematics.
On a slightly related topic, static on disk files of LLMs are not incompressible, I have a number of "archived, maybe I'll use it later" Q8 quantized GGUF files that are about 90% of their original file size when run through xz with default options. It's not a ton of disk space savings, but disk space also isn't as cheap as it used to be. BF16 GGUFs will compress a lot.
Compression is just counting. Probability, also, pretty much, just counting. For these reasons I think the role of information theory in describing the process of the development of reasoning and the gain of understanding has been overstated.
Relevant old school compression benchmarks where people have been using different models (incl. transformers) for compression: https://www.mattmahoney.net/dc/text.html
I was thinking about the same topic and the conclusion can be wrong. LLMs are compressors, but compressors are not LLMs. Mixing this can let you believe that you can use a compressor to do the same thing as LLMs, which you cannot.
Specifically I was thinking about a way to inject knowledge into LLMs training by using statistical properties of text in such a way that you don't have to train the LLM to achieve some level of predictions. There are actually some papers that inject n-grams statistics as a part of the neural network weights.
Any compressor actually can be used, trivially, as an autoregressive language model.
Given a context (for LLMs, this would include the entire pretraining dataset, plus the prompt), you compress `context + next_token` for every possible next token. The tokens that co-compress best with the existing context are the 'least surprising' continuations. Choose one of them and iterate.
You can easily generate text with gzip this way. It won't be very good text, because gzip compression is not as sophisticated as a transformer + SGD, but the principle is the same.
The fact that the best compression algorithms are, in the LLM setting, still catastrophically bad at prediction suggests that the overlap between compression and prediction is weak at best.
This shows that prediction algorithms (AI models) are also very good at compression, but compression algorithms (like the ones used in gzip) are not likewise very good at prediction. Which is evidence that compression is necessary but not sufficient for prediction.
>Mixing this can let you believe that you can use a compressor to do the same thing as LLMs, which you cannot.
You can, actually! Any compressor can be losslessly converted into a generator, and vice versa.
Traditional compressors like gzip are of course very simple and can only replicate rough patterns from the input. But they are technically doing the same thing.
I agree that technically they are doing the same thing but in practice LLMs are better compressors than PNGs (learned this while I was researching this topic). That was quite surprising to me.
That sounds like boostrapping the weights involved in early layers, to obviate the need for those layers to learn (optimize) for the distribution in the training set.
Makes me wonder idly,
- is this conceptually akin in some sense to a "universal grammar," and if so
- with a broad enough training set, is there a latent durable universal grammar that might be similarly recovered and injected to the benefit of all training,
- does that grammar go beyond morphological/syntactical/grammatical features, into e.g. semantics and pragmatics
That was my thinking as well mainly to increase the speed of training. But it may turn out that the simple statistics that you can capture like this may account for 1% of the training and are likely to be captured as the first thing during the training.
But actually these techniques are used but they are hidden as speculative decoding with increasing complexity of approximations. For example you can have a part of the network that predicts the next word based on the markov chain, the next approximation is more complex etc.
Compression is not prediction, it is recall. Can we make predictions based on compression? Absolutely. Is memory encoded into physical neurons technically compression? I would argue also yes.
However, going from compression to prediction is a large jump that is unsubstantiated by this article and based on the claim that probabilistic recall is also prediction.
Two perfect counterpoints to this are markets and weather patterns. One cannot predict future events based on past performance or behavior. Change is the only thing that's constant, and chaos/entropy is everywhere we look.
For simple problems like programming, sure predictive recall works amazingly well, but let's not pretend LLMs are actually predicting something. This is exactly why LLMs suck at doing anything novel; they lack imagination and creativity.
Statistical compressors are all about prediction. They can trivially be reversed to make a generator (not as good as an LLM, but only because it would be a much much much smaller model). And an llm can trivialyl be plopped in there to replace the whole thing.
And a counterpoint to your counterpoint: If it was raining 10 seconds ago, it is very likely that it is still raining right now. There... you can make a prediction about weather based on historical data. It's like perlin noise, random but with discernible patterns
I know less about this than every other commenter here, but both weather patterns and market performance do seem predictable based on past behavior when modeled at the right level of abstraction. “Sunshine on Monday” does not imply “rain on Tuesday”, but “cold front moving in Monday night” does. (Likewise “stock up Monday” doesn’t imply “stock down Tuesday” but “CEO arrested for fraud on Monday” does.)
I think this is relevant to the discourse on LLMs/programming because for months, people said “they’re just regurgitating their training set,” but now I think people are seeing (I am seeing) that they do learn more abstract models of the world than that. I don’t really know how, but it’s why they can generalize from other codebases and tools and so on.
> Likewise “stock up Monday” doesn’t imply “stock down Tuesday” but “CEO arrested for fraud on Monday” does
Stock market is even more pathological than weather here. Most statistical tools have baked-in assumptions about some degree of independence between factors under analysis. In weather, things are hard, because everything affects everything but in some random sense. With stock market, it's worse, because some things affect others in an adversarial way. A pattern discovered is not merely drowned in noise, but rather actively canceled out as other people discover and act on it. The way to describe this in formal terms is, I believe, that the stock market is "anti-inductive".
Fortunately, most things we deal with life are much friendlier to predictions than weather or stock prices.
> I think people are seeing
Better late than never, given that this has been empirically demonstrated two years ago by Anthropic. I actually don't understand why people don't make a bigger deal out of it - maybe it's because the AI hype/hate fight is sucking all attention from the room. But the point is: we not only have a thing that can learn concepts we recognize and build abstract world model, but also we have a degree of visibility and control of it we never had before. We can't see, much less suppress or stimulate individual concepts or concept groups in a human brain directly, but we can with LLMs.
Good points. I looked up the definition for prediction and I suppose I'm stretching what I view as prediction.
> A prediction is a statement about what you think will happen in the future, often based on experience or knowledge. It can also be referred to as a forecast or an informed guess
Based on my reading of this definition, compression may inform prediction but it is not itself prediction. The examples cited in the blog post are examples of probabilistic recall based on past events or instances. More context means a higher chance that the recall is more likely to be aligned.
But it's hard for me to accept the leap to compression == prediction because in my mind a prediction is an informed guess about something that hasn't yet come to pass. But thinking more about it, time is a human concept and so who's to say the temporal reference means anything at all here. Maybe probabilistic recall is the same as predictive forecasting if time is an invented concept and essentially means nothing?
Is everything fundamentally deterministic if you know everything in the universe or does free will exist?
IDK to be honest, I'm just more frequently surprised by new things that happen every day than I am at things that stay the same, even if mostly things stay the same. Maybe I just don't notice them and nothing actually ever happens.
Side note: the inevitable consequence of this line of reasoning will eventually become that LLMs given enough power are in fact intelligent and sentient, and I'm worried about how that affects humanity as a whole. Are we about to subjugate the most intelligent thing humanity has ever created, or is it about to subjugate us?
The rabbit hole gets deep quick when making the leap between a fancy recall mechanism and novel prediction, but I agree they're not that different in the end. I just believe it's important to be nuanced or else we'll miss when AGI actually happens (maybe it's already here).
I often wonder how would language fare if we didn't have redundancy in abstractions, why do things get different terms, and if there is such a smaller set that contains everything in a lossless way (english-wise)
This immediately reminded me of the Hutter Prize (http://prize.hutter1.net/) - a contest that has run since 2005(?) based on the premise that compression is closely related to intelligence.
1) am I allowed to scan the entire corpus in advance before I populate the dictionary? Is this a stream, or is their an EOF marker I will know in advance?
2) if 1) then "prediction" isn't the word I'm looking at.
The comparison can be carried on to another even crazier level: Evolution is compression. All the complexity of biology is executed at the highest possible efficiency.
Unrelated to the content: I was really pleased to see that this site defaults to the bare minimum for cookie consent. I reflexively clicked "Reject all" only to see that it was already the default, which threw me off.
It's more or less obvious that the LLM is a lossy-compressed version of the training data; it reproduces sequences of tokens that are the sort of thing that could plausibly occur in the training data, and avoids sequences that are implausible.
Because most of the training data has good grammar, the LLM is strongly trained on grammar; it will rarely predict ungrammatical gibberish. Even if there are grammar mistakes in the data, they are not systematic and so don't reinforce each other.
Something Ngrok is doing is working, because first they manage to get Sam Rose of samwho.com and now this? At this point I care more about their blog than their products
How about dictionary based compression as a counter example? Or the zig zag encoding scheme used in JPEG? I find it difficult to cast some of the things that effectively compress data as prediction.
Dictionary-based compression is based on prediction that recently seen words will be used again. That happens to be generally true for lots of datasets, including human languages (zipf distribution).
JPEG's zig-zag is a primitive for quantization, throwing data away based on rough approximation of human perception and biology. That isn't compression itself. However, the rounded and zeroed-out data is then compressed using a combination of RLE and Huffman, set up to predict the data will have lots of zeroes and few other distinct values (which the earlier step forces to be true). Or if you think about the system as a whole, you could say that JPEG predicts images will be blocky low-frequency patterns of DCT.
I see it that prediction is a form of compression.
Say you have a computer file composed of two parts, the first represents the setup of an experiment, and the second is the data produced by the experiment.
If you have a good theory relating to this type of experiment, then you can predict much of the second part of the file. So you only need to store the first part and possibly some corrections to the least significant bits of some of the parts of the second part of the file.
Thus with good prediction, you can compress this type of file.
Another example is encrypted data. Statistically, encrypted data is indistinguishable from random. Truly random data is impossible to compress losslessly.
But if you had a predictor so smart that it could crack the encryption key, it could start predicting the rest of the encrypted stream, and therefore compress it.
Physics laws are the ultimate form of compression because they are so universal and say so much about so many things in few words, or a formula. This is why Newton's laws were a big achievement. And they enabled predicting the behavior of machines and started the industrial revolution. We are at yet another inflection point.
Recently I wrote a blog post[0] expanding on a similar idea from the angle of ancient Greek philosophy, particularly Parmenides: to think is to compress.
i do agree with that point of view.
I often referred to models as 'modern mp3s' storing a lossfull but lookalike version of information in order to counter that 'AI is totally new and not violating copyright by storing information in a magic fashion' argument.
So a company with an wildly superior compression algorithm stumbling into a society-breaking AI isn't so far-fetched. Jesus...Silicon Valley really was ahead of its time.
I mean, save for the part where the founders recognised the threat it posed to society and acted responsibly rather than unleashing it on the public and sucking down billions in VC money.
Joel from ngrok here, certs are all good! You're likely seeing some corp-level block, which does happen to us on occasion. Hope you'll check it out again from the relative freedom of your home network.
Not really, compression doesn't require a world model, it's mathematically pure. Any AGI system must periodically reset it's prediction since the world is inherently stochastic. When we look at prediction markets they only seem to work in the long term because human language is abstracted away from the real world, again it's mathematically pure. That's why we get bugs in code and disputes with prediction outcomes.
A better title, you can improve your compression if you make an accurate prediction. Much like how a branch predictor can make a CPU do the same work in less time. Or when your symbols are true uncompressable rules of reality (which is probably meaningless both semantically and physically again due to inherent randomness)
The main difference between minimalist and maximalists are how much that set of uncompressable rules gives you. I suspect the search space is too large. What we see in practice is that lossy rules let you cover more ground but eventually you hit a wall and have to move to a lower level of abstraction to make progress. There are 10^360 paths in a go tree, but something like 10^300,000 for molecular chemistry and that's not even all the way up (or down, say 10^3000 for the standard model of physics that's 10^900,000,000 if you want to do chemistry without chemistry abstractions.). Just semiconductor fab is 10^(10^11) so 10^(10^16) with molecular chemistry (think finding an implementation for some sort of desired self assembly outcome). AI can be way way way smarter than humans and there just not be enough energy in the universe to find these needles. So we definitely need abstractions, but those are at odds with predictions and the choice of symbols often introduces abstractions that the designer didn't consider.
This is the thesis behind the "Information Theory, Inference, and Learning Algorithms" course that was taught at Cambridge University.
> Why unify information theory and machine learning? Because they are two sides of the same coin. In the 1960s, a single field, cybernetics, was populated by information theorists, computer scientists, and neuroscientists, all studying common problems. Information theory and machine learning still belong together. Brains are the ultimate compression and communication systems. And the state-of-the-art algorithms for both data compression and error-correcting codes use the same tools as machine learning.
Book (creative commons): https://www.inference.org.uk/mackay/itila/book.html
Lectures: https://m.youtube.com/playlist?list=PLruBu5BI5n4aFpG32iMbdWo...
I had a long ranting comment I deleted. I just don't like this trend of people presenting work in a way that makes you think some combo of 1) they discovered from scratch themselves 2) it's new 3) they didn't try to cite or acknowledge where they learned it/point to good sources 4) they don't really care about trying to teach something deeply, they want shiny stuff that makes them seem deep.
This post references specific parts/calculations, but you'd never know it was not news if you didn't know better.
I don't think this is a fair critique.
The author of the post uses standard terminology like entropy coding and arithmetic coding, and cited a paper "in 2023, Google DeepMind released a paper arguing that language modeling and compression are two views of the same thing" which discusses it further.
This blog post is great. Well explained, and clearly took a lot of effort.
I don't interpret it as them claiming to have to discovered it independently.
Citing 2023 makes it seem like this is newer than it is. Compression, prediction and intelligence have long been known to be deeply connected.
You expect every blog post to find the earliest relevant paper to cite, just so one could look at the year (without reading said paper - which would have made clear that the connection isn’t recent) to assess novelty? I don’t think that’s reasonable.
It’s a blog post. If it was, say, a peer reviewed paper by Hinton or LeCunn that fails to cite Schmidhuber, that would be reasonable criticism in my opinion. (Spoiler: they fail to cite him)
Why would blog posts not be subject to such criticism?
Either the author knew of prior work that argues the same thing and they ignored it, or they didn't know. And if one writes a 1000+ word article premised on this idea, wouldn't one be presumed to know at least in which century the idea originated from?
Arguably these kind of blog posts should be more subject to such criticisms, because the blog posts purport to "teach" the general public about a concept in an authoritative tone (or at least the author seems to pose as knowledgeable in the subject), while for academic papers, everyone who actually reads the paper knows where the ideas came from anyway and it's mainly an issue of attribution (and maybe about fairly distributing the citation count...)
I think you shouldn't criticize the single blog post in vacuum.
If the blog is a personal collection of "today I've learned" then different standards apply then "my blog is my research publishing platform".
Both are valid blogs. The fact that it ended on HN front page didn't mean it was targeted towards this audience
> You expect every blog post to find the earliest relevant paper to cite
This should be expected out of everyone. If you don't respect the reader enough to do this, why should we read your posts?
I think papers should be retracted for not citing prior art, even if you weren't aware of it.
If the blog were about calculus, and stated that an elegant proof of the Fundamental Theorem of Calculus could be found in such and such undergraduate textbook, would you be upset that the citation wasn't to either Newton's or Leibniz' work?
At least since the 1960s: https://en.wikipedia.org/wiki/Solomonoff%27s_theory_of_induc...
The two contributioms that come to mind are the Hutter cash prize for the best compressor of the English Wikipedia and the work on PPM compression-based text classification by the late Prof. Ian Witten's group at Waikato (NZ) [1,2]. The model that best compresses the input string was likely generated by the distribution from which the compression model was 'trained'
[1] https://news.ycombinator.com/item?id=27440093
[2] https://www.computer.org/csdl/proceedings-article/dcc/2000/0...
That compression and prediction are deeply connected is obvious from simply considering how compression algorithms work.
Nothing about the concept of "intelligence" is nearly as well defined as the other two, nor as well agreed upon.
"Understanding" is clearly linked to compression. Taking observations and coming up with a more compact representation that explains them, analogous to coming up with a compact set of axioms that generate facts, or a small Turing machine or short program that generates a list of strings.
Intelligence is a broader concept but definitely involves understanding how a system/envoronment works and making predictions about its unfolding, especially actionable ones that allow you to steer that state towards some goal states.
These are not new.
The post says this is all part of gzip and LLMs, what are you saying? I’ve been using gzip my entire life. I read between the lines “this is common knowledge” throughout the piece. Throwing in some names and dates only makes this super clear story harder to read (and more like studying then the playful exploration this post was intended as).
A short paragraph at the end on the origin of these ideas can be an easy way to dispel the misconception of potential beginner readers that the insights are novel.
I'm of two minds here. The pro is that the "you could have invented this" walkthrough from first principles is more engaging than "and then so and so introduced this term in 1972 and the definition is such and such". This style is a reaction to that boring and dry teaching style and tries to push towards what eg Feynman pointed at in the Brazil critique.
The con is that you don't get to understand and see any of the history of the ideas or even the ballpark when it was discovered, you attribute it to the blog mentally and you don't know what is how new or old and can't reference it properly when talking to others.
Disconnecting idea development from it's historic development is a disservice to the audience that may want to dig deeper.
There's no reason a compelling walkthrough can't throw in historical citations, is there?
Often the most straightforward way to walk through an idea while teaching it is not the same order that the ideas were developed, and might not even use the same set of ideas in building up to it, so it can be tricky to get both the best explanation of the idea and the historical context in at the same time without making things more confusing.
I like to see ideas presented as the evolved. Each solution is developed as a perceived reaction to the shortcomings of the previous. This becomes a contrast and comparison as to why one idea is appropriate for a particular context.
Exactly. The best science exposition combines both. See, The Mechanical Universe and Veritasium.
What specific historical citations would you like to see in this post?
Perhaps you could contact the author with your suggestions?
I'm speaking in general terms and I have not studied much on this subject so I don't have specific suggestions for this post. Please refer to earlier posts in this comment chain for a general idea of what citations would be useful.
They did not come up with the ideas themselves, so they got them somewhere. Follow the source and all the citations show up. It must be a modern thing where online blogging randos pretend they are all geniuses.
Better too assume they are just not aware. Technology is multi-layered cake of development. I have no doubt the only reason I know a lot of details is that I lived their development.
When standing on the shoulders of giants it's hard to tell what is below them.
You're reading this the wrong way I think, citations aren't given because its obviously a pedagogical article about well established stuff. Much like you wouldn't give citations in a blog post explaining calculus.
One could give citations regarding calculus it's pretty interesting. Since it was done twice by both Newton and Liebniz. There must have been cultural developments in the 1660's that demanded calculus be invented.
they discovered from scratch themselves
If you followed the data compression scene in the 80s and early 90s, there were plenty of reinventions of LZ-ish and Huffman-ish algorithms (I also coded my own variant...), and people even tried to patent some of them, so at least for the basics I think it is something that many can discover independently; of course in these times, it's more likely they didn't.
The first sentence says she came across it when reading about compression. I didn't read that as her claiming to have discovered the idea or that it was a new idea, I read it as "today I learned". I think somebody who was unfamiliar with how compression algorithms or language models work would find this an approachable and interesting introduction. Not everybody studied information theory.
Perhaps tangential to your point, but I often write blog posts (although finish and publish far fewer than I start) where I write about something as it has occurred to me, informed by things I've absorbed no doubt, but without specific research. In such cases I explicitly avoid searching out prior work as a) seeing that something is well discussed and explored can take away the motivation to explore (in the same way reading puzzle solutions before starting might), and b) to avoid having green shoots of ideas shaped by the current of existing consensus. Now that doesn't mean I don't come back after doing my own thinking to see what the more well developed literature of people cleverer than me, who've thought far longer than me think; I just don't want to snuff out my own exploration at the start.
As I say most of these I never publish as I'm mainly using writing as a vehicle for thought, but when I do I'm never sure how to flag them. I don't want (imaginary, lets be honest) readers thinking I'm deluded into thinking I've found something new. I want to come up with a tag I can put on them which adds a pithy disclaimer card at the top or something so I feel more comfortable publishing them.
Does everything have to be "news"?
I think you'll find that 95% of all academic presentations are telling stories out of other peoples work.
I'm glad to see someone feels similarly. There is nothing wrong with ignorance, but there's no excuse mistaking learning for invention. Especially from someone bearing the title "Developer Educator"
I don't think it's the case here, but worth noting too that LLM-written blog posts adopt this tone seemingly by default.
Never the least bit of surprise, wonder, doubt, or frustration to get in the way of the steady staccato beat of metaphors, conclusions... and three-item lists.
> I'm glad to see someone feels similarly. There is nothing wrong with ignorance, but there's no excuse mistaking learning for invention. Especially from someone bearing the title "Developer Educator"
>> a Developer Educator at ngrok with a passion for nerd-sniping developers.
Maybe more the latter than former...
And yet to this day, in AI threads, so many people act shocked and surprised if you dare follow the obvious implication and claim that understanding is a form of lossy compression.
Of course it is but again "X is just Y" is often used to mislead. A brain is just neurons! A computer is just transistors! An LLM just predicts the next token! It's just like a parrot! It's just like a blurry jpeg of the internet! Kinda yes, but what do you use this for? It's a bad intuition pump is it leads people to conclude demonstrably false things about capabilities.
Shorter description isn't understanding, let alone of it is lossy.
When you shorten a description in a lossy way, you are deciding a priori that some differences in the object don't matter, and it's not because you understand the object, but because it serves your goal of shortening the description.
Or you actually do understand it.
You can't just assume smaller is better but it often is. And very often it's more information-dense.
You can compress syntax, losslessly even, with zero understanding of its semantics. Zero understanding not only imbued into the compressor/decompressor, but even the designer of the compressor doesn't require understanding the semantics. Actually, even of the syntax.
A compression program can compress a book written in a language that the author of the program doesn't understand, on a topic he knows little about.
Finding common characters and building a list of words is a low level type of understanding. Doing it better does actually start directly representing syntax patterns and that's a less-low level of understanding.
I think "losslessly even" is the wrong way to think about it. Lossless compression often requires less understanding than high quality lossy compression. If you can do a lossy compression that correctly decides what details are unimportant, that's a good sign of understanding.
> I think "losslessly even" is the wrong way to think about it. Lossless compression often requires less understanding than high quality lossy compression. If you can do a lossy compression that correctly decides what details are unimportant, that's a good sign of understanding.
This is the crux and reminds me of things like mp3 that exploit the nature of human hearing being limited to a frequency range.
We model the data. The model, hopefully, captures something real in the data. If it does, then it's fair to say that we understand the data better.
But it's frankly a philosophical question what's real or not. No model is going to capture absolutely everything about the thing it models - at that point, it would be the thing. The best we can hope for is that it captures everything we care about.
And no experiment or metric can tell you if you care about the right things. At best it can tell us if we care about a thing given other things we care about. "No cares in, no cares out".
To make it a little more concrete: you could compress a string from back to front. You could build an LLM to help you do that. If you care about file size, that's almost certainly a bad idea, the forward LLM will be better for that purpose. But are there purposes for which the backward LLM might be better? I think that's not so hard to imagine. Often we wonder about "what came before".
A model will not capture every nuance of what it models, but it might capture every nuance you managed to measure properly.
Including the things you didn't even realize you measured, or even conceptualized at all.
I'm not aware of a better definition of "understanding" that would allow me to tell whether some system "understands" some other system. Do you happen to know one?
See: A. M. Turing (1950) Computing Machinery and Intelligence. Mind 49: 433-460.
I mean, my interpretation is that the question Turing tried to answer is equivalent to "How can we determine whether machines understand humans/human thought?"
This only works when both systems can talk about pretty much arbitrary things, but if you want a more general method for less complex systems, perhaps having one system simulate another system is sufficient. (Which is also another Turing invention)
There are many people who would claim that passing the Turing test is insufficient to show "understanding" (compare for example the Chinese Room thought experiment).
Yes but it's (kind of?) a definition as you asked for.
At this point, I am unaware of a better definition. I know the Chinese Room argument (and I disagree with it), but I'm not aware whether the proponents of that argument have a better definition of understanding other than "well, the Turing Test isn't enough"...
---
PS: Interestingly the issue of compression is highly relevant regarding the Chinese Room argument -- the essential element in the Chinese Room argument is that the information is not compressed...
If you can predict other things by understanding something, is it truly lossy compression?
>understanding is a form of lossy compression..
No. Teaching is compression. Understanding and application is decompression.
I would roughly say that understanding is compression and application is decompression.
You don't memorize the specific details, you memorize the flow of systems. But you get back to the details to put things into practice.
I was speaking in the context of humans. When someone teaches you, the content coming from the teacher is very compressed. One decompress it when they can generalize and apply it. So understanding is compressed, but is not the act of compressing. I mean it is not compressed from a larger data or made by compressing a larger data. The larger data it represents never existed. It is like the definition of a fractal...
> I was speaking in the context of humans.
So was I.
> One decompress it when they can generalize and apply it.
Generalizing is compression. Generalizing and applying are very different things.
> So understanding is compressed, but is not the act of compressing.
Understanding (noun) is compressed. Understanding (verb) is the act of creating understanding (noun) and is compression.
> The larger data it represents never existed.
The larger data is everything you take in about the subject. Experience and instruction.
>creating understanding (noun) and is compression.
What is it compressing from?
examples
Reminds me of Stephen Wolfram "discovering" things in the sense that other people would say "today I learnt".
I find it bothersome that language works this way. You can spend your whole life discovering things that are well known by the rest of the world. But the minute that you mention to a large group that you “discovered” it, suddenly you’re taking credit for discovering it for all of mankind.
Wolfram was my example in my original rant!
I'm glad you're pointing this out because not only are these old insights, but I'm also pretty sure I've seen variations of this blog post years ago on even HN already.
The author acting as if they discovered this independently had me feel the exact same way. Kinda irritating and almost ... disrespectful? Not sure of the right words to describe it tbh
Fabrice Bellard published this in 2023: https://bellard.org/ts_zip/
> The ts_zip utility can compress (and hopefully decompress) text files using a Large Language Model. The compression ratio is much higher than with other compression tools.
It's not only an old idea it's been totally done already.
As much as I want to, I sadly don't think Information Theory makes sense in this setting, and I really wanted to believe this. When Shannon made his theory of information, he was always dealing with informational representations on the abstraction level of bits. At Bell Labs, a lot of the work was on the compression of data for transfer over telephone wires. Entropy coding, later codexes like algorithmic coding, and all compression on this level assumes that you have a bit-based X, and you compress it.
However, in deep neural networks, you are dealing with compression on different levels of abstraction. How do you decide what shared features a peacock and a palm tree have? At what scale should they be represented? How do you deal with invariance under affine transforms? Do you want to open the box of invariance under non-affine transforms?
When you start looking at what it would mean to compress feature representations, you immediately get to the question of data. You realize that Shannon simply was given a form of a very low abstraction data and that information theory came out to handle data at this level, but it's not suited for the data representations of many higher level modalities.
If you read Society of Mind by Marvin Minsky, which has aged well to about 80%, you can get the hint of the kind of abstractions that humans make and what would be needed to represent them, this is not representable in bits, you need to go to higher level shared features, and then you open all of the questions above as well as credit assignment, mutual information approximation, Fischer information between bayesians, etc.
> How do you decide what shared features a peacock and a palm tree have? At what scale should they be represented? How do you deal with invariance under affine transforms? Do you want to open the box of invariance under non-affine transforms?
The whole point is that the representation is learned. When you talk about various levels of abstraction, you're missing that all of these levels are representable with words and the relationships between them. That is verbatim what LLMs are optimized for. Interestingly, when you take an embedding, you do see that some transformations in embedding space actually hold which is quite interesting (e.g. tree + many ~ forest)
1) I am talking about representations beyond language models and language embeddings. If you take for example image, video, audio, 3D-spatial DICOM or combinations like VLMs. If you ask a language model to make an image of a Begonia ferox leaf without training it with images as well, it will not be able to represent this.
2) Language is already a higher-order lossy compressed abstraction made by humans to communicate fast and fill out the left out information with a learned prior. If you train a model on language only, it will not have the opportunity to have a non-compressed representation to make its own abstraction from.
3) If you are LLM-pilled and believe that we will be able to reach arbitrary levels of precise informational representation using language only, and that all abstractions that we may ever want can live on every single embedding layer in an LLM, your argument is fair.
1) If you ask a language model to make an image of a Begonia ferox leaf without training it with images as well
is this not true also of humans?
2) The idea that there is a non-compressed prior is already philosophical no? It seems to just be a rehash of whether an example of a "Chair" is a representation of a perfect Idea of a chair, or whether accidental things put together read as a chair. Its just dualism or not, Plato vs aristotle but for LLMs vs human learning.
If we do happen to just form an abstraction from less examples than an LLM, then its a question of efficiency in the learning algo and the capability to fill the gaps faster/better.
Obvs a world model and multi modal learning would close the gap between LLM latent space and what the human mind encodings might look like. But you can formulate a reality where humans do not have access to a non compressed representation (or even say that doesnt exist) and we are just more efficient at abstraction and recollection
3) There are multiple answers to this. One is that language can just be a prong in the learning modes available. Another is that language is many times redundant and includes supplemental information and nuances that despite being a higher order abstraction has enough additional information to reconstruct entire messages even after the lossy transmission.
Btw I am pretty against LLMs as path forward, but I have been interested in their ability to solve problems with Convergent solutions and their learning model as a lossy compression algorithm where instead of saving the actual data, you save a very good guesser of the next word. So if a text is part of its traning algo then you get almost a 1:1 (like a zip dictionary) but if its not, then he gives you something that plausibly could be in an imaginary Zip with that title.
Check out https://en.wikipedia.org/wiki/Kolmogorov_complexity
Ah Sir David MacKay. I so respect him. Great explainer and speaker. He had built this text entry tool called Dasher [0] - that I'd heard him introduce at Princeton around 2006. It was basically an early language model that predicted which characters are more likely than others, given what you've already types and it would adjust the sizes of the available next characters based on their probabilities.
[0] https://dasher.at/about/
He really was fantastic, and prolific in multiple fields.
He wrote https://www.withouthotair.org/ (creative commons) and was the Chief Scientific Advisor to the UK Department of Energy and Climate Change.
Dedicated to "to those who will not have the benefit of two billion years' accumulated energy reserves".
I also went to a couple of his (fantastic) undergraduate courses, and have a huge amount of respect for him.
That said, I think it's worth mentioning that Climate Change Without the Hot Air has aged pretty badly, and I'd be reluctant to recommend it to people who don't already have the background to understand what's aged well and what hasn't.
The high level approach of making high level numerical estimates makes sense, but it dismisses solar energy in about a page due to assumed high costs. It turns out that even if you're David Mackay you can still be caught out by exponentials :) I notice now that the version you link has some inline updates pointing out how off the assumptions in this section were, but it seems to me that's not enough; you probably need to redo the entire analysis based on what we know today rather than trying to make purely local adjustments.
On the other hand the point at biofuels are even more inefficient, and therefore a dead end even before you consider broader environmental impacts, are well made and something that is sadly not yet widely reflected in policy.
> biofuels are even more inefficient, and therefore a dead end
Only if energy density doesn't matter. But it really does, though. Battery powered electric trucking? Dead end. Battery powered aviation? Dead end. Battery powered shipping? Dead end.
[edit] Maybe there's some sustainable way to convert solar energy into sufficiently energy dense fuels that isn't biological, but so far it seems like seed oils or algae are probably the least bad?
> It was basically an early language model that predicted which characters are more likely than others, given what you've already types and it would adjust the sizes of the available next characters based on their probabilities.
"adjust the sizes of the available next characters": you mean assigning less bits to characters the most likely to come next, and more bits to the least likely characters to come next?
So it's not totally unlike adaptive Huffman coding but instead of updating the tree on already visited characters, here the tree with the weights of the possible following character is predictive?
Adjust the size as in making the GUI element larger: https://en.wikipedia.org/wiki/Dasher_%28software%29 has a picture of completing "Wikiped".
Try it out: `apt install dasher`. https://packages.debian.org/sid/dasher
It is infact a visual implementation of arithmetic coding, and he used it in his lectures to teach arithmetic coding in the lecture following the one on huffman coding. The video lectures posted earlier are highly recommended, best course I took as an undergrad
This is basically a thesis supported by Shannon’s information theory.
Any rigorous CS program should cover this in depth.
It also helps explain to people that LLMs are as likely as bzip to develop "consciousness".
Let alone a sack of wet, self replicating protein! Just endless copying... How could it ever do anything more?
I suspect you're being snarky :-) but this is a really interesting question, and one that has had a lot of research done. I'm not current (I stopped following folks doing this research closely around 2019) but what we 'didn't' know about how brains work was still huge. Signaling levels, enzymes, the connectome, quantum effects, it is a really deep question.
That said, once we do get a working idea of how it works, and can perhaps synthesize a brain artificially with proteins, it will inform us on the next steps for silicon realization of that.
I think the point is that tremendous complexity can arise from relatively simple mechanisms. That is what life is, at many levels. I’m not at all convinced that the current LLM approach will yield something we can broadly call consciousness but saying that it’s a simple concept and therefore won’t support consciousness is a specious argument imo.
I completely agree, tremendous complexity can arise from simple mechanisms. Gleick's Chaos is a really good introduction to that. I was talking about the article though, and the mechanisms currently used for training and inference in LLMs. Those mechanisms are mathematically precise (unlike Chaotic attractors) and as the author points out, achieve the same function as compressors do in a strict bit pattern minimization role. Sometimes tensor math is pretty complex, like the FFT and DCTs on JPEG compression, but with the same inputs you get the same results. And while a JPEG will never decompress to a different image than the one that was compressed in the first place, LLMs do not 'infer' token streams that haven't been trained in their training process. The big difference here is that if you imagine a JPEG compressor that compresses 100 different images into one 'chunk', you can see how to provoke it to produce any one of the images it previously compressed. And with a bit of creativity you can have it express different images in different parts of the resulting composite. FWIW I looked at patenting something like this for digital cameras to give them more "shots" space for a given amount of SD storage.[1]
Given the way that models work in 'inference' mode (vs 'training' mode) you can't forward bias the result into the correct result when there are multiple forward results that have identical weights. It's the root cause of hallucinations, and you've lost information in the training phase that you can't then use to discriminate between the 'right' answer and an equally valid 'wrong' answer.
[1] FWIW I could never recover enough state to insure that the image it regenerated was all of the same image you took. So you might get the street but one of the houses might be a house that was in a different picture you took. That kind of bug. Mostly arising out of the same kind of problem you have with using hashes to find documents, when you get a hash collision two documents have the same hash, so you don't know which one to return.
I do not understand this intuition that "true consciousness has to be random". The things that make me me are highly deterministic!
> LLMs do not 'infer' token streams that haven't been trained in their training process
While we're at it, this is simply untrue (in-context learning) unless you generalize "token streams" so radically that it could be readily analogized to humans as well.
This self replicating proteins can do a lot of stuff that a program running inside a computer cannot do. Like producing secretions, generate electricity, generate light, generate force etc etc....
Better prediction being better compression is Shannon 1948, and the link to machine learning is MacKay 2003 at Cambridge.
Compression and error correction also go hand in hand: in compressed data every bit carries more information and therefore errors are more detrimental. This is one of the results that Shannon phrased exactly in terms of entropy.
My PhD supervisor had a beautiful example. Take an English message and ‘compress it’ by removing vowels. It’s still intelligible because English has redundancy:
rrrs cn mk mssgs nrdbl
Similarly, an uncompressed message with errors (swapped characters) is also intelligible because of the redundancy of English:
erwurs lan nake wesaagis unfeatable
But now we do both: we compress the message AND add errors. The result should be much harder (if not impossible) to read:
rwrs ln nk wssgs nftbl
Grant Sanderson has an excellent video on the same topic [0]. It's part of a series that is ongoing.
[0] Compression is Intelligence Part 1 - https://youtu.be/l6DKRf-fAAM?si=yyLWq8x4sSRkWd98
I wonder if the author of the article knew about the series, or do they both just independently came across this topic to talk about it.
Anybody working in the field will be very familiar with these concepts.
…for years. Because it is so apparent if you actually try to look at the problem and what is being solved by it.
The extraction of features from a corpus, the features significant to certain solution, is always and since day zero - compression. As this is the definition of compression - efficient and potentially lossless feature extraction.
common theory. see https://prize.hutter1.net/
And the Hutter Prize for AI which measures how good AI is by measuring how well it compresses data is over 20 years old now just to really drive the point home.
It's basic information theory, which has been around since the end of WWII. It's a common topic today because some of its subtle insights are becoming increasingly relevant in our current era of AI, as we learn to understand these black boxes.
it was vaguely in my understanding of information & intelligence with compression; it was also brought up in several of the initial trials against AI companies where they discussed how the AI is akin to compression.
So they're both sourcing a bit broader zeitgeist.
The page source appears to contain all the actual text within <p> tags, but structured in a completely illogical way. With JavaScript disabled, there are a bunch of shaded bars where the text should appear, which look like placeholders for something that hasn't loaded yet even though it was there from the beginning. The <p> tags don't even seem to show up in the DOM. (I didn't check closely, but maybe they're embedded in an inline script.)
This is actively user-hostile. The site is going out of its way to interfere with the most basic possible function of HTML, i.e., the presentation of minimally marked-up plain text. The needless complexity is especially ironic in the context of an article about compression.
I think every website should be support noscript with minimum requirements.
The article is using probability where it really means proportion and prediction where it means evaluation. The mathematical equivalency is both much less surprising and less revealing once reframed.
If we consider the first example with the arithmetic code, the initial presupposition that only the characters A, B, and C appear in the string already reduces the entropy from 56 ascii bits to 14 bits (A vs Not A and B vs Not B for each character). If you further consider that you only need to distinguish B vs Not B if it's not A, then you can just represent As with a single zero bit and only represent the non-As as two bits (the first of which will necessarily always be a 1 bit). This gets you to 10 bits without even having the proportions of the string. Of course this would be a poor convention if there were say only a single A; in that worst case scenario you would need 13 bits, but simply knowing which character appears the most, without knowing by how much, 11 bits is the worst case scenario for a length 7 string with 3 potential characters. The last bit can be made implicit if you further choose the second conditional appropriately - i.e. if instead of B vs Not B we chose C vs Not C, our last bit would be zero and could simply be dropped meaning both 10 and a single 1 bit encode C - allowing you to encode the example string in just 9 bits and an arbitrary string of that length in 10, again regardless of proportions. That improvement over the arithmetic encoding result in the example is just a case of us cramming a little extra information into the encoding algorithm.
Arithmetic encoding is more clean and more easily extensible, it makes more sense to use than this custom encoding of 7 trits to binary but the point is the "probability" the article mentions is a superficial quality of life feature, not the secret sauce that is the actual key to compression.
Prediction is compression, but I am not sure if it is true the other way around.
It's obvious that an accurate predictor enables encoding only the data that the predictor gets wrong.
But a compressor can encode patterns that defy prediction by looking at the data as a whole. It doesn't have to look at everything in sequence as it arrives.
Applying transformations prior to entropy encoding often isn't just 'rearranging into an easier to compresss format' the transformation can be doing the job of peeking into the future. That makes the encoding a whole lot easier, but it is much harder to call it prediction.
Indeed. If you're going for a catchy generalization, at least write it correctly. Most compression is history, and only extrapolates under the assumption that "nothing changes".
there are dictionary compressors (decent compression, most common, fast), and statistical compressors (better compression, slower). Statistical compressors are much closer to LLMs in that an llm is learning statistics about the data too.
And yes, compression is history, that's what statistics are all about. Statistics can only measure the past to make a prediction about the future. And LLMs work in the same way. The context is the history, and given that history, it predicts the next token. An LLM can, almost trivially, be dropped into something like the PPM statistical compressor (it's just replacing one implementation of a markov chain with another).
Anything can only represent past measurements. Statistics is not an exception. But they don't make a prediction about the future. That comes from a model you have, and it often is implicit: "the linear trend from the last 12 months will hold in the next month" or whatever. So compression isn't by definition prediction. The other way around doesn't have to hold either, but in the case of LLMs it does.
There is Compression done by Prediction by partial matching [0]
There is the Kolmogorov Complexity [1], Normalized Information Distance [2] and Normalized compression distance [3] that correlates those.
Finally, there's the Pre-Big Bang Informational Compression and the Delayed Release of Antimatter [4]
All big {rabbit/black} holes to lose some time, if you have any.
[0] https://en.wikipedia.org/wiki/Prediction_by_partial_matching
[1] https://en.wikipedia.org/wiki/Kolmogorov_complexity
[2] https://homepages.cwi.nl/~paulv/papers/chapter08.pdf
[3] https://en.wikipedia.org/wiki/Normalized_compression_distanc...
[4] https://philarchive.org/rec/GREPBI
I'll add Minimum Description Length to the mix. Under certain definitions and conditions, it equals the Bayesian Information Criterion plus an extra term, which I consider a very interesting result in this "two faces of the same coin" perspective.
This is exactly why I think they are one and the same. It's relatively trivial to just plop a (lossy) machine learned markov chain instead of one learned (perfectly) from the data into PPM. With zero changes to the rest of the algorithm.
Nope; there is a bit more nuance and the distinction is important.
Compression is functionally equivalent to prediction when the data distribution is exactly representative of all future problems. The story changes drastically if you want generalization -- because the test distribution could be arbitrarily different, even if it had the same support! Eg: you observe a rare edge case in your training data and (lossy) compression could simply ignore it. But if you wanted generalization in that particular part of the space -- either because an adversary was testing you, or for design freedom where you choose to build in that specific corner -- then you don't just want data compression, but good prediction performance on a test distribution which peaks in that corner.
Assuming that the training data distribution is exactly the distribution you will ever care for is implicitly doing a lot of the heavy lifting in the claim that compression = prediction, and I'm peeved at how much this statement is unthinkingly repeated like a manifesto.
There is nothing natural about the training data distribution, especially if the data generation process is exploratory while the downstream usage will be exploitative.
How does that invalidate the "compression is prediction"? If the future data is different and you failed to generalize, then the failure to predict means you got worse at compressing and have to spend more bits storing the new information. Conversely, if the future data is the same as that you've seen previously, you could predict it very well, and compress better as a result.
A maximally efficient compressor for the existing data distribution is not in general (and often will not be) maximally efficient for future data. The former may only be enabled by convenient local optima of the input distribution that a compressor accounting for the latter could not take advantage of.
For instance, consider the distribution of strings drawn from the language '0+'. Now consider the same for the language '[01]+'. A compressor looking at only the strings of the first language within those of the second can do a much better job if it does not have to account for future data.
This also relates distantly to the idea of overfitting in machine learning.
It might not be optimal, but it's not wrong to call it the best available guess. That's basically assuming Occam's razor / Solomonoff induction. Hutter published a bunch of work about what it means to have an "optimal" compressor and famously spent the past couple decades running a compression contest on the idea that it'd lead to insights in AI.
There is a distinction between a compressor for a fixed dataset and one for an unknown population from which we have a sample. The optimal compressor for the sample may be the single best guess for the population, but that's not what Solomonoff induction does. It begins with a prior that allows all possible programs, and it never assigns all probability to the single optimal compressor, so it has no problem with the all-zeroes example.
But the Hutter prize (of which I'm a big fan) is for ever-more-optimal compressors, and in fact many of the solutions don't generalise to other input data without stripping out various tricks.
I think it's better to think of the hutter prize as a challenge to see how close people can get to efficiently approximating AIXI without the exponential cost of AIXItl. The fact that winners don't generalize well is just that they're not great approximations, because the behavior is different in the limit.
The winner generalize really well. They're just incredibly slow.
Most of the neural compressors are trying to squeeze every last bit out by sacrificing runtime. You can speed them up a fair bit and they're pretty good. fast-cmix with a small priming corpus will probably be about twice as compact as something like lzma or brotli.
In a world where we see 0+ strings more often the first compressor would be better
"Compression can be prediction" would be more accurate.
I’m not exactly sure what distinction you are trying to make, but a compressor that generalizes from currently observed day to potential future data _is_ a better compressor, as it will have better predictions as it observes more of the source data distribution. Maybe you are making a point about whether compressors can “learn” after compressing the input data, but this also isn’t a fundamental property of compressors. Also compressors do have to deal with non-stationary data sources, so it also isn’t correct to say a compressor only models stationary distributions.
But your definition of compression as prediction is tautological. How would you operationalize that into a measurable metric of compression?
The crucial idea is that once you do that and start optimizing for compression, at some point more compression on the training dataset can lead to worse predictions on the futures you care for. Which is very incompatible with the poster statement.
> because the test distribution could be arbitrarily different,
In that case all bets are off for all machine learning algorithms anyway. If the changes are not arbitrarily different then some of the lost performance can be recovered.
You talk about generalization a lot, but perhaps you are not aware that the technical term is defined under the assumption that test and train distribution are the same.
The record setting file compressors are all very generalizable. This comes about because you score compression by adding the program size (including any training data or dictionary) to the compressed data. If they didn't score it this way you could just ship a dictionary of the test data and your program just prints that. So of course they combine it and effectively measure the Kolmorogov complexity.
This means that training data isn't that helpful to top of the line compressors. eg. Fabrice Bellards nncp that's currently the record holding compressor for enwik9 is an LLM that learns on the fly. It's 628KB as shipped (LLM code complexity is high) and it wins on data sets gigabytes in size due to it's ability to create the training data on the fly. No pre-trained LLM comes close when you measure it as above (program size + compressed result).
So it's really not worth that much concern. There's a reason we all say AI is compression and we dismiss comments like the above. They don't pan out. No one's shipping significant dictionaries or pre-trained data. You want to win competitive compression? You'd better have raw code that learns on the fly and is naturally generalizable.
Statistical compression works by learning statistics about the data as compression progresses. This is a(n admittedly very good) implementation of that, but the idea of learning about the data is as old as time (or computers, whichever came last). Pretty much all compressors do this, wither by building a huffmann tree on blocks and shipping the tree allowing the data to be decoded, or by incrementally updating statistics after each token, and using the updated statistics for the next one.
You're just assuming a flawed implementation of a predictor and complain that it's flawed. That doesn't invalidate the general premise that predictors are compressors.
Predictors don't have to give you some corpus-average probability. They can give probabilities adjusted per symbol. The probability could even be exactly 1 for some symbol when the predictor is absolutely certain (and then it costs 0 bits to encode).
If you had a magic oracle predictor that already knew all the data you wanted to send, you'd have perfect compression.
Compression is still prediction, the predictions just might not be good.
> Compression is functionally equivalent to prediction when the data distribution is exactly representative of all future problems.
At some level, nothing is representative of all future "problems".
I think as far as corner cases is concerned, people can optimize for what they think matters. Larger models, higher sensitivities, better filtering.
Beyond this, I don't know what you're peeved by. There's no manifesto, there's no claiming that any particular method to prediction is perfect yet.
The point you make is observed in compression too, though, once you start optimizing for certain use cases. When LAME encodes sound into an mp3 it doesn't give equal weight to all frequencies. It throws away data outside the range of human hearing to leave more room for what's in-range.
I'd approach this distinction differently. Prediction from compression is valid within the distribution of the compressed data. Which brings it much closer to LLMs in this case (can an LLM talk about a topic it has never seen in training? unlikely if it cannot be derived from other training data)
> Which brings it much closer to LLMs in this case (can an LLM talk about a topic it has never seen in training? unlikely if it cannot be derived from other training data)
This is a great experiment. Do some world building offline about Smergle the Keen finding the Hoopla of Kynashiusous. Then ask ChatGPT what reagents would have been more effective in breaking through the borantiastic door than squeeble stomach juices.
Of course we know the answer - it will either drop into a 'I have only been trained on data before blah blah blah' or 'I cannot talk about current world leaders' or 'I have not been trained on your hackneyed and unpublished romantasy saga, why don't you tell me about it?'
On the other hand, if it is aware of the general concepts of your concept, and has bridging concepts for related topics it has been trained upon, it might try to apply the general concepts to a specific but different scenario.
That’s interesting.
Also sparked the thought that the assumption only holds if the future looks like the present.
If your compression algrotihm is deep enough (think LLM), it will capture a lot of abstraction, making it compress well even in future cases that differ from the passed but fit the scheme in some other way.
But not cases that don't fit the scheme, and the future always has some of those. We are continually surprised by what comes next, and have been notoriously bad at predictions that aren't clear trends (until those trends change).
I think Hutter would vehemently disagree with you on that one ;)
100% agreed.
This perspective is a useful source of intuition against the “LLMs can’t have new ideas, they’re just next-token-predictors” style arguments. What if you shift your perspective to thinking of training as optimization over a vast parametrized family of compression algorithms? Well, it suddenly looks a lot more plausible that “new” “ideas” can emerge from that process!
> it suddenly looks a lot more plausible that “new” “ideas” can emerge from that process
This is not intuitive to me. It seems like a "new idea" is something that (almost by definition) isn't in the training set. Can you elaborate a bit?
Edit: but perhaps a good model could arise from training, which would be a good idea in the sense that parsimonious ideas are good scientific ideas.
How I see it, is if the human brain does lossy compression/prediction of the natural world that learns from its "training set" (sensory inputs) and we have been able to come up with new ideas, then it seems like AI would be able to as well.
Therefore Einstein couldn't have had the idea of relativity because it wasn't in his training data.
Joking aside, the reality is that a pre-Einstein in 1850 couldn't have had the idea of relativity because Maxwell wasn't in his training data.
And a pre-Maxwell couldn't have had the idea of electromagnetism in 1820 because Faraday wasn't in his training data.
What we call "new ideas" are actually just deduced from the available training data, which includes all the previous ideas and all experimental results. AI can do the exact same thing (I assert). Perhaps not today's AI, but some AI, probably not too far in the future.
Wow, I didn’t even notice how badly I underexplained that! Yeah, thanks, your edit is what I had in mind. jbay808 explains it well, too.
An idea is a construction on existing concepts, a newer idea links more previously unlinked concepts.
Imagine you're curve-fitting a bunch of data points on, say, the orbital motion of planets and asteroids. You get tons and tons of data on these orbital motions, and then put it into a huge black-box optimization algorithm that compresses the heck out of it. It compresess and compresses until it can't find a more compact representation, no matter how much more effort it applies. The output is a function, where you can provide an initial condition, and it gives you the predicted future orbital position at any requested time.
Of course, one thing you get out of this is a great curve-fit for your existing data, which you can interpolate to find the position of any observed planet at any desired time.
But could this function also succeed in predicting the orbital motion of objects that aren't in the dataset? If I spot a new comet, and put it into the compressed function, would I get an accurate prediction of its orbital motion, even though that object wasn't in the training data?
The answer is "it depends, but probably yes". Newton's laws of orbital motion turn out to be simple compared to the size of the training data. So if the black-box compression has done a good job, it might output that function, or a close approximation of it. With a sufficient quantity of sufficiently accurate data, it might even improve on it; random errors can't be compressed, but where the deviations between observations and Newton's law turn out not to be random but rather the influence of an unobserved gravity source, or general relativity, the black-box algorithm will likely capture that as well.
A lot of people seem to think of the training process as curve-fitting data (the "stochastic parrot" model), but I think of it more as "solving an inverse problem to approximate the unknown source that generated the training data". Machine learning has proven to be quite good at solving inverse problems, and this is just a very abstract one of them.
(A forward-problem is something like solving for the electric fields from a set of charged particles; an inverse problem, https://en.wikipedia.org/wiki/Inverse_problem, is one where you have data on the electric fields at various points and want to reconstruct the arrangement of charged particles that produced it. Or more generally, you have sampled data on the output of an unknown process, and want to reconstruct the process that produced the data).
The inverse-problem-solving happens at the ML training step. The language model itself, that comes out of that, is solving the forward-problem: it has a generative-process baked in and now it's generating new data from it. But if the training process has done a good job of compression, it will certainly be able to generate valid new ideas that aren't in the training set, because the inverse model has solved for the underlying features of the real process that generated the training data, and those features can generate additional valid outputs that it wasn't trained on.
> A lot of people seem to think of the training process as curve-fitting data (the "stochastic parrot" model), but I think of it more as "solving an inverse problem to approximate the unknown source that generated the training data"
I just wanted to confirm your underlying point here: training a model isn’t about finding a function that fits the observed data (even though that’s the outcome) but instead finding an approximation of the unknown source that generated the source data in the first place.
In the case of LLMs an approximation not of a human but of the sum of humanity that produced the training copora?
Yes, exactly. And it's the compression that makes this happen. If it was just curve-fitting then the LLM really would act like a stochastic parrot, and it wouldn't generalize; also the model would also need to be much much bigger. As it is, the models are very big but they're still tiny compared to the dataset that they compress, so the compression process forces them to approximate a solution to the inverse problem.
And yeah, that means the base model training process is solving the inverse problem of finding an approximation for the processes that generate all human records and also anything else in the dataset with a compressible pattern to discover (weather data, etc). So it's not surprisng that some kind of world model emerges out of this.
Once MP3s were invented, I had the idea for the Apple IPOD; but obviously I didn't have a giant manufacturing wing, the ability to make small hard drives, or anything else.
I don't think Apple invented the ipod anymore than I invented it; LLMs likely would have also come to the same conclusion about an ipod like device.
Original ideas either dont exist or have a functionally irrelevent definition in comparison with inputing tokens to LLMs to get novel ideas out.
Prediction is literally what's allowed computers to make amazingly creative chess and go moves that a human would never have thought of.
More generally prediction allows you to path find towards a solution to reach some goal that no human might ever see.
If someone states "LLMs can't have new ideas because they only predict" you have to laugh. Prediction literally enables new ideas as you use those predictions to path find to a goal.
Being able to read out positions (minimax, alpha-beta pruning etc.) is not "prediction".
Incidentally, the relationship is bidirectional. You can try it out just for fun. zstd is a pretty crappy language model :)
There's another element to this that I almost never see discussed. Ideas are not facts. Neither LLMs nor humans can generate new knowledge, as opposed to ideas, by thinking alone. Physical investigation and experimentation is necessary.
The exception being pure mathematics since it exists solely in the realm of ideas. I'm willing to call that knowledge, but it's still a distinction, the old analytic/synthetic dichotomy of Kant.
That conception of knowledge is interesting, but I think using the label 'knowledge' for it is very problematic, it's too far from common definitions. The fact that you have to carve out an exception for mathematics already shows there's a problem. Because if maths, shouldn't thought experiments also produce new knowledge? You're excluding special and general relativity. It seems to me that what the concept actually describes is "information about the world".
> The fact that you have to carve out an exception for mathematics already shows there's a problem.
I don't see how that's an exceptional case. Mathematical theories may not correspond to physical objects, but they still have to be tested for consistency, logical validity etc. As for relativity, one could argue that Einstein's theories were indeed not "knowledge" until their predictions were confirmed (like, say, the gravitational lensing thing). The Michelson–Morley experiment preceded Einstein, but it had to be checked that the theory was consistent with that, and so on and so forth.
Oh, since the topic of semantics compression via LLMs came up, here is some interesting research result that I had found earlier this year that I posted here and failed to explain properly, with a benchmark as well for you to try on your own if you want.
https://github.com/yuechen-li-dev/GenerativeCompressionProto...
Essentially, copypaste the codeblock in the Markdown into any LLM chat, and it will return with the benchmark results. Very easy benchmark to run.
Essentially, semantic compression refers to reducing the size of a set of data while retaining its full semantic meaning. The useful application of that is of course, with prompt compression to save context. I know a lot of people essentially sends their prompt to another LLM to compress into JSON first before they send it out, and this came out of an experiment to see the best method to accomplish that task, and the idea is that the compressed and uncompressed prompts will return the same result if sent to another LLM.
What that block of Chinese text is essentially a kind of "meta-prompt" that causes the LLM to reflect on itself as well as the method of how to compress information into the highest possible density form, and the reason it is in Chinese is because it is the language with the highest semantic density that I know of. You can ask an LLM to explain what the text in the block means to have an explanation of what everything means and why it works, but overall it tends to greatly increase the efficiency of semantic compression task of turning prose to JSON across the board on pretty much every LLM that I've tested it on.
That's basically the explanation of it, I thought it was a crazy discovery when I found it a couple of months ago, but now I just think it is pretty neat.
This is a lot less surprising when you learn how non-LZ compressors work, that is, by modeling a probability distribution and using those probabilities to encode information in the minimum number of bits required to transmit the data. A less obvious conclusion is that LZ compressors do this to implicitly, the length of each symbol they could emit (literal or match, etc.) can be converted to the probability distribution the LZ compressor induces, since the number of bits to encode the symbol is related to its probability by the information content.
A common design in compressors is to use LZ as a first step, but to then represent the constant data and/or offset-length pairs from LZ using an entropy coder.
Deflate (as used in gzip) uses a Huffman coder. LZMA (as used by xz) uses a predictive range coder. Zstandard can use either Huffman or FSE. Some high-speed compressors like LZ4 skip the entropy coding stage entirely at the expense of compression ratio.
Bzip2 is an interesting aversion of this pattern - it uses the Burrows-Wheeler transform as a first pass instead of LZ. Unfortunately, this is one of the major reasons why it's so slow.
The first LZ-step pretty much directly maps to BPE tokenization in LLMs.
If doesn't correspond cleanly. I can see why you draw the link, because LZ compression will replace words with symbols but BPE is a non-contextual entropy encoding while LZ is contextual and adaptive and that makes it very different. I think BPE actually has more in common with Huffman encoding.
Yep, for example for predicting future access patterns in a VM subsystem.
Practical Prefetching via Data Compression; Vitter, Krishnam, Curewitz. 1993
The page addresses ('names') were the characters and the built-up LZ dictionary used to predict which "characters" → pages would come next.
https://www.ittc.ku.edu/~jsv/Papers/CKV93.practical-prefetch...
Optimal Prediction for Prefecting in the Worst Case; Vitter, Krishnan
https://dl.acm.org/doi/pdf/10.5555/314464.314575
Apparently the same trick was later rediscovered for web-pages.
Small world. I just did a podcast on this same topic, but coming at it from a different direction, ie. me and my neighbor trying to beat the hutter prize for compression.
Hutter Prize being where you are paid if you can compress wikipedia small enough. LLMs do very well at that, if, big if, you ignore the cost of initial weights.
A cool Claude Shannon story:
Sounds a lot like next token prediction to me.https://corecursive.com/the-hutter-prize/
http://prize.hutter1.net/
https://github.com/hkust-nlp/llm-compression-intelligence
https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf
I think a better headline would be something like: Compression is Abstraction and Decompression is Extrapolation.
Many of the debates in the comments seem to come down to whether people believe prediction and extrapolation are synonymous.
I stumbled across a connection between LLMs and compression when researching N-dim polytope emergence in neural networks. Toy Models of Superposition (Anthropic, 2022) suggests that gradient descent can independently discover efficient geometric packing arrangements for sparse features. LVQ compression uses regular lattice structures, including some based on 4D lattices.
I found this interesting and wonder whether LLMs have a higher density ceiling, since training and inference don't rely on a fixed lattice and can instead learn their own representational geometry.
If a string produced from random noise gets compressed (because it has invariably some repetitions in it if long enough), is there any prediction? Even getting the probability distributions right doesn't get to any way to reliably to predict the next symbol out of the sample string. Any functions fitted etc. will be incorrect, too.
Cool visuals and breakdown. I wrote something in early 2025 about how LLMs seem to be an emergent behavior of lossy compression, but did not have the knowledge or verbiage at the time to get this detailed. In retrospect my writing seems naive and I'm happy to have found this and the Google paper linked inside. To be a fly on the wall in some of the labs, man.
Another thought that came from the same post is that, insofar as we see LLMs as human-style intelligence, they're more like stream of consciousness devices. Essentially incessant talking and buying enough time until you get to a usable answer. I think I associate some subset of intelligence with what you don't say, which is impossible with the SOC-style outputs, so this is something I think about a fair bit.
What could maybe differentiate current gen models from next gen is the ability to call tools modeled within the layers themselves, not externally. I think as far as I understand it, model trainers expect the model to do this itself in a way we don't understand or control, like a version of the bitter lesson. But I posit we can model many determinate tools as NNs themselves and figure out how to get the internal states of the LLM to make use of them during inference, e.g. calculators, indexes, citations. Just an enthusiast though, so grain of salt and all.
I keep on seeing this claim, especially from popular creators such as 3Blue1Brown. How is this not borderline vacuous?
I'm not a LessWrong^TM rationalist guy, but one really good thought experiment I always keep in the back of my mind from them is Solomonoff induction. AIT people take it as a framework to work with - it's pretty cool, I agree. But I (and some other people, such as certain AI execs at Amazon - according to my interpretation of their public interviews) think it just highlights the trap - given an arbitrarily powerful oracle, you can get compression down pat. Like, if you assume the source is generatable with a turing machine, and you write a function to brute force over all turing machines, then whoa, your compression works. You will necessarily find the optimal compression at some point because your search function is literally searching over all possible turing machines that could've generated the input sequence, anyways (because the input sequence was generated by a turing machine)
These are the kinds of results you can get if you don't have any actual constraints on what the compressor can do.
(Of course, again - this is not the point of solomonoff induction - it's to use this as a base truth, to then layer parsimony on top of that. There are infinite number of turing machines that could match your prefix, parsimony filters, throw some bayesian inference on top of that, and you get Solomonoff induction. They constrain it afterwards. But I think to that intuition as a base whenever people claim new results.).
But I see in casual conversation, people constantly making claims like, "LLM's are so good because they compress a model of the world". What is that model then? Scott Aaronson has made points like this before - your "model" could just be a massive lookup table, so you can't just claim "compression" and win - the compressor must be reasonably small, too.
I don't object to the notion that LLM's have some notion of world models more sophisticated than memorization. That's proven by actual interventional experiments, such as the ones that actual interperability researchers do. But mere compression is vacuously powerful. "Vacuous" not in the sense that "oh, you might be suboptimal and be a little more complex", vacuous as in "the philosophical point you were trying to make is vacuous because you make a vacuously powerful statement".
(I'm not a total fan of intervention either, as an end-all gospel as some people use, but it's far, far better than not having it).
The key principle is simple.
If you want to best predict what state comes next from a space of possibilities, you have to figure out how probable each next state is and pick the most probable one.
If you want to compress something, you have to figure out how probable each next next state is and assign the smallest code to the most probable state.
These two processes are essentially the same, and the resulting structure of a system which regulates either process will be similar, approaching the same structure at high confidence.
‘ I was reading about compression recently when I stumbled upon something crazy: that compressors and LLMs are, at their core, trying to solve the exact same problem.’
This reads as written by someone who just happened to understand what LLMs do, so I totally fail to understand how anything further said can have any real credibility…
As a matter of fact the best compression by Fabrice Bellard’s models have been achieved with NNs long before LLMs.
And also MP3 and MPEG in general are very apparent neural networks, yet not deep as in modern VLMs
Schmidhuber did it first:
Driven by Compression Progress: A Simple Principle Explains Essential Aspects of Subjective Beauty, Novelty, Surprise, Interestingness, Attention, Curiosity, Creativity, Art, Science, Music, Jokes
https://arxiv.org/abs/0812.4360
Dessalles is good too
https://simplicitytheory.telecom-paris.fr/
page created 8 days after Schmidhuber's paper.
Ted Chiang made a similar point in his article "ChatGPT is a blurry JPEG of the web" a few years ago:
https://www.newyorker.com/tech/annals-of-technology/chatgpt-...
It's a great line, but that's obviously not all it is. You don't get new results in e.g. mathematics by looking carefully at the pixels of a JPEG.
If you have a validator for if a JPEG is a valid proof in mathematics, and can generate lots of "plausible" JPEGs, then you can arrive at hew results in mathematics.
A few years ago it might have been. You have to judge assertions in the context of when they were made. Obviously some things have changed.
It's a good line! It's even sort of useful. It's just not the whole story.
On a slightly related topic, static on disk files of LLMs are not incompressible, I have a number of "archived, maybe I'll use it later" Q8 quantized GGUF files that are about 90% of their original file size when run through xz with default options. It's not a ton of disk space savings, but disk space also isn't as cheap as it used to be. BF16 GGUFs will compress a lot.
Compression is just counting. Probability, also, pretty much, just counting. For these reasons I think the role of information theory in describing the process of the development of reasoning and the gain of understanding has been overstated.
Relevant old school compression benchmarks where people have been using different models (incl. transformers) for compression: https://www.mattmahoney.net/dc/text.html
Also interesting the top entry is from fabrice bellard: https://bellard.org/nncp/nncp.pdf
I was thinking about the same topic and the conclusion can be wrong. LLMs are compressors, but compressors are not LLMs. Mixing this can let you believe that you can use a compressor to do the same thing as LLMs, which you cannot.
Specifically I was thinking about a way to inject knowledge into LLMs training by using statistical properties of text in such a way that you don't have to train the LLM to achieve some level of predictions. There are actually some papers that inject n-grams statistics as a part of the neural network weights.
Any compressor actually can be used, trivially, as an autoregressive language model.
Given a context (for LLMs, this would include the entire pretraining dataset, plus the prompt), you compress `context + next_token` for every possible next token. The tokens that co-compress best with the existing context are the 'least surprising' continuations. Choose one of them and iterate.
You can easily generate text with gzip this way. It won't be very good text, because gzip compression is not as sophisticated as a transformer + SGD, but the principle is the same.
The fact that the best compression algorithms are, in the LLM setting, still catastrophically bad at prediction suggests that the overlap between compression and prediction is weak at best.
https://bellard.org/ts_zip/
This shows that prediction algorithms (AI models) are also very good at compression, but compression algorithms (like the ones used in gzip) are not likewise very good at prediction. Which is evidence that compression is necessary but not sufficient for prediction.
>Mixing this can let you believe that you can use a compressor to do the same thing as LLMs, which you cannot.
You can, actually! Any compressor can be losslessly converted into a generator, and vice versa.
Traditional compressors like gzip are of course very simple and can only replicate rough patterns from the input. But they are technically doing the same thing.
Actually, it's trivial. I did it for fun once when I was learning about the PPM algorithm. It took about 15 minutes to reverse the whole thing.
I agree that technically they are doing the same thing but in practice LLMs are better compressors than PNGs (learned this while I was researching this topic). That was quite surprising to me.
That sounds like boostrapping the weights involved in early layers, to obviate the need for those layers to learn (optimize) for the distribution in the training set.
Makes me wonder idly, - is this conceptually akin in some sense to a "universal grammar," and if so - with a broad enough training set, is there a latent durable universal grammar that might be similarly recovered and injected to the benefit of all training, - does that grammar go beyond morphological/syntactical/grammatical features, into e.g. semantics and pragmatics
That was my thinking as well mainly to increase the speed of training. But it may turn out that the simple statistics that you can capture like this may account for 1% of the training and are likely to be captured as the first thing during the training.
But actually these techniques are used but they are hidden as speculative decoding with increasing complexity of approximations. For example you can have a part of the network that predicts the next word based on the markov chain, the next approximation is more complex etc.
This paper proposes something similar where you can inject memory without training https://arxiv.org/abs/2605.16893
I tried asking a zip file to write a program for me but it did nothing.
I’m starting to think that compression is not, in fact, prediction.
Compression is not prediction, it is recall. Can we make predictions based on compression? Absolutely. Is memory encoded into physical neurons technically compression? I would argue also yes.
However, going from compression to prediction is a large jump that is unsubstantiated by this article and based on the claim that probabilistic recall is also prediction.
Two perfect counterpoints to this are markets and weather patterns. One cannot predict future events based on past performance or behavior. Change is the only thing that's constant, and chaos/entropy is everywhere we look.
For simple problems like programming, sure predictive recall works amazingly well, but let's not pretend LLMs are actually predicting something. This is exactly why LLMs suck at doing anything novel; they lack imagination and creativity.
Statistical compressors are all about prediction. They can trivially be reversed to make a generator (not as good as an LLM, but only because it would be a much much much smaller model). And an llm can trivialyl be plopped in there to replace the whole thing.
And a counterpoint to your counterpoint: If it was raining 10 seconds ago, it is very likely that it is still raining right now. There... you can make a prediction about weather based on historical data. It's like perlin noise, random but with discernible patterns
I know less about this than every other commenter here, but both weather patterns and market performance do seem predictable based on past behavior when modeled at the right level of abstraction. “Sunshine on Monday” does not imply “rain on Tuesday”, but “cold front moving in Monday night” does. (Likewise “stock up Monday” doesn’t imply “stock down Tuesday” but “CEO arrested for fraud on Monday” does.)
I think this is relevant to the discourse on LLMs/programming because for months, people said “they’re just regurgitating their training set,” but now I think people are seeing (I am seeing) that they do learn more abstract models of the world than that. I don’t really know how, but it’s why they can generalize from other codebases and tools and so on.
> Likewise “stock up Monday” doesn’t imply “stock down Tuesday” but “CEO arrested for fraud on Monday” does
Stock market is even more pathological than weather here. Most statistical tools have baked-in assumptions about some degree of independence between factors under analysis. In weather, things are hard, because everything affects everything but in some random sense. With stock market, it's worse, because some things affect others in an adversarial way. A pattern discovered is not merely drowned in noise, but rather actively canceled out as other people discover and act on it. The way to describe this in formal terms is, I believe, that the stock market is "anti-inductive".
Fortunately, most things we deal with life are much friendlier to predictions than weather or stock prices.
> I think people are seeing
Better late than never, given that this has been empirically demonstrated two years ago by Anthropic. I actually don't understand why people don't make a bigger deal out of it - maybe it's because the AI hype/hate fight is sucking all attention from the room. But the point is: we not only have a thing that can learn concepts we recognize and build abstract world model, but also we have a degree of visibility and control of it we never had before. We can't see, much less suppress or stimulate individual concepts or concept groups in a human brain directly, but we can with LLMs.
Good points. I looked up the definition for prediction and I suppose I'm stretching what I view as prediction.
> A prediction is a statement about what you think will happen in the future, often based on experience or knowledge. It can also be referred to as a forecast or an informed guess
Based on my reading of this definition, compression may inform prediction but it is not itself prediction. The examples cited in the blog post are examples of probabilistic recall based on past events or instances. More context means a higher chance that the recall is more likely to be aligned.
But it's hard for me to accept the leap to compression == prediction because in my mind a prediction is an informed guess about something that hasn't yet come to pass. But thinking more about it, time is a human concept and so who's to say the temporal reference means anything at all here. Maybe probabilistic recall is the same as predictive forecasting if time is an invented concept and essentially means nothing?
Is everything fundamentally deterministic if you know everything in the universe or does free will exist?
IDK to be honest, I'm just more frequently surprised by new things that happen every day than I am at things that stay the same, even if mostly things stay the same. Maybe I just don't notice them and nothing actually ever happens.
Side note: the inevitable consequence of this line of reasoning will eventually become that LLMs given enough power are in fact intelligent and sentient, and I'm worried about how that affects humanity as a whole. Are we about to subjugate the most intelligent thing humanity has ever created, or is it about to subjugate us?
The rabbit hole gets deep quick when making the leap between a fancy recall mechanism and novel prediction, but I agree they're not that different in the end. I just believe it's important to be nuanced or else we'll miss when AGI actually happens (maybe it's already here).
I always feel like people leave out the third case of the analogy: indexing
The article itself has decision trees for the compression explanation, which is also a lookup index.
In each case you try to recognise (re)usable structure.
Self-indexing succinct data-structures are a good example of the third side of the coin.
So it's a trinity: compression, prediction, indexing
There is a correct sense, but we're sort of garbling concepts here:
Predictability is the inverse of information density.
Low information density enables high compression, and vice versa.
It's called entropy. This is basic information theory to be quite frank..
She covers all of this in the post
> compressors and LLMs
Why only LLMs? All statistical models are compressor. You can say "model" and "compressor" are synonyms.
Article does not mention "embeddings" at all, even though it's commonly viewed as a compression method. Also "encoder" part on "auto-encoders".
I often wonder how would language fare if we didn't have redundancy in abstractions, why do things get different terms, and if there is such a smaller set that contains everything in a lossless way (english-wise)
This immediately reminded me of the Hutter Prize (http://prize.hutter1.net/) - a contest that has run since 2005(?) based on the premise that compression is closely related to intelligence.
A few years ago I published BIDEN: Binary Inference Dictionaries for Electoral NLP, based on this idea - https://matthodges.com/posts/2023-10-01-BIDEN-binary-inferen...
1) am I allowed to scan the entire corpus in advance before I populate the dictionary? Is this a stream, or is their an EOF marker I will know in advance?
2) if 1) then "prediction" isn't the word I'm looking at.
The comparison can be carried on to another even crazier level: Evolution is compression. All the complexity of biology is executed at the highest possible efficiency.
DNA is a biological embedding.
I don't think this is true. Evolution is lazy and suboptimal, not characterized by the "highest possible efficiency".
Unrelated to the content: I was really pleased to see that this site defaults to the bare minimum for cookie consent. I reflexively clicked "Reject all" only to see that it was already the default, which threw me off.
It's more or less obvious that the LLM is a lossy-compressed version of the training data; it reproduces sequences of tokens that are the sort of thing that could plausibly occur in the training data, and avoids sequences that are implausible.
Because most of the training data has good grammar, the LLM is strongly trained on grammar; it will rarely predict ungrammatical gibberish. Even if there are grammar mistakes in the data, they are not systematic and so don't reinforce each other.
The prediction happens when it is uncompressed right?
LLM embeddings are compressed training data.
To decompress that is to make a prediction (in this case to convert the embedding into readable text)
https://mattmahoney.net/dc/dce.html
Something Ngrok is doing is working, because first they manage to get Sam Rose of samwho.com and now this? At this point I care more about their blog than their products
How about dictionary based compression as a counter example? Or the zig zag encoding scheme used in JPEG? I find it difficult to cast some of the things that effectively compress data as prediction.
Dictionary-based compression is based on prediction that recently seen words will be used again. That happens to be generally true for lots of datasets, including human languages (zipf distribution).
JPEG's zig-zag is a primitive for quantization, throwing data away based on rough approximation of human perception and biology. That isn't compression itself. However, the rounded and zeroed-out data is then compressed using a combination of RLE and Huffman, set up to predict the data will have lots of zeroes and few other distinct values (which the earlier step forces to be true). Or if you think about the system as a whole, you could say that JPEG predicts images will be blocky low-frequency patterns of DCT.
> dictionary based compression
That corresponds to PCFG models.
I see it that prediction is a form of compression.
Say you have a computer file composed of two parts, the first represents the setup of an experiment, and the second is the data produced by the experiment.
If you have a good theory relating to this type of experiment, then you can predict much of the second part of the file. So you only need to store the first part and possibly some corrections to the least significant bits of some of the parts of the second part of the file.
Thus with good prediction, you can compress this type of file.
Another example is encrypted data. Statistically, encrypted data is indistinguishable from random. Truly random data is impossible to compress losslessly.
But if you had a predictor so smart that it could crack the encryption key, it could start predicting the rest of the encrypted stream, and therefore compress it.
Physics laws are the ultimate form of compression because they are so universal and say so much about so many things in few words, or a formula. This is why Newton's laws were a big achievement. And they enabled predicting the behavior of machines and started the industrial revolution. We are at yet another inflection point.
Awesome read and absolutely loved the interactivity. A lot of effort was put into this.
3Blue1Brown - "Compression is Intelligence": https://www.youtube.com/watch?v=l6DKRf-fAAM
Recently I wrote a blog post[0] expanding on a similar idea from the angle of ancient Greek philosophy, particularly Parmenides: to think is to compress.
[0]: https://faingezicht.com/articles/2026/05/28/shape-of-what-we...
See also: Bellard's Lossless Data Compression With Neural Networks
https://news.ycombinator.com/item?id=19589848
https://news.ycombinator.com/item?id=27244004
And also the LLM version, and LLMZip
https://bellard.org/ts_zip/
https://arxiv.org/abs/2306.04050
great talk by ilya sutskever on how this https://www.youtube.com/watch?v=AKMuA_TVz3A&t=2121s
i do agree with that point of view. I often referred to models as 'modern mp3s' storing a lossfull but lookalike version of information in order to counter that 'AI is totally new and not violating copyright by storing information in a magic fashion' argument.
Author's bio:
> Annie Sexton is a Developer Educator at ngrok with a passion for nerd-sniping developers.
So a company with an wildly superior compression algorithm stumbling into a society-breaking AI isn't so far-fetched. Jesus...Silicon Valley really was ahead of its time.
I mean, save for the part where the founders recognised the threat it posed to society and acted responsibly rather than unleashing it on the public and sucking down billions in VC money.
did the SSL cert expire? i'm getting a big scary warning about this blog
Joel from ngrok here, certs are all good! You're likely seeing some corp-level block, which does happen to us on occasion. Hope you'll check it out again from the relative freedom of your home network.
Reminds me of this...
In Silicon Valley, Richard Hendricks creates a revolutionary lossless data compression algorithm for his startup, Pied Piper.
This is simply wrong.
Compression requires prediction.
The better the prediction, the better the compression, whether you are measuring fidelity or result size.
This doesn't mean that compression is prediction.
I was rushing to post this and then found out somebody had already
You snoze, you loze!
https://www.youtube.com/watch?v=B6u-FPskfAE
There
Perhaps a similar observation; https://news.ycombinator.com/item?id=48703636 :
> Compression, Predictive modeling, or Complexity?
Perhaps a bad example: https://news.ycombinator.com/item?id=38400380 :
> "78% MNIST accuracy using GZIP in under 10 lines of code" (2023) https://news.ycombinator.com/item?id=37583593
Not really, compression doesn't require a world model, it's mathematically pure. Any AGI system must periodically reset it's prediction since the world is inherently stochastic. When we look at prediction markets they only seem to work in the long term because human language is abstracted away from the real world, again it's mathematically pure. That's why we get bugs in code and disputes with prediction outcomes.
A better title, you can improve your compression if you make an accurate prediction. Much like how a branch predictor can make a CPU do the same work in less time. Or when your symbols are true uncompressable rules of reality (which is probably meaningless both semantically and physically again due to inherent randomness)
The main difference between minimalist and maximalists are how much that set of uncompressable rules gives you. I suspect the search space is too large. What we see in practice is that lossy rules let you cover more ground but eventually you hit a wall and have to move to a lower level of abstraction to make progress. There are 10^360 paths in a go tree, but something like 10^300,000 for molecular chemistry and that's not even all the way up (or down, say 10^3000 for the standard model of physics that's 10^900,000,000 if you want to do chemistry without chemistry abstractions.). Just semiconductor fab is 10^(10^11) so 10^(10^16) with molecular chemistry (think finding an implementation for some sort of desired self assembly outcome). AI can be way way way smarter than humans and there just not be enough energy in the universe to find these needles. So we definitely need abstractions, but those are at odds with predictions and the choice of symbols often introduces abstractions that the designer didn't consider.
I'm surprised no one has mentioned the recent 3 Blue 1 Brown video on this topic: https://www.youtube.com/watch?v=l6DKRf-fAAM
Literally one of the first 10 comments on the thread? Huh?
https://news.ycombinator.com/item?id=49263792
Oops, I think I looked past it because I only imagine 3Blue1Brown as a floating imaginary educational voice not a real person with a name .