The most interesting is the realization that if the LLM's input is only the output of a professional (human), then by definition the LLM cannot mimic the process the (human) professional applied to get from whatever input they had to produce the output.
In other words an LLM can spit out a plausible "output of X", however it cannot encode the process that lead X to transform their inputs into their output.
"Explain how to solve" and "write like X" are crucially different tasks. One of them is about going through the steps of a process, and the other is about mimicking the result of a process.
Neural networks most certainly go through a process to transform input into output (even to mimic the results of another process) but it's a very different one from human neutral networks. But I think this is the crucial point of the debate, essentially unchanged from Searle's "Chinese Room" argument from decades ago.
The person in that room, looking up a dictionary with Chinese phrases and patterns, certainly follows a process, but it's easy to dismiss the notion that the person understands Chinese. But the question is if you zoom out, is the room itself intelligent because it is following a process, even if it's just a bunch of pattern recognition?
like OP originally said, the LLM doesn't have access to the actual process of the author, only the completed/refined output.
Not sure why you need a concrete example to "test", but just think about the fact that the LLM has no idea how a writer brainstorms, re-iterates on their work, or even comes up with the ideas in the first place.
i don't buy this logic. if i have studied an author greatly i will be able to recognise patterns and be able to write like them.
ex: i read a lot of shakespeare, understand patterns, understand where he came from, his biography and i will be able to write like him. why is it different for an LLM?
You will produce output that emulates the patters of Shakespeare's works, but you won't arrive at them by the same process Shakespeare did. You are subject to similar limitations as the llm in this case, just to a lesser degree (you share some 'human experience' with the author, and might be able to reason about their though process from biographies and such)
As another example, I can write a story about hobbits and elves in a LotR world with a style that approximates Tolkien. But it won't be colored by my first-hand WW1 experiences, and won't be written with the intention of creating a world that gives my conlangs cultural context, or the intention of making a bedtime story for my kids. I will never be able to write what Tolkien would have written because I'm not Tolkien, and do not see the world as Tolkien saw it. I don't even like designing languages
that's fair and you have highlighted a good limitation. but we do this all the time - we try to understand the author, learn from them and mimic them and we succeed to good extent.
that's why we have really good fake van gogh's for which a person can't tell the difference.
of course you can't do the same as the original person but you get close enough many times and as humans we do this frequently.
in the context of this post i think it is for sure possible to mimic a dead author and give steps to achieve writing that would sound like them using an LLM - just like a human.
You can understand his biography and analyses about how shakespeare might have written. You can apply this knowledge to modify your writing process.
The LLM does not model text at this meta-level. It can only use those texts as examples, it cannot apply what is written there to it's generation process.
Only if the LLM knows the inputs connected to particular outputs, pre-digital era or classified material might not be available, neither informal discussions with other experts.
Most importantly, negative but unused signals might not be available if the text does not mention it.
An LLM can always output steps, but it doesn’t mean they are true, they are great at making up bullshit.
When the “how many ‘r’ in ‘strawberry’” question was all the rage, you could definitely get LLMs to explain the steps of counting, too. It was still wrong.
Replace "LLM" with "student" and read that again. You don't just blindly give students output, you teach them, like what you are supposed to do with an LLM.
Enough with this analgoy. It's flawed on so many levels. First and foremost, stop devaluing humanitiy and hyping up AI companies by parroting their party line. Second, LLMs don't learn. They can hold a very limited amount of context, as you know. And every time you need to start over. So fuck no, "teaching" and LLM is nothing like teaching an actual human.
„Fitting“ is still too nice of a word choice, because it implies that it’s easy to identify the best solution.
I suggest „randomly adjusting parameters while trying to make things better“ as that accurately reflects the „precision“ that goes into stuffing LLMs with more data.
It was called learning already back when the field was called cybernetics and foundational figures like Shannon worked on this kind of stuff. People tried to decipher learning in the nervous system and implement the extracted principles in machines. Such as Hebbian learning, the Perception algorithm etc. This stuff goes back to the 40s/50s/60s, so things must have gone south pretty early then.
That isn't learning, it can read things in its context, and generate materials to assist answering further prompts but that doesn't change the model weights. It is just updating the context.
Unless you are actually fine tuning models, in which case sure, learning is taking place.
i don't know why you think it matters how it works internally. whether it changes its weights or not is not important. does it behave like a person who learns a thing? yes.
if i showed a human a codebase and asked them questions with good answers - yes i would say the human learned it. the analogy breaks at a point because of limited context but learning is a good enough word.
Grammarly seemed pretty dead on arrival the moment they added AI features. They would have said a lot more relevant and kept the costs down if they were strictly no-ai imo.
The funny thing is, their core "grammar" engine has to work on a language model + some hard heuristics anyway. So they were on a path to utilize this thing for real good, with concrete benefits.
Generative AI is a plague at this point. Everybody is adding to their wares to see what happens. It's almost like ricing a car. All noise, no go.
This feels like a desperate attempt to stay relevant in a post-LLM world. They’re basically wrapping an LLM in a "professional" skin and calling it an expert review. The problem is that once you start letting an AI "expert" dictate tone and logic, you effectively lobotomize the writer’s original intent. We’re reaching a point where AI is just reviewing other AI-generated text, creating a feedback loop of pure mediocrity. Copium for middle management, if you ask me.
Grammarly even from the start was very distracting to me even as a someone using english as a second language to communicate. I have developed my own taste and way of articulating thoughts, but grammarly (and LLMs today) forced me to remove that layer of personality from my texts which I didn't wanted to let go. Sure I sounded less professional, but that was the image I wanted to project anyways.
Unrelated but surprising to me that I've found built-in grammar checking within JetBrains IDEs far more useful at catching grammar mistakes while not forcing me to rewrite entire sentences.
It's great. Now that fancy writing is cheap and infinite, fields whose entire scholarship value was in obscurantist jargon bending have to actually start to turn on their brains and care about making more sense than an LLM can.
I disagree. You write when you have something to say. A service like Grammarly tries to help you convey what you want to say, but better. What you want to say is still up to you.
Words paint the picture, but the meaning of the picture is what matters.
You’re not counting all the office workers who have to write reports or emails, or all the scammers who write those websites to manipulate SEO or show you ads.
Children and young students, certainly. Adult students: almost 100%. If writing is your job, then by definition, and your problem is more often finding something to say, not writing it.
A few things worth flagging:
On GDPR: Using a named individual's identity to generate commercial AI output isn't obviously covered by "legitimate interest." Affected EU-based individuals likely have real grounds to object or request erasure.
On IP/publicity rights: You can't copyright an editing style — but you absolutely can have a right of publicity claim when a company profits from your name and simulated judgment without consent. The Lanham Act's false endorsement provisions could also be in play here.
The kicker: The "sources" cited by the feature were broken, spammy, or pointed to completely unrelated content. So the defense that suggestions are inspired by someone's actual work may not even hold up technically.
Frankly, I am surprised this was not shut down by their legal counsel (assuming they have one and they actually asked). The legal exposure here is significant. This could be defamation, there are publicity rights issues, copyright, and maybe even criminal liability.
It really feels so wrong to spare nobody, not even dead writer/people.
All it's gonna do is something similar to em-dashes where people who use it are now getting called LLM when it was their writing which would've trained LLM (the irony)
If this takes off, hypothetically, we will associate slop with the writing qualities similar to how Ghibli art is so good but it felt so sloppy afterwards and made us less appreciate the Ghibli artstyle seeing just about anyone make it.
The sad part is that most/some of these dead writers/artists were never appreciated by the people of their time and they struggled with so many feelings and writing/art was their way of expressing that. Van Gogh is an example which comes to my mind.[0] Many struggled from depression and other feelings too. To take that and expression of it and turn it into yet another product feels quite depressing for a company to do
yes i hate that. they still have the chutzpah of keeping doing it. and i am sure it's illegal in multiple legislation. because they are not writing articles where you can cite people, they are selling a product.
The most interesting is the realization that if the LLM's input is only the output of a professional (human), then by definition the LLM cannot mimic the process the (human) professional applied to get from whatever input they had to produce the output.
In other words an LLM can spit out a plausible "output of X", however it cannot encode the process that lead X to transform their inputs into their output.
i don't get what the point of what you are saying is? i can ask it to explain how to solve an integral right now with steps.
i can ask it to tell me how to write like a person X right now.
Is the reason it can show steps for solving an integral because the training set contained webpages or books showing how to do it?
"Explain how to solve" and "write like X" are crucially different tasks. One of them is about going through the steps of a process, and the other is about mimicking the result of a process.
Neural networks most certainly go through a process to transform input into output (even to mimic the results of another process) but it's a very different one from human neutral networks. But I think this is the crucial point of the debate, essentially unchanged from Searle's "Chinese Room" argument from decades ago.
The person in that room, looking up a dictionary with Chinese phrases and patterns, certainly follows a process, but it's easy to dismiss the notion that the person understands Chinese. But the question is if you zoom out, is the room itself intelligent because it is following a process, even if it's just a bunch of pattern recognition?
but llm can do both. so what's the point?
can you give a specific example of what an llm can't do? be specific so we can test it.
like OP originally said, the LLM doesn't have access to the actual process of the author, only the completed/refined output.
Not sure why you need a concrete example to "test", but just think about the fact that the LLM has no idea how a writer brainstorms, re-iterates on their work, or even comes up with the ideas in the first place.
i don't buy this logic. if i have studied an author greatly i will be able to recognise patterns and be able to write like them.
ex: i read a lot of shakespeare, understand patterns, understand where he came from, his biography and i will be able to write like him. why is it different for an LLM?
i again don't get what the point is?
You will produce output that emulates the patters of Shakespeare's works, but you won't arrive at them by the same process Shakespeare did. You are subject to similar limitations as the llm in this case, just to a lesser degree (you share some 'human experience' with the author, and might be able to reason about their though process from biographies and such)
As another example, I can write a story about hobbits and elves in a LotR world with a style that approximates Tolkien. But it won't be colored by my first-hand WW1 experiences, and won't be written with the intention of creating a world that gives my conlangs cultural context, or the intention of making a bedtime story for my kids. I will never be able to write what Tolkien would have written because I'm not Tolkien, and do not see the world as Tolkien saw it. I don't even like designing languages
that's fair and you have highlighted a good limitation. but we do this all the time - we try to understand the author, learn from them and mimic them and we succeed to good extent.
that's why we have really good fake van gogh's for which a person can't tell the difference.
of course you can't do the same as the original person but you get close enough many times and as humans we do this frequently.
in the context of this post i think it is for sure possible to mimic a dead author and give steps to achieve writing that would sound like them using an LLM - just like a human.
You can understand his biography and analyses about how shakespeare might have written. You can apply this knowledge to modify your writing process.
The LLM does not model text at this meta-level. It can only use those texts as examples, it cannot apply what is written there to it's generation process.
no it does and what you said is easily falsifiable.
can you provide a _single_ example where LLM might fail? lets test this now.
This is the plot of a short story of Borges’ called “Pierre Menard, the Author of Don Quixote.”
Is it not possible for the process of input to output be inferred by the llm and therefore applied to new inputs to create appropriate outputs.
Only if the LLM knows the inputs connected to particular outputs, pre-digital era or classified material might not be available, neither informal discussions with other experts.
Most importantly, negative but unused signals might not be available if the text does not mention it.
challenge: provide a single example where the LLM can only provide the output and not the steps? (in text scenario)
An LLM can always output steps, but it doesn’t mean they are true, they are great at making up bullshit.
When the “how many ‘r’ in ‘strawberry’” question was all the rage, you could definitely get LLMs to explain the steps of counting, too. It was still wrong.
can you provide a single example now with gpt 5.4 thinking that makes up things in steps? lets try to reproduce it.
Replace "LLM" with "student" and read that again. You don't just blindly give students output, you teach them, like what you are supposed to do with an LLM.
If you change the words in a sentence then it changes its meaning.
Yeah but obviously my point is in this context is that it doesn't.
You can't "teach" an LLM. It can't think. It's a simple pattern-matching algorithm, basically just an Eliza bot with a huge table of phrases.
You're not thinking, just regurgitating catch phrases that are factually incorrect hallucinations. So how are you any better than an LLM?
Speak for yourself...
Enough with this analgoy. It's flawed on so many levels. First and foremost, stop devaluing humanitiy and hyping up AI companies by parroting their party line. Second, LLMs don't learn. They can hold a very limited amount of context, as you know. And every time you need to start over. So fuck no, "teaching" and LLM is nothing like teaching an actual human.
> very limited amount of context
This isn't 2023 anymore
It all went south when we started to call it "learning" instead of "fitting parameters".
„Fitting“ is still too nice of a word choice, because it implies that it’s easy to identify the best solution.
I suggest „randomly adjusting parameters while trying to make things better“ as that accurately reflects the „precision“ that goes into stuffing LLMs with more data.
It was called learning already back when the field was called cybernetics and foundational figures like Shannon worked on this kind of stuff. People tried to decipher learning in the nervous system and implement the extracted principles in machines. Such as Hebbian learning, the Perception algorithm etc. This stuff goes back to the 40s/50s/60s, so things must have gone south pretty early then.
I agree with ya so much. I have seen so many people even in hackernews somehow give human qualities to LLM's.
This Grammarly thing seems to be a bastardized form of that not even sparing the dead.
I'd say that there was some incentive by the AI companies to muddle up the water here.
absolutely they can learn. you are being emotional and the original point is correct.
i give the LLM my codebase and it indeed learns about it and can answer questions.
That isn't learning, it can read things in its context, and generate materials to assist answering further prompts but that doesn't change the model weights. It is just updating the context.
Unless you are actually fine tuning models, in which case sure, learning is taking place.
i don't know why you think it matters how it works internally. whether it changes its weights or not is not important. does it behave like a person who learns a thing? yes.
if i showed a human a codebase and asked them questions with good answers - yes i would say the human learned it. the analogy breaks at a point because of limited context but learning is a good enough word.
Grammarly seemed pretty dead on arrival the moment they added AI features. They would have said a lot more relevant and kept the costs down if they were strictly no-ai imo.
The funny thing is, their core "grammar" engine has to work on a language model + some hard heuristics anyway. So they were on a path to utilize this thing for real good, with concrete benefits.
Generative AI is a plague at this point. Everybody is adding to their wares to see what happens. It's almost like ricing a car. All noise, no go.
This feels like a desperate attempt to stay relevant in a post-LLM world. They’re basically wrapping an LLM in a "professional" skin and calling it an expert review. The problem is that once you start letting an AI "expert" dictate tone and logic, you effectively lobotomize the writer’s original intent. We’re reaching a point where AI is just reviewing other AI-generated text, creating a feedback loop of pure mediocrity. Copium for middle management, if you ask me.
Grammarly even from the start was very distracting to me even as a someone using english as a second language to communicate. I have developed my own taste and way of articulating thoughts, but grammarly (and LLMs today) forced me to remove that layer of personality from my texts which I didn't wanted to let go. Sure I sounded less professional, but that was the image I wanted to project anyways.
Unrelated but surprising to me that I've found built-in grammar checking within JetBrains IDEs far more useful at catching grammar mistakes while not forcing me to rewrite entire sentences.
It's great. Now that fancy writing is cheap and infinite, fields whose entire scholarship value was in obscurantist jargon bending have to actually start to turn on their brains and care about making more sense than an LLM can.
> … have to actually start to …
Or do they?
I disagree. You write when you have something to say. A service like Grammarly tries to help you convey what you want to say, but better. What you want to say is still up to you.
Words paint the picture, but the meaning of the picture is what matters.
That's a tiny fraction. Most people write because they're told to write.
Are you talking about children or students? I think most people write because they want to communicate.
You’re not counting all the office workers who have to write reports or emails, or all the scammers who write those websites to manipulate SEO or show you ads.
Children and young students, certainly. Adult students: almost 100%. If writing is your job, then by definition, and your problem is more often finding something to say, not writing it.
A few things worth flagging: On GDPR: Using a named individual's identity to generate commercial AI output isn't obviously covered by "legitimate interest." Affected EU-based individuals likely have real grounds to object or request erasure. On IP/publicity rights: You can't copyright an editing style — but you absolutely can have a right of publicity claim when a company profits from your name and simulated judgment without consent. The Lanham Act's false endorsement provisions could also be in play here. The kicker: The "sources" cited by the feature were broken, spammy, or pointed to completely unrelated content. So the defense that suggestions are inspired by someone's actual work may not even hold up technically.
I would be surprised if the living writers can't sue over this.
Frankly, I am surprised this was not shut down by their legal counsel (assuming they have one and they actually asked). The legal exposure here is significant. This could be defamation, there are publicity rights issues, copyright, and maybe even criminal liability.
Man I really don't like this at all.
It really feels so wrong to spare nobody, not even dead writer/people.
All it's gonna do is something similar to em-dashes where people who use it are now getting called LLM when it was their writing which would've trained LLM (the irony)
If this takes off, hypothetically, we will associate slop with the writing qualities similar to how Ghibli art is so good but it felt so sloppy afterwards and made us less appreciate the Ghibli artstyle seeing just about anyone make it.
The sad part is that most/some of these dead writers/artists were never appreciated by the people of their time and they struggled with so many feelings and writing/art was their way of expressing that. Van Gogh is an example which comes to my mind.[0] Many struggled from depression and other feelings too. To take that and expression of it and turn it into yet another product feels quite depressing for a company to do
[0]: https://en.wikipedia.org/wiki/Health_of_Vincent_van_Gogh
> It really feels so wrong to spare nobody, not even dead writer/people.
That train left at full steam when companies scraped the whole internet and claimed it was fair use. Now it's a slippery slope covered with slime.
I believe there'll be no slowing down from now on.
They are doing something amazing, will they ask for permission? /s.
that's so scummy. why they even needs "names"? it's a rhetorical question...
Moreover, they don't even apologize:
"The work is public, hence the name. It's well known, it's in the data. Who cares".
What will they do next? Create similar publications with domainsquatting and write all-AI articles with the "public" names?
Is it still fair use, then?
yes i hate that. they still have the chutzpah of keeping doing it. and i am sure it's illegal in multiple legislation. because they are not writing articles where you can cite people, they are selling a product.
I think we can thank the current times and developments as a whole for unearthing the greediest of the greedy among us.
It's very enlightening, if you ask me.
Authority washing.
[dead]