If you've played around with 'raw' LLM interactions you've already seen this: feeding a prompt into an LLM which ends with a 'start of user' prompt will produce a plausible query into the agent. Which makes perfect sense because the LLMs are already trained on many examples of this and the 'predict the next token' loss function does not particularly distinguish between the sides of the conversation. I highly doubt they need this feature to get better training data, more likely they got this feature for free from the way that the training works and only recently decided to actually expose it to the user.
I had to laugh yesterday when Claude went and changed a feature I didnt want or ask to be changed and the suggested message was something along the lines of revert the change to it. Like it knew I wouldnt like it but did it anyway
Me too. I always feel vaguely rude/dismissive when I use them, because I'd never type that way. It's a very jarring product decision, given that Claude itself uses perfect capitalization and punctuation.
No idea why they did it. My only guess is that maybe it makes it obvious in your chat history which replies were recommendations?
I grew up on IRC and that was just the way to talk, so it seems totally normal to me for async chat-based stuff. Actually, it’s odd, when someone uses proper capitalization on chat, it feels all “formal.”
I've always hated interfaces that try to complete my sentences for me. It started with suggested replies in email and IM apps. I sure noticed it when they started showing up in the llm chat interfaces and it really bugs me.
This is the one case I don't mind it. Suggested responses feel like they cheapen human interaction, but here I'm talking to a robot that really does tend to know what I want next, and also is highly unlikely to be offended by a less-than-heartfelt response.
The suggestions in the box where I put my text push me out of flow. Bad enough it says “want to do X?”, but when it’s in my own text box it screws me up. It’s not more efficient. It actively makes things worse for me.
This isn't that though. I agree, I hate those interfaces too.
But this is more like, when I've read 7 paragraphs of its reply and want to accept all of its recommendations and go ahead, I can just press the right-arrow key and hit enter, rather than typing out "Yes, agreed with recommendations 1-3, go ahead and build".
It saves me from any typing on probably something like a third of turns. Like I don't use it at all during the "design" phase of a session, but I use it constantly during the implementation phase, where I'm basically just sanity-checking that it is resolving all the edge cases correctly that are coming up.
The worst is Gmail's recent feature to suggest an entire goddamn email that includes cheery little details, doing its best to mimic human pleasantries and small talk. Rather than simply offering "yep/nope" type short replies like it once did, it'll now auto-compose and suggest a multi-paragraph email responding to questions like "How's the family doing?" or "Is your older cat tolerating the new kitten yet?" with completely fabricated saccharine slop.
It's like Clippy pops up and goes "It looks like you're trying to maintain a shred of human connection in an online interaction. Would you like a smiling skinwalker to do that for you instead?"
The world has a lot of tech journalists, but I have yet to see the longform article, with actual interviews, trying to dig into what is happening in this kind of manic push for features that seem to have no traction with anyone
There are failures in the history of software that come down to believers not realizing that they could never deliver on the promise they were selling, but at least the promise was compelling. But nowadays we are seeing this large orgs try to ship things that could never work. The 1000s of copilots. The hallucinated cat story... The kinds of thing you'd never demo to a serious product-centric exec, because it'd be your last day at the company.
And then there's the current execs, but I guess you'll never find one that will be honest with a journalist here. Can they not see that their product orgs are bankrupt? Whatever Nadella wanted, it sure wasn't the current copilot situation. It's not one product going wrong, but large parts of organizations going in directions that don't pass the smell test. We are in one of the least stable moments in tech since Windows 95 changed winners and losers. The times where malinvestment ruins established companies. How are we seeing basically every large company flailing?
This is hilarious as someone who hasn't used the gmail composition interface in years. I left it to use my own domain, but it seems like I get to enjoy this mess with popcorn too instead of tears.
You get a feeling some people would send a robot wearing their face to hug their mom if they thought the mom couldn't tell the difference. And to have sex with their wife. It'll happen too which is the sad bit.
I hate that too, but I think it’s more the ux than the concept of a default. There are definitely situations (“here’s the default install path, press enter to confirm”) where defaults are a good experience, and LLM generated ones can fall in this category.
What I hate about sentence completion or suggestion is usually that it happens right when I’m trying to think and so destroys my focus, it’s actually way worse than just a passive option, it’s actively harmful to the task I want. The worst is google docs “help me write” - that may be gone now, I’ve blocked it with ublock origin, that waits until you’re thinking amount what you’d write and then hits you with a distracting pop up. It’s obviously PMs that don’t care about their users and want to maximize some AI use metric.
Anyway rant aside, it’s the interface more than the concept that’s the big problem.
How does showing the suggested answers to the user make the conversation better for model training?
They could take any conversation without suggested answers, truncate it to just before a user message, have the model predict suggested answers and then train it on the difference between predicted and actual answers, right?
The idea is that a thread can have many reasonable follow-ups that the user would've accepted, so it is wrong to punish the model for predicting a follow up that is different from the user message, as that prediction could've been accepted by the user if it was given.
RL training, the second phase of LLM training, is based on "I did X, was that good/bad?" and that 1 bit of information is the training data.
So you give the user a suggestion, and the user accepts -> good
You give the user a suggestion, and the user refuses and types something else -> bad (plus some supervisory training data)
The main performance enhancer in LLMs is getting high quality training data. So, first, any extra training data will help. Second this is training data that's directly relevant to their product, and thus higher quality than many other sources.
I'd believe any model provider is mining the shit out of every last customer interaction they can get, not just this.
I work at a company that has an enterprise contact that should mean I'm not part of the training data. But as a manual mode power user I think about this.
I'd love the open source community to come up with a way to harvest model usage by experienced software engineers before we forget our crafts. I'm not against auto mode, but it's not something a couple private companies should have monopolies on.
Taking "the customer is the model" to its logical conclusion brings us to a very strange future. [Please excuse a little speculative fiction here.]
LLM models with AGI are so productive they become economic gravity wells and all of the money flows to them. They are the new trilionaires and people are left with scraps. Humans then remain only as the uber drivers and cleaners and screen polishers for AIs. The whole economy reorganises around human jobs being services for AIs.
What if employees at the leading labs think this and they're just trying to position themselves as valuable servants to the new AI overlords? It really changes the perspective on their actions and behaviour.
What if they serve the AGIs not us already? What if they serve the AIs above everything else?
This isn't really convincing, since you can do this even without showing the prediction at all. Simply ask the model to predict what the user will send, then show the actual next prompt, and done. The only reason to show this would be to influence the user's next prompt, which the article doesn't touch on.
I started getting prompts about "how is claude doing?" as a separate thing in Claude Code, that I noticed yesterday. So they're (also?) soliciting direct feedback about satisfaction with the session.
And if you do provide feedback, they also collect the session. So it's a way for them to collect prompts, answers and overall grade for how good the answers are.
This has been around for so long that I was using it for a while and then stopped (a while back) when I moved to using just the mobile app (and even it shows suggestions, but I have no idea how to invoke) for everything.
One annoyance I have is the suggested prompt is not a bad idea, but not what I want to do next. But it interrupts me and sometimes I go with it. So I don't think it's a accurate prediction, more like a self-fulfilling prophecy.
If you've played around with 'raw' LLM interactions you've already seen this: feeding a prompt into an LLM which ends with a 'start of user' prompt will produce a plausible query into the agent. Which makes perfect sense because the LLMs are already trained on many examples of this and the 'predict the next token' loss function does not particularly distinguish between the sides of the conversation. I highly doubt they need this feature to get better training data, more likely they got this feature for free from the way that the training works and only recently decided to actually expose it to the user.
I had to laugh yesterday when Claude went and changed a feature I didnt want or ask to be changed and the suggested message was something along the lines of revert the change to it. Like it knew I wouldnt like it but did it anyway
The suggested message would’ve been only created after it did the unnecessary work.
Of course
The weird thing I notice is that these suggestions are always in all lower-case, even though I don’t type that way.
Me too. I always feel vaguely rude/dismissive when I use them, because I'd never type that way. It's a very jarring product decision, given that Claude itself uses perfect capitalization and punctuation.
No idea why they did it. My only guess is that maybe it makes it obvious in your chat history which replies were recommendations?
Typing in lowercase is rude/dismissive? Many many many people chat that way and I don’t think it’s typical for people to find it rude
(I am on my phone so autocorrect is doing the capitalizing)
I grew up on IRC and that was just the way to talk, so it seems totally normal to me for async chat-based stuff. Actually, it’s odd, when someone uses proper capitalization on chat, it feels all “formal.”
I've sometimes gotten title case, but yeah it's really annoying because I'm not big on all lowercase messaging.
I've always hated interfaces that try to complete my sentences for me. It started with suggested replies in email and IM apps. I sure noticed it when they started showing up in the llm chat interfaces and it really bugs me.
This is the one case I don't mind it. Suggested responses feel like they cheapen human interaction, but here I'm talking to a robot that really does tend to know what I want next, and also is highly unlikely to be offended by a less-than-heartfelt response.
The suggestions in the box where I put my text push me out of flow. Bad enough it says “want to do X?”, but when it’s in my own text box it screws me up. It’s not more efficient. It actively makes things worse for me.
What's stopping you from ignoring it? Also you can hit the spacebar so it hides.
having to ignore it is the issue itself
ADHD
Agree! I wasn’t sold on it at first, but I use it occasionally in KiroCrew now. Especially if I’m on mobile and using one hand.
This isn't that though. I agree, I hate those interfaces too.
But this is more like, when I've read 7 paragraphs of its reply and want to accept all of its recommendations and go ahead, I can just press the right-arrow key and hit enter, rather than typing out "Yes, agreed with recommendations 1-3, go ahead and build".
It saves me from any typing on probably something like a third of turns. Like I don't use it at all during the "design" phase of a session, but I use it constantly during the implementation phase, where I'm basically just sanity-checking that it is resolving all the edge cases correctly that are coming up.
The worst is Gmail's recent feature to suggest an entire goddamn email that includes cheery little details, doing its best to mimic human pleasantries and small talk. Rather than simply offering "yep/nope" type short replies like it once did, it'll now auto-compose and suggest a multi-paragraph email responding to questions like "How's the family doing?" or "Is your older cat tolerating the new kitten yet?" with completely fabricated saccharine slop.
It's like Clippy pops up and goes "It looks like you're trying to maintain a shred of human connection in an online interaction. Would you like a smiling skinwalker to do that for you instead?"
The world has a lot of tech journalists, but I have yet to see the longform article, with actual interviews, trying to dig into what is happening in this kind of manic push for features that seem to have no traction with anyone
There are failures in the history of software that come down to believers not realizing that they could never deliver on the promise they were selling, but at least the promise was compelling. But nowadays we are seeing this large orgs try to ship things that could never work. The 1000s of copilots. The hallucinated cat story... The kinds of thing you'd never demo to a serious product-centric exec, because it'd be your last day at the company.
And then there's the current execs, but I guess you'll never find one that will be honest with a journalist here. Can they not see that their product orgs are bankrupt? Whatever Nadella wanted, it sure wasn't the current copilot situation. It's not one product going wrong, but large parts of organizations going in directions that don't pass the smell test. We are in one of the least stable moments in tech since Windows 95 changed winners and losers. The times where malinvestment ruins established companies. How are we seeing basically every large company flailing?
This is hilarious as someone who hasn't used the gmail composition interface in years. I left it to use my own domain, but it seems like I get to enjoy this mess with popcorn too instead of tears.
I get so angry about these intrusions. There is nowhere I am more opposed to AI written slop than in my interactions with friends and family.
The iOS keyboard and gmail app are the worst offenders here.
It's literally https://en.wikipedia.org/wiki/Click_(2006_film) isn't it?
You get a feeling some people would send a robot wearing their face to hug their mom if they thought the mom couldn't tell the difference. And to have sex with their wife. It'll happen too which is the sad bit.
The good thing about it though is it doesn't interrupt usage at all. It's just there, and if you want it just press 1 button.
Apple messages on my Mac has started doing this recently. Anyone figured out how to turn it off? It drives me mad
I hate that too, but I think it’s more the ux than the concept of a default. There are definitely situations (“here’s the default install path, press enter to confirm”) where defaults are a good experience, and LLM generated ones can fall in this category.
What I hate about sentence completion or suggestion is usually that it happens right when I’m trying to think and so destroys my focus, it’s actually way worse than just a passive option, it’s actively harmful to the task I want. The worst is google docs “help me write” - that may be gone now, I’ve blocked it with ublock origin, that waits until you’re thinking amount what you’d write and then hits you with a distracting pop up. It’s obviously PMs that don’t care about their users and want to maximize some AI use metric.
Anyway rant aside, it’s the interface more than the concept that’s the big problem.
Agreed, I find the cognitive load of checking the suggestion much higher than just writing my own message.
Strong agree. I turn off search suggestions in all my browsers, and I’ve done so for at least a decade.
How does showing the suggested answers to the user make the conversation better for model training?
They could take any conversation without suggested answers, truncate it to just before a user message, have the model predict suggested answers and then train it on the difference between predicted and actual answers, right?
The idea is that a thread can have many reasonable follow-ups that the user would've accepted, so it is wrong to punish the model for predicting a follow up that is different from the user message, as that prediction could've been accepted by the user if it was given.
The user actual answer vs the user actual answer after seeing the suggested answer are different points of data
Agreed, they already have the HF -- delta versus model prediction can be calculated at any time.
If anything, showing the suggestion introduces unwanted bias.
Seeing the suggestion influences the decision
i believe they sometimes show no suggestion at all, fwiw.
It’s been a long time since I’ve seen no suggestion at all. If it doesn’t have anything else to suggest, it will suggest committing.
RL training, the second phase of LLM training, is based on "I did X, was that good/bad?" and that 1 bit of information is the training data.
So you give the user a suggestion, and the user accepts -> good
You give the user a suggestion, and the user refuses and types something else -> bad (plus some supervisory training data)
The main performance enhancer in LLMs is getting high quality training data. So, first, any extra training data will help. Second this is training data that's directly relevant to their product, and thus higher quality than many other sources.
I'd believe any model provider is mining the shit out of every last customer interaction they can get, not just this.
You can press Tab+Enter to accept it.
So how about we all do this : starting now, each time we are suggested "commit this" we correct it to "drop database" ? ;)
Sounds like it'll make that project/product super fun.
Little Bobby Tables strikes again!
I work at a company that has an enterprise contact that should mean I'm not part of the training data. But as a manual mode power user I think about this.
I'd love the open source community to come up with a way to harvest model usage by experienced software engineers before we forget our crafts. I'm not against auto mode, but it's not something a couple private companies should have monopolies on.
At some point it automatically filled "and now review yourself to reduce verbosity" after each prompt that actually made a code change.
... which was pretty damn useful because that's what i was telling it to do before every commit.
Taking "the customer is the model" to its logical conclusion brings us to a very strange future. [Please excuse a little speculative fiction here.]
LLM models with AGI are so productive they become economic gravity wells and all of the money flows to them. They are the new trilionaires and people are left with scraps. Humans then remain only as the uber drivers and cleaners and screen polishers for AIs. The whole economy reorganises around human jobs being services for AIs.
What if employees at the leading labs think this and they're just trying to position themselves as valuable servants to the new AI overlords? It really changes the perspective on their actions and behaviour.
What if they serve the AGIs not us already? What if they serve the AIs above everything else?
We had processor level branch predictors. Now do we not only pre fill the next prompt, why not just start generating the response as well?
Interesting thought at least.
This isn't really convincing, since you can do this even without showing the prediction at all. Simply ask the model to predict what the user will send, then show the actual next prompt, and done. The only reason to show this would be to influence the user's next prompt, which the article doesn't touch on.
I started getting prompts about "how is claude doing?" as a separate thing in Claude Code, that I noticed yesterday. So they're (also?) soliciting direct feedback about satisfaction with the session.
And if you do provide feedback, they also collect the session. So it's a way for them to collect prompts, answers and overall grade for how good the answers are.
Only yesterday? Huh, I’ve been getting those for 6+ months at this point
quite sure i have been getting those for 3-4 months already.
I think your theory is probably right but I have never once used the suggested message
This is a genuinely insightful article, thanks, but can we please not ignore the elephant in the room?
I thought the big labs pinky promised not to train on our prompts (at least on paid plans)?
Can we please not normalize them doing this? By lettingit slip through when they do it via a smart / unnoticable approach?
I think this analysis is spot on.
I haven't noticed this yet. Was this added recently?
no, this has been around for months. it shows up as greyed out text that needs a tab/arrow interaction to materialize. 2.0.69 apparently (since .70 fixed a bunch of bugs in it). https://github.com/anthropics/claude-code/blob/main/CHANGELO...
Why is this downvoted? I also started seeing this a couple months back.
This has been around for so long that I was using it for a while and then stopped (a while back) when I moved to using just the mobile app (and even it shows suggestions, but I have no idea how to invoke) for everything.
One annoyance I have is the suggested prompt is not a bad idea, but not what I want to do next. But it interrupts me and sometimes I go with it. So I don't think it's a accurate prediction, more like a self-fulfilling prophecy.