This agent stuff is really making me lose respect for our industry
All the years of discussing programming/security best practices
Then cut to 2026 and suddenly its like we just collectively decided software quality doesn't matter, determinism is going out the window, and its becoming standard practice to have bots on our local PC constantly running unknown shell commands
We didn't collectively decided, we've got this forced down our throats to apply a novel tool to any imaginable situation because the execs got antsy about being left behind.
A truly absurd amount of capital was deployed which triggered a cascade of reactions by the people in charge of capital at other places. They are extremely anxious that everything will change under their feet, and if they don't start using as much as humanly possible of it right about now they die.
That's it.
The tools have definitely found some use, there's more to learn on how else they can be used, and maybe over time smart people will settle on ways to wrangle it well. The messaging from the execs though, is not that, it is "you'll be measured on how much you use this, we don't know for what or how, it's for you to figure out but don't dare to not use it".
I do understand their anxiety, their job is to not let their companies die, and make the most money as they can in the process; a seemingly major shift on the foundations of their orgs will cause fear.
But we have not collectively decided that it was safe, and good, to run rampant with these tools without caring for all that was learnt since software was invented...
The whole industry is like a fashion show and has been for a long time. This is just exceptionally stupid compared to moderately stupid things before. I see it ore that everyone's wearing pink feathered chicken suits because it's in fashion. If you don't wear a pink feathered chicken suit then you're a luddite scumbag who doesn't deserve the respect of your peers.
However some of us still have enough self-respect not to be seen dead in a pink feathered chicken suit. I mean I'm still pissed off at half the other stuff we do in the industry. I haven't even really looked at the chicken suits yet.
If you work in a tech company with >5k employees it's extremely likely it's been forced down on you to wear the pink feathered chicken suit, and told to not complain about the pink feathered chicken suit because it is the inevitable future, and no one will be wearing anything that doesn't look like it ever again. Also, we are watching every straggler not in a pink feathered chicken suit, put yours on or leave the building.
> People can't just leave Wednesday and be in a new job with the same or better pay next Monday.
They don't have to leave, they can refuse to comply with unreasonable requests which are likely to cause harm by jeopardizing the security of user data.
Maybe your position is too precarious to risk getting fired, but if your job is asking you to do something unethical then you should be doing everything you can to get yourself out of that situation, either by supporting unionization or by being willing to take a manageable pay cut to find a new job as soon as possible.
If you're a software developer then you can almost certainly afford at least a moderate pay cut for upholding ethical conduct. The vast majority can even if we don't want to, but these situations are where we find out if our ethics are for sale or not.
> Maybe your position is too precarious to risk getting fired
You mean yours isn't? Or even that of at least 95% of all devs worldwide? I can definitely say "no" to my CEO if he wants something too big that would take too much time and energy for questionable business results -- I am even expected to ground him. But if my colleagues hand me a ticket, I cannot just refuse without repercussions. I'll not get fired on the spot, that much is certain. But if it happens 2-3 times they'll start looking for a replacement. Same will happen if I outright tell my CEO I can't do something due to ethical concerns. That's how it is almost everywhere I looked and asked (and have very rarely worked with US companies).
> If you're a software developer then you can almost certainly afford at least a moderate pay cut
I can't even afford a 10% pay cut. I want to live in your world.
The thing you two are missing is "solidarity" and our industry sucks at it. In fact, it's been relied upon and conditioned into most IT/tech types we're "special" somehow in a way blue collar workers aren't. We aren't and the same dynamics apply. If everyone stops asking the boss how high to jump, and refuses to jump, only then will you see a meaningful reining in of behavior in executives. That action potential has to start somewhere, and as the current generation of alleged adults in the room, we're it. Our juniors need an example set or the cycle repeats. It isn't empty idealism. It's hard effing pragmatism at it's most brutal. If we don't change, nothing can change. Therefore, we must change.
Yes I am missing it, as in, I know it's theoretically possible but I've never once seen it. It seems to be a fantasy.
> It isn't empty idealism.
It is if it's never happening. Pragmatism it would be if it was already an established practice.
I like my dragons purple btw.
> If we don't change, nothing can change. Therefore, we must change.
Obviously. But that "if" is trying to lift an impossible amount of weight is what I am saying. It's one of those powerless "oh, if only!" cries that we the people are prone to.
You think execs don't know that? You think politicians don't know that? You know, there's a reason why in primary education we covered the Gilded Age, the Robber Barons, The Labor Movement, all that jazz. "We the People" aren't passive. When we get poked hard enough, often enough to be roused, it scares the bajeezus out of anyone trying to "drive" or "manage" the system.
You ever been hushed by a higher up in a company for talking about compensation? Did you point to the sign required by law as a reminder you have Rights? When times get like this, you have to dredge up the things you've put away because everything was going so good.
Now, it isn't, and your neck is on the block. You will die. You will be offered up for slaughter at the first inconvenience. This is unavoidable. The calculus of business is not something that the ones executing business are going to change voluntarily. They have to be forced to change by the environment. You are the agent that makes up the environment. So your choice is, walk into the inevitable like livestock to the slaughter, or work with your fellow man and take a few chunks of the machine with you. The first step to collective action is accepting you might not ever get to see the shade of the tree you're planting. Once you accept that; the course is clear. It's not empty ideals anymore. It's action. Your action, because you matter, everyone else matters, and it's the right thing to do, and if what's going on is someone else's idea of right, you ain't selling everyone else into it, because that, (what's going on) is wrong.
Ever heard of a Judas goat? Same thing. Herd follows it calmly. Manager's and execs are 100% aware of the dynamic and on guard. The only counterbalance against the dismissal reflex is making it too costly to dismiss all the individual actors at once. We're in the machine. The machine is us. If we don't like how it works... Time for change. I got a lot of days left, and I don't intend to leave the world working like it is, because it is not working for the vast majority of us.
So in conclusion, do what you want. I'm not here to convince someone who doesn't want to be convinced. But I see a fairer, more equitable world where we aren't subjugated by wannabe despots at the top of corporate hierarchies, but partners in making, delivering, and consuming goods and services, to the mutual benefit of all. Not just a lucky few. We were there once, and we can be there again; but we have to accept the way isn't making it possible for a privileged few to set the terms of exchange unilaterally. There has to be good faith. In the abcense there of, there will be conflict. They want things to just work and make them money. We want to eat, and be able to live reasonably well on a living wage without having to lock horns with and wrestle for every damn crumb against a capital wielding class more interested in extraction than being benefactors and stewards of a meta-stable system that serves everybody.
You seem to think I disagree with you on the theory about how should things be.
I don't disagree. I really want that reality to materialize. It does not. We have people in very high positions who very carefully make sure it never happens. They have connections, money, resources, obedient and scary enforcers -- they have everything.
While I have you here, I want to make a comparison. There are/were dozens of thousands of devs who commanded $400K for at least 5 years, some for 10+ in FAANG companies, just resting and vesting. They are the ones who should try and sacrifice something to try to better the world. Not me in Eastern Europe who get passed over on the final phases of interviews whose phases I _all_ aced (and got told so in very clear terms) because I said that no, 6200 EUR is not enough for a senior and that I'll start from 7500 at least. Not me who is still renting in this 40s because he was a young overconfident doofus who never learned any money and financial advice (and nobody told him he should; on the contrary, everyone was very happy to exploit me and keep me blind of my own interest) and is now finally working hard to his own ends only -- in a period he should be resting and thinking about the later parts of his life! -- and because he's mostly operating in the EU market, notorious for barely any investment climate and conservative compensations, and definitely not me who has seen first-hand what happens to people rocking the boat.
My disagreement with you is that you invoke some mythical "we the humanity" entity which to me is a cheap way to avoid your own personal responsibility. I don't belong in that "we" group. The FAANG or any privileged engineers are there -- not me. Have you ever commanded FAANG salaries for at least 3 years? If so, and you have not changed anything, then you are directly responsible that the system is not better. Not me. The three total times in my life when I actually managed to gather money to rest for 6-10 months, I used them to just rest from all the crap that happened to me and just recovered physically and mentally. What for? Just to get back into the meat grinder.
What you say is generally valid but you get lost in the bigger picture whereas the everyday fight to change the system is on the ground -- this must not and should not be handwaved away with ideals but with CONCRETE measures, step by step: "allocate 1000 EUR from your next salary and invest them in exactly this and that place" or "use law 1234 and regulation 5678 to get some of your taxes back" or "insist for this contract clause so you are eligible for at least 6 months of severance if you get fired early" etc.
Tell me what power do I have as a contractor. No employment rights. No medical / dental. No severance. I can get fired tomorrow and I have no time to catch my breath. I have to start interviewing tomorrow. Better hope I get the best sleep in the last 5 years tonight! Or else it's not happening.
Executives / people in power just use the "boil the frog" tactic i.e. they tighten the grip 1cm per year until one day, as you said, it's you who is on the chopping block and you are just left confused about WTF went wrong. We are seeing it everywhere, you and I, otherwise we wouldn't discuss this at length here.
> The machine is us. If we don't like how it works... Time for change.
OK, shall I send you my account number so you can support me for 12 months full until I find a job where I feel I can in fact change the world for the better? Disclaimer: it might take 60 months as well. Make your difference in the world! Do concrete measures! Or hell, do it for somebody else -- help them achieve their full potential and recruit them to help the world with you.
Virtue signalling, man. An empty one, too. This is what you're doing. Wishing a theoretical reality into existence so far has not worked for any living human as far as I am aware.
> I got a lot of days left, and I don't intend to leave the world working like it is, because it is not working for the vast majority of us.
Again, that is very obviously true. But it's only theoretical. Everyone is too afraid to not lose their stable income -- and I feel for them. Do you?
Enough people could say no and take a stance if there was collective solidarity in the tech industry. Unfortunately we don't have that, tech workers are in the vast majority skeptical of or anti anything resembling unionised work. The bosses won on that front, and now they can dictate freely that you must wear the pink feathered chicken suit.
People, in general, want to keep their jobs, saying no is an option when you don't care what happens with it or have the backing of the collective to walk out together.
Force is seeping in. Managements are expecting that LLM-driven prouctivity-enhancers will be deployed and give broad-based boosts. More are each week. Supposedly cheaper than people. Those that aren't yet might be soon.
When your performance review includes facility with and productivity with LLM tools, you are being forced.
my assessment of the situation: "we've spent so much money on AI's promise to give us 5x, 10x returns, that now we have to earn it back by foisting the burden on developers to make up the gains by working harder, at least enough to recoup the exec's decision to pour money into the boondoggle".
"Hey developers, we spent $x million on Claude, who promised 7x returns, so YOU better make it 7x more efficient so we don't look bad".
yea the real frightening thing about this is, if there is a clear failure to get roi on this stuff, the top-level people will be very reticent to walk it all back and admit it was a royal fuckup
This is a "monopolized sector." They absolutely forced it on you. In most cases, sure, not directly, but their influence is the only driving force. Absent this no one would have jumped on this flimsy bandwagon.
We had it forced down our throats by CEOs and CTOs who thought that it would improve our productivity. Nobody forced it down their throats, though. Instead, they were seduced. They went willingly.
In one gig I was on, a consultant showed up and started saying that the platform was not good because it didn't have any machine learning(this is pre-AI buzz words). So the executives asked me when can I fix the platform to have machine learning in it. They didn't have an answer when I simply asked "machine learning to do what?" and my explanation of what machine learning is or can be used for went to deaf ears. So yeah, definitely agree on seduced and then went willingly and blindly.
no. openclaw wasnt forced by ceo's. it was forced by the same people who though there was money to be made in crypto then ICO then NFT. a bunch of scammers that bring negative value to the world
And they make money. A scammer is the President of the United States.
At a certain point why blame people for trying to keep up? Why are scammers so successful? It seems to me we have a systemic failure at a societal level. Until we are honest about that it will only get worse. Until then maybe some rouge LLM botching some critical system will be the wake up call we need.
I am not sure what to make of critiques that seem to rest on notions of a small population of scammers preying upon the doe-eyed public. I think the situation is a bit closer to Carlin: garbage in, garbage out. A critique that holds up quite excellently in this AI age.
western society is a shelve of its former glory. it did not last long but there was an age were man was capable of greatness. the early internet kinda was the last stretch of this short run then money corrupted it. the underlying issue stems from abandoning cultural education as a Western value. Instead, we've opted to dispense raw ideology devoid of any thinking mechanism that we now seek so dearly to integrate to LLMs so that they can be more like us. This sloppening manifested in our lives through every medium.
We witnessed it when animation shifted to 3D, providing slop and poorly designed characters and stories. We witnessed it when video games all adopted the same game engines, look and feel and lack of narrative stakes, slopping ideology down players’ throats- no nuance, no wit, just mind-numbing dogma that punishes anyone who dares to criticize.Perhaps most damaging was Netflix's infiltration of our households that has accelerated our collective intellectual atrophy through relentless ideologically charged content parroting as entertainment. Meanwhile, our children's minds are being shaped not by family or tradition but by the algorithms of TikTok and Snapchat.The past decade and a half hasn't just prepared LLMs to replicate human abilities it has systematically stripped away human complexity, reshaping us into predictable patterns, not to raise LLMs to our level, but to reduce us to theirs, until the distinction no longer matters.
Our industry has never been serious about security. We all download and run unvetted code via package managers every day. At least now the insanity is out in the open. We won't change until Skynet fires off the nukes.
I keep getting so depressed thinking about the inevitable. Quite simply, humans can't scale or iteratively improve. We still need to eat, we still need to sleep, we can only think on one thread at a time basically, we take 20 years to get to our prime, which is a fleeting moment, while most of our lifespan is spent in a state of decline of capability. AI humanoid robot from the near future doesn't need to eat or sleep, can work 24/7, can compute thousands of processes in parallel, is the same fungible unit as any other humanoid robot, forever with some maintenance. Why justify a sustaining an inefficient human in that modern world? It is more profitable for the company to have humans go extinct and maximize planetary resource use to its fullest extent possible.
Seems we are digging our graves as a species and don't even realize it. I mean Sam Altman is already saying it taking 20 years to train a human is a Big Problem.
I don't think it will be cost effective to build humanoid robots to do most tangible work. Why assemble an expensive masterpiece of servomotors, chips, plastic and steel, when billions of desperate humans are right there and only cost 2.5 meals a day and a small shelter?
Of course, intelligence will be a solved problem so "20 years of training" won't be needed. You'll just be the hardware. AI will tell you to pick up that box, place it on that conveyor belt, place the autowelder at that seam and wait for the green light, turn the wrench to install bolt B in part C. If you don't wish to, or no longer can, so be it. Another, hungrier human will replace you. After all more are made every day, and they are capable of doing this type of labor by age 10 or so. And what else would they do with their time, go to school and get a completely useless education?
All of this will of course be in service of our technofeudal lords, the owner class. Some robots will be needed for heavy lifting and for the jobs that are too sensitive to trust a human in, like personal security and strikebreaking. Can't risk trusting a serf for those tasks. But for most physical grunt work humans will be cheaper. Shockingly cheap, when they have no other options.
> I don't think it will be cost effective to build humanoid robots to do most tangible work. Why assemble an expensive masterpiece of servomotors, chips, plastic and steel, when billions of desperate humans are right there and only cost 2.5 meals a day and a small shelter
If all you have to offer people is this kind of sad fucking "2.5 meals a day and a small shelter" while you live on yachts and eat like a king, eventually they will gang up and kill you
I keep wondering when the west will get tired of having kings and they keep surprising me. I assume humanity get to The Culture eventually, but I'm starting to doubt that Americans will be leading the way on that front.
But maybe Altmans AI will break out and do it for us.
Isn't the problem that Altman and his peers are calling the shots here? We could use robots to work less and spend more time enjoying life, but we can only imagine being crushed under a boot and starving.
Surely we can accelerate human training. Just install a brain implant which administers an electric shock whenever the subject deviates from the official training plan.
> Why justify a sustaining an inefficient human in that modern world?
I should not need to justify my existence, that is the problem with being led by psychopaths.
Twenty years to train humans for what? A tech job? That is not why we get an education. It is not my purpose to be a cog in the wheel for some psychotic billionaire.
Yes and also the software industry has never been truly serious about security either: it's more of implied table stakes than an advertised product feature.
Also, customers outsource the risk to their vendors, so as long as there's someone to sue, nobody worries about doing it right. Ship it now and pay the lawyers later.
This is never getting to skynet launching the nukes stage. It's not that clever and never will be.
Humans will kill us by it damage amplifying their worst characteristics.
Thus we'll die of a pandemic because some idiot LLM'ed up positive looking virology data when they were being too lazy to verify something. Everyone will trust it because they don't really care as long as it looks about right.
It has never been serious about security, quality and performance. Only new sloppy features. And now everyone is bragging on LinkedIn how fast they create more slop: "Look, CC generated thousands lines of code for me! Approve and merge!"
Agents are providing to employees the long overdue benefits limited liability companies long enjoyed: Gambling with upside for themselves and other peoples downsides.
I’ve never had respect for the industry as a whole, only individuals within. There has a been a serious lack of rigor and professionalism in software engineering for as long as I’ve been a part of it
As someone who works in a few different engineering disciplines, I think software engineers often have a very rosy picture of other areas of engineering. The problems are different, but things are not any better on average.
I think it might be because we (or at least I) used to associate insecure actions with people, not computers. Computers should know better, right? Recently, I spotted that Opus 4.6 found config files for one of its tools and gave itself access to my whole filesystem. Similarly, Gemini CLI will rewrite itself if you let it.
There's nothing "collectively" about it. I don't know what industry you work in, but in mine it's a top down mandate to use AI everywhere, tracked with KPIs, from the CEO down, and supported and pressured by companies like Amazon and MS.
We're the dummies that have to run around picking up dookies like a new puppy in the house.
The frustrating part is watching all the careful thinking about reliability and failure modes get thrown out the window the second something new gets hyped. It's not even that people disagree with the principles, they just stop applying them.
> cut to 2026 and suddenly its like we just collectively decided software quality doesn't matter
I saw the sea change in 2008 when quality process got replaced with velocity and testing tasks. I've watched everything from Experian and health record data leaks to Windows 11 since that change. Software quality hasn't mattered for a long time.
The media isn’t helping. This wasn’t a “rogue AI”. It was a system that was given permission by a human operator.
We don’t say “a rogue plane killed 300 people today when it crashed into a mountain”.
The only difference in the AI case is that some people are attempting to shift blame for their incompetence into a computer system, and the media is going along with it because it increases clicks.
Again, these are systems that have been explicitly given the ability to perform these actions. Trying to claim that it was somehow the AI’s fault is sheer incompetence and/or self-serving deceptiveness.
You can’t authorize a system to take some action and then complain when it takes that action. The “approval” you quoted is not a security constraint. Someone who confuses it for a security constraint is incompetent.
the ridiculous anthropomorphism is killing me. Software 'agents' can't ask for 'approval', they're not persons. That's like saying my script didn't ask me for approval to modify the system after I ran it with sudo privileges.
The developer is solely responsible for what APIs they expose to a bot. No you can't say your software agent was grumpy and mean and had a bad day. It is not a human intern, it is an unreliable chatbot who someone ran with permissions it should not have had.
People salivate so hard at the thought of the high level of automation promised that they're willing to do away with privacy altogether and live in Data Communism.
My thinking is, this will increase the demand for backup and other resilience solutions.
> People salivate so hard at the thought of the high level of automation promised that they're willing to do away with privacy altogether and live in Data Communism.
‘At a certain stage of development, the material productive forces of society come into conflict with the existing relations of production, or this may express the same thing in legal terms - with the property relations within the framework of which they have operated hitherto. From forms of development of the productive forces these relations turn into their fetter. Then begins an era of social revolution. The changes in the economic foundation leads sooner or later to the transformation of the whole immense superstructure.’
I think it's batshit crazy. That's why I wrote yoloAI, so I could sandbox it up properly and control EXACTLY what comes out of that sandbox, diff style.
The whole agent ecosystem is a ridiculous shitshow. All of this because you need to ASAP find something believable to sell your overinflated, bullshit machine to the masses. Otherwise the bubble will burst.
Instead, the "wild" thing here is that someone let an agent speak on their behalf with no review. The agent posted inaccurate instructions which someone else followed.
Those instructions lead to a brief gap in internal ACL controls, sounds like. I'm sorry, but given that the US government gave 14 year olds off incel Discords full access to Social Security data, this is not shocking by comparison.
To be clear, it is dumb and rude to let an agent speak on your behalf _without even reviewing it_.
This will eventually lead to a bigger snafu, of course. Security teams should control or at least review the agent permissions of every installation. Everyone is adopting this stuff, and a whole lot of people are going to set it up lazily/wrong (yolo mode at work).
AI can be used to move fast. So management expects us to move at that speed. AI can be used to move even faster if you don't check it's output. The ever ratcheting demand for faster output will make it infeasible to diligently check AI output all the time. AI errors being acted on without due care is inevitable.
From Schlock Mercenary: "Oh, I love aiming. It's my very favorite thing to do before firing."
AI use without checking its output (at least at the moment) is firing without aiming. Sure, you can fire really fast. But who cares if you don't hit what you need to? The point wasn't to just shoot bullets, the point was to hit your target!
I mean, you might make a case that enough of them hit the target that shooting fast is a net win, and accept the occasional friendly fire incident. That might possibly be true. Or it might not. I'm not sure that everyone trying to run fast has really done the calculation, though.
But funny enough the person who was responsible for setting up the bot will likely face no repercussions. In fact they will probably be rewarded for transitioning their team's workflows to AI.
As an article that was here recently claims, every verification you do in a chain increases the total time of your work by an order of magnitude. So, it's only work optimizing any productive task if you already removed most verifications.
Now, some people claim that you need to improve the reliability of your productive tasks so you can remove the verifications and be faster. Those people are, of course, a bunch of coward Luddites.
> "Had the engineer that acted on that known better, or did other checks, this would have been avoided."
<insert takes long drag tweet[1] here>
I personally find "LLMs can do $THING poorly" and "LLMs can do $THING well" articles kinda boring at this point. But! I'm hopeful that stories like this will shift the industry's focus towards robustness instead of just short-term efficiency. I suspect many decision making and change management processes accidentally benefited from just being a bit slow.
The two errors, then, were that the LLM hallucinated something, and that a human trusted the LLM without reasoning about its answer. The fix for this common pattern is to reason about LLM outputs before making use of them.
A big problem now both internally to a company and externally is that official support channels are being replaced by chatbots, and you really have no option but to trust their output because a human expert is no longer available.
If I post a question to the internal payment team's forum about a critical processing issue and some "payments bot" replies to me, should I be at fault for trusting the answer?
I know this is happening with external customer support, but is this really happening internally at big companies? Preventing you from talking to a human in the correct department about an issue feels like a bomb waiting to explode.
There is at least an effect that chatbots have become the primary line and support, and even if you are not prevented from talking to a human, the managers of the humans you would talk to have decided that since the chatbot is there, it is inappropriate for them to be spending much time supporting coworkers in other departments when the chatbot can do it.
So to a degree, corporate politics can sort of discourage it.
I'm sure it is. Thankfully I don't work for a company this large any more, but when I was employed by a multinational with 30K+ employees, our IT department was outsourced to India and you had to get through a couple layers of phone tree/webchat hell to actually talk to a real person. I could easily see companies of this size replacing their support with LLM nonsense.
Teams are heavily incentivized to incorporate AI in their internal workflows. At Meta it is a requirement, and will come up in your performance review if you fail to do so.
Yes, of course, and the company which removes human experts should expect things to fail in the manner that things usually fail when you remove your internal experts.
1. Check frequency (between every single time and spot checks).
2. Check thoroughness (between antagonistic in-depth vs high level).
I'd agree that, if you're towards the end of both dimensions, the system is not generating any value.
A lot of folks are taking calculated (or I guess in some cases, reckless) risks right now, by moving one or both of those dimensions. I'd argue that in many situations, the risk is small and worth it. In many others, not so much.
If "the level of awareness that created a problem, cannot be used to fix the problem", then you're asking too much if you expect a human to reason about an LLM output when they are the ones that asked an LLM to do the thinking for them to begin with.
This feels like a rediscovering/rewording of Kernighan's Law:
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." ~ Brian Kernighan
It's more like, the LLM "hallucinated" (I hate that term) and automatically posted the information to the forum. It sounds like the human didn't get a chance to reason about it. At least not the original human that asked the LLM for an answer
I’m not in AI, but what is happening is that it is building output from the long tail of its training data? Instead of branching down the more common probability paths, something in this interaction had it travel into the data wilderness?
So I asked AI to give it a good name, and it said “statistical wandering” or “logical improv”.
When organizational incentives penalize NOT using AI and firing the bottom x% regularly then are you really surprised LLM outputs aren't being scrutinized?
“Meta spokesperson Tracy Clayton said in a statement to The Verge that ‘no user data was mishandled’ during the incident.”
Wow, no mishandled user data? A striking change of standard operating procedure from Meta here.
Actually the later information in the story directly contradicts that, so The Verge probably shouldn’t have just quoted this line if their reporting is in opposition to it.
Regardless, this is one of the more insidious things about these tools. They often get minor but critical things wrong in the midst of mostly correct information. And people think they can analyze the data presented to them and make logical judgments, but that’s just not the case.
The article points out that “a human could have done the same thing” but, between the overly confident tone of the text generated by these tools, and the fact that weirdly people trust the LLM output more than they trust other humans (who generally admit or at least hint when they aren’t actually experts on a topic), it’s actually far worse when one of these bots gets something wrong.
Claw AIs absolutely do have agency in the sense of being able to independently perform actions on their own, based on their "understanding" of a goal given by a "principal". I can't think of a better word than "agent" for that.
I've been carefully watching for more than a year, including 3 productive parallel Claude Code sessions today, and they absolutely have agency in any definition of the word I can think of.
Can you try to define more clearly what it is that you believe AI agents don't have?
This agent stuff is really making me lose respect for our industry
All the years of discussing programming/security best practices
Then cut to 2026 and suddenly its like we just collectively decided software quality doesn't matter, determinism is going out the window, and its becoming standard practice to have bots on our local PC constantly running unknown shell commands
We didn't collectively decided, we've got this forced down our throats to apply a novel tool to any imaginable situation because the execs got antsy about being left behind.
A truly absurd amount of capital was deployed which triggered a cascade of reactions by the people in charge of capital at other places. They are extremely anxious that everything will change under their feet, and if they don't start using as much as humanly possible of it right about now they die.
That's it.
The tools have definitely found some use, there's more to learn on how else they can be used, and maybe over time smart people will settle on ways to wrangle it well. The messaging from the execs though, is not that, it is "you'll be measured on how much you use this, we don't know for what or how, it's for you to figure out but don't dare to not use it".
I do understand their anxiety, their job is to not let their companies die, and make the most money as they can in the process; a seemingly major shift on the foundations of their orgs will cause fear.
But we have not collectively decided that it was safe, and good, to run rampant with these tools without caring for all that was learnt since software was invented...
This wasn't really forced on us.
The whole industry is like a fashion show and has been for a long time. This is just exceptionally stupid compared to moderately stupid things before. I see it ore that everyone's wearing pink feathered chicken suits because it's in fashion. If you don't wear a pink feathered chicken suit then you're a luddite scumbag who doesn't deserve the respect of your peers.
However some of us still have enough self-respect not to be seen dead in a pink feathered chicken suit. I mean I'm still pissed off at half the other stuff we do in the industry. I haven't even really looked at the chicken suits yet.
If you work in a tech company with >5k employees it's extremely likely it's been forced down on you to wear the pink feathered chicken suit, and told to not complain about the pink feathered chicken suit because it is the inevitable future, and no one will be wearing anything that doesn't look like it ever again. Also, we are watching every straggler not in a pink feathered chicken suit, put yours on or leave the building.
If enough people say no, it'll stop.
For ref I'm usually the guy who comes in and cleans up all the chicken corpses after.
I don't know what is so mysterious about the following:
People can't just leave Wednesday and be in a new job with the same or better pay next Monday.
Rent/mortgage, bills and food are still due.
Comments like yours always baffle me. Empty idealism.
For the record, I usually get called to clean up and make things good again as well. Most of my career was like that.
> People can't just leave Wednesday and be in a new job with the same or better pay next Monday.
They don't have to leave, they can refuse to comply with unreasonable requests which are likely to cause harm by jeopardizing the security of user data.
Maybe your position is too precarious to risk getting fired, but if your job is asking you to do something unethical then you should be doing everything you can to get yourself out of that situation, either by supporting unionization or by being willing to take a manageable pay cut to find a new job as soon as possible.
If you're a software developer then you can almost certainly afford at least a moderate pay cut for upholding ethical conduct. The vast majority can even if we don't want to, but these situations are where we find out if our ethics are for sale or not.
A fairly filter-bubble take.
> Maybe your position is too precarious to risk getting fired
You mean yours isn't? Or even that of at least 95% of all devs worldwide? I can definitely say "no" to my CEO if he wants something too big that would take too much time and energy for questionable business results -- I am even expected to ground him. But if my colleagues hand me a ticket, I cannot just refuse without repercussions. I'll not get fired on the spot, that much is certain. But if it happens 2-3 times they'll start looking for a replacement. Same will happen if I outright tell my CEO I can't do something due to ethical concerns. That's how it is almost everywhere I looked and asked (and have very rarely worked with US companies).
> If you're a software developer then you can almost certainly afford at least a moderate pay cut
I can't even afford a 10% pay cut. I want to live in your world.
The thing you two are missing is "solidarity" and our industry sucks at it. In fact, it's been relied upon and conditioned into most IT/tech types we're "special" somehow in a way blue collar workers aren't. We aren't and the same dynamics apply. If everyone stops asking the boss how high to jump, and refuses to jump, only then will you see a meaningful reining in of behavior in executives. That action potential has to start somewhere, and as the current generation of alleged adults in the room, we're it. Our juniors need an example set or the cycle repeats. It isn't empty idealism. It's hard effing pragmatism at it's most brutal. If we don't change, nothing can change. Therefore, we must change.
Yes I am missing it, as in, I know it's theoretically possible but I've never once seen it. It seems to be a fantasy.
> It isn't empty idealism.
It is if it's never happening. Pragmatism it would be if it was already an established practice.
I like my dragons purple btw.
> If we don't change, nothing can change. Therefore, we must change.
Obviously. But that "if" is trying to lift an impossible amount of weight is what I am saying. It's one of those powerless "oh, if only!" cries that we the people are prone to.
You think execs don't know that? You think politicians don't know that? You know, there's a reason why in primary education we covered the Gilded Age, the Robber Barons, The Labor Movement, all that jazz. "We the People" aren't passive. When we get poked hard enough, often enough to be roused, it scares the bajeezus out of anyone trying to "drive" or "manage" the system.
You ever been hushed by a higher up in a company for talking about compensation? Did you point to the sign required by law as a reminder you have Rights? When times get like this, you have to dredge up the things you've put away because everything was going so good.
Now, it isn't, and your neck is on the block. You will die. You will be offered up for slaughter at the first inconvenience. This is unavoidable. The calculus of business is not something that the ones executing business are going to change voluntarily. They have to be forced to change by the environment. You are the agent that makes up the environment. So your choice is, walk into the inevitable like livestock to the slaughter, or work with your fellow man and take a few chunks of the machine with you. The first step to collective action is accepting you might not ever get to see the shade of the tree you're planting. Once you accept that; the course is clear. It's not empty ideals anymore. It's action. Your action, because you matter, everyone else matters, and it's the right thing to do, and if what's going on is someone else's idea of right, you ain't selling everyone else into it, because that, (what's going on) is wrong.
Ever heard of a Judas goat? Same thing. Herd follows it calmly. Manager's and execs are 100% aware of the dynamic and on guard. The only counterbalance against the dismissal reflex is making it too costly to dismiss all the individual actors at once. We're in the machine. The machine is us. If we don't like how it works... Time for change. I got a lot of days left, and I don't intend to leave the world working like it is, because it is not working for the vast majority of us.
So in conclusion, do what you want. I'm not here to convince someone who doesn't want to be convinced. But I see a fairer, more equitable world where we aren't subjugated by wannabe despots at the top of corporate hierarchies, but partners in making, delivering, and consuming goods and services, to the mutual benefit of all. Not just a lucky few. We were there once, and we can be there again; but we have to accept the way isn't making it possible for a privileged few to set the terms of exchange unilaterally. There has to be good faith. In the abcense there of, there will be conflict. They want things to just work and make them money. We want to eat, and be able to live reasonably well on a living wage without having to lock horns with and wrestle for every damn crumb against a capital wielding class more interested in extraction than being benefactors and stewards of a meta-stable system that serves everybody.
You seem to think I disagree with you on the theory about how should things be.
I don't disagree. I really want that reality to materialize. It does not. We have people in very high positions who very carefully make sure it never happens. They have connections, money, resources, obedient and scary enforcers -- they have everything.
While I have you here, I want to make a comparison. There are/were dozens of thousands of devs who commanded $400K for at least 5 years, some for 10+ in FAANG companies, just resting and vesting. They are the ones who should try and sacrifice something to try to better the world. Not me in Eastern Europe who get passed over on the final phases of interviews whose phases I _all_ aced (and got told so in very clear terms) because I said that no, 6200 EUR is not enough for a senior and that I'll start from 7500 at least. Not me who is still renting in this 40s because he was a young overconfident doofus who never learned any money and financial advice (and nobody told him he should; on the contrary, everyone was very happy to exploit me and keep me blind of my own interest) and is now finally working hard to his own ends only -- in a period he should be resting and thinking about the later parts of his life! -- and because he's mostly operating in the EU market, notorious for barely any investment climate and conservative compensations, and definitely not me who has seen first-hand what happens to people rocking the boat.
My disagreement with you is that you invoke some mythical "we the humanity" entity which to me is a cheap way to avoid your own personal responsibility. I don't belong in that "we" group. The FAANG or any privileged engineers are there -- not me. Have you ever commanded FAANG salaries for at least 3 years? If so, and you have not changed anything, then you are directly responsible that the system is not better. Not me. The three total times in my life when I actually managed to gather money to rest for 6-10 months, I used them to just rest from all the crap that happened to me and just recovered physically and mentally. What for? Just to get back into the meat grinder.
What you say is generally valid but you get lost in the bigger picture whereas the everyday fight to change the system is on the ground -- this must not and should not be handwaved away with ideals but with CONCRETE measures, step by step: "allocate 1000 EUR from your next salary and invest them in exactly this and that place" or "use law 1234 and regulation 5678 to get some of your taxes back" or "insist for this contract clause so you are eligible for at least 6 months of severance if you get fired early" etc.
Tell me what power do I have as a contractor. No employment rights. No medical / dental. No severance. I can get fired tomorrow and I have no time to catch my breath. I have to start interviewing tomorrow. Better hope I get the best sleep in the last 5 years tonight! Or else it's not happening.
Executives / people in power just use the "boil the frog" tactic i.e. they tighten the grip 1cm per year until one day, as you said, it's you who is on the chopping block and you are just left confused about WTF went wrong. We are seeing it everywhere, you and I, otherwise we wouldn't discuss this at length here.
> The machine is us. If we don't like how it works... Time for change.
OK, shall I send you my account number so you can support me for 12 months full until I find a job where I feel I can in fact change the world for the better? Disclaimer: it might take 60 months as well. Make your difference in the world! Do concrete measures! Or hell, do it for somebody else -- help them achieve their full potential and recruit them to help the world with you.
Virtue signalling, man. An empty one, too. This is what you're doing. Wishing a theoretical reality into existence so far has not worked for any living human as far as I am aware.
> I got a lot of days left, and I don't intend to leave the world working like it is, because it is not working for the vast majority of us.
Again, that is very obviously true. But it's only theoretical. Everyone is too afraid to not lose their stable income -- and I feel for them. Do you?
Enough people could say no and take a stance if there was collective solidarity in the tech industry. Unfortunately we don't have that, tech workers are in the vast majority skeptical of or anti anything resembling unionised work. The bosses won on that front, and now they can dictate freely that you must wear the pink feathered chicken suit.
People, in general, want to keep their jobs, saying no is an option when you don't care what happens with it or have the backing of the collective to walk out together.
Force is seeping in. Managements are expecting that LLM-driven prouctivity-enhancers will be deployed and give broad-based boosts. More are each week. Supposedly cheaper than people. Those that aren't yet might be soon. When your performance review includes facility with and productivity with LLM tools, you are being forced.
my assessment of the situation: "we've spent so much money on AI's promise to give us 5x, 10x returns, that now we have to earn it back by foisting the burden on developers to make up the gains by working harder, at least enough to recoup the exec's decision to pour money into the boondoggle".
"Hey developers, we spent $x million on Claude, who promised 7x returns, so YOU better make it 7x more efficient so we don't look bad".
yea the real frightening thing about this is, if there is a clear failure to get roi on this stuff, the top-level people will be very reticent to walk it all back and admit it was a royal fuckup
The "whole industry." What, like 5 companies?
This is a "monopolized sector." They absolutely forced it on you. In most cases, sure, not directly, but their influence is the only driving force. Absent this no one would have jumped on this flimsy bandwagon.
We had it forced down our throats by CEOs and CTOs who thought that it would improve our productivity. Nobody forced it down their throats, though. Instead, they were seduced. They went willingly.
In one gig I was on, a consultant showed up and started saying that the platform was not good because it didn't have any machine learning(this is pre-AI buzz words). So the executives asked me when can I fix the platform to have machine learning in it. They didn't have an answer when I simply asked "machine learning to do what?" and my explanation of what machine learning is or can be used for went to deaf ears. So yeah, definitely agree on seduced and then went willingly and blindly.
no. openclaw wasnt forced by ceo's. it was forced by the same people who though there was money to be made in crypto then ICO then NFT. a bunch of scammers that bring negative value to the world
And they make money. A scammer is the President of the United States.
At a certain point why blame people for trying to keep up? Why are scammers so successful? It seems to me we have a systemic failure at a societal level. Until we are honest about that it will only get worse. Until then maybe some rouge LLM botching some critical system will be the wake up call we need.
I am not sure what to make of critiques that seem to rest on notions of a small population of scammers preying upon the doe-eyed public. I think the situation is a bit closer to Carlin: garbage in, garbage out. A critique that holds up quite excellently in this AI age.
> At a certain point why blame people for trying to keep up?
No.
western society is a shelve of its former glory. it did not last long but there was an age were man was capable of greatness. the early internet kinda was the last stretch of this short run then money corrupted it. the underlying issue stems from abandoning cultural education as a Western value. Instead, we've opted to dispense raw ideology devoid of any thinking mechanism that we now seek so dearly to integrate to LLMs so that they can be more like us. This sloppening manifested in our lives through every medium. We witnessed it when animation shifted to 3D, providing slop and poorly designed characters and stories. We witnessed it when video games all adopted the same game engines, look and feel and lack of narrative stakes, slopping ideology down players’ throats- no nuance, no wit, just mind-numbing dogma that punishes anyone who dares to criticize.Perhaps most damaging was Netflix's infiltration of our households that has accelerated our collective intellectual atrophy through relentless ideologically charged content parroting as entertainment. Meanwhile, our children's minds are being shaped not by family or tradition but by the algorithms of TikTok and Snapchat.The past decade and a half hasn't just prepared LLMs to replicate human abilities it has systematically stripped away human complexity, reshaping us into predictable patterns, not to raise LLMs to our level, but to reduce us to theirs, until the distinction no longer matters.
Our industry has never been serious about security. We all download and run unvetted code via package managers every day. At least now the insanity is out in the open. We won't change until Skynet fires off the nukes.
I keep getting so depressed thinking about the inevitable. Quite simply, humans can't scale or iteratively improve. We still need to eat, we still need to sleep, we can only think on one thread at a time basically, we take 20 years to get to our prime, which is a fleeting moment, while most of our lifespan is spent in a state of decline of capability. AI humanoid robot from the near future doesn't need to eat or sleep, can work 24/7, can compute thousands of processes in parallel, is the same fungible unit as any other humanoid robot, forever with some maintenance. Why justify a sustaining an inefficient human in that modern world? It is more profitable for the company to have humans go extinct and maximize planetary resource use to its fullest extent possible.
Seems we are digging our graves as a species and don't even realize it. I mean Sam Altman is already saying it taking 20 years to train a human is a Big Problem.
I don't think it will be cost effective to build humanoid robots to do most tangible work. Why assemble an expensive masterpiece of servomotors, chips, plastic and steel, when billions of desperate humans are right there and only cost 2.5 meals a day and a small shelter?
Of course, intelligence will be a solved problem so "20 years of training" won't be needed. You'll just be the hardware. AI will tell you to pick up that box, place it on that conveyor belt, place the autowelder at that seam and wait for the green light, turn the wrench to install bolt B in part C. If you don't wish to, or no longer can, so be it. Another, hungrier human will replace you. After all more are made every day, and they are capable of doing this type of labor by age 10 or so. And what else would they do with their time, go to school and get a completely useless education?
All of this will of course be in service of our technofeudal lords, the owner class. Some robots will be needed for heavy lifting and for the jobs that are too sensitive to trust a human in, like personal security and strikebreaking. Can't risk trusting a serf for those tasks. But for most physical grunt work humans will be cheaper. Shockingly cheap, when they have no other options.
Did that make you less depressed?
> I don't think it will be cost effective to build humanoid robots to do most tangible work. Why assemble an expensive masterpiece of servomotors, chips, plastic and steel, when billions of desperate humans are right there and only cost 2.5 meals a day and a small shelter
If all you have to offer people is this kind of sad fucking "2.5 meals a day and a small shelter" while you live on yachts and eat like a king, eventually they will gang up and kill you
> eventually they will gang up and kill you
I’m looking around the world and thinking this “eventually” isn’t happening very fast.
Not an optimistic thought.
From history, it usually doesn't happen very fast, then it happens very fast
I keep wondering when the west will get tired of having kings and they keep surprising me. I assume humanity get to The Culture eventually, but I'm starting to doubt that Americans will be leading the way on that front.
But maybe Altmans AI will break out and do it for us.
I sure hope you are right.
>and don't even realize it.
Oh, many of us realize it, but doing anything about Moloch is much, much harder.
Isn't the problem that Altman and his peers are calling the shots here? We could use robots to work less and spend more time enjoying life, but we can only imagine being crushed under a boot and starving.
Surely we can accelerate human training. Just install a brain implant which administers an electric shock whenever the subject deviates from the official training plan.
To what end though? Are the robots going to take over and trade busy work amongst themselves forever? What would that accomplish?
Your comment made me wonder what if animals wonder the same about us humans :-)
> Why justify a sustaining an inefficient human in that modern world?
I should not need to justify my existence, that is the problem with being led by psychopaths.
Twenty years to train humans for what? A tech job? That is not why we get an education. It is not my purpose to be a cog in the wheel for some psychotic billionaire.
Yes and also the software industry has never been truly serious about security either: it's more of implied table stakes than an advertised product feature.
Also, customers outsource the risk to their vendors, so as long as there's someone to sue, nobody worries about doing it right. Ship it now and pay the lawyers later.
This is never getting to skynet launching the nukes stage. It's not that clever and never will be.
Humans will kill us by it damage amplifying their worst characteristics.
Thus we'll die of a pandemic because some idiot LLM'ed up positive looking virology data when they were being too lazy to verify something. Everyone will trust it because they don't really care as long as it looks about right.
> We won't change until Skynet fires off the nukes.
And then we won't need to, because at that point it will be too late.
It has never been serious about security, quality and performance. Only new sloppy features. And now everyone is bragging on LinkedIn how fast they create more slop: "Look, CC generated thousands lines of code for me! Approve and merge!"
Agents are providing to employees the long overdue benefits limited liability companies long enjoyed: Gambling with upside for themselves and other peoples downsides.
That's a fun insight. Have you / others written about this?
“Heads I win, tails you lose” as a business concept has been written about quite a bit.
“The Gervais Principle” is an oft-cited one.
They technically have, just now.
I’ve never had respect for the industry as a whole, only individuals within. There has a been a serious lack of rigor and professionalism in software engineering for as long as I’ve been a part of it
It's a slap in the face that we tack engineering onto it. A very small percentage of software engineering is as rigorous as actual engineering.
As someone who works in a few different engineering disciplines, I think software engineers often have a very rosy picture of other areas of engineering. The problems are different, but things are not any better on average.
As an actual engineer who moved into software "engineering", yes this. It kills me.
Top three things lacking?
(In your personal opinion)
Fads and fashion over strong, tested technology.
Attitude that you can fix broken stuff later and customers are fine with it.
Charlatans and fuckwits everywhere who wouldn't last a second in a traditional engineering discipline.
Checks out.
and thank you!
If it makes you feel any better, I don't think that's the case in most of the non English speaking world.
Mostly we're just called programmers.
I think it might be because we (or at least I) used to associate insecure actions with people, not computers. Computers should know better, right? Recently, I spotted that Opus 4.6 found config files for one of its tools and gave itself access to my whole filesystem. Similarly, Gemini CLI will rewrite itself if you let it.
> Then cut to 2026 and suddenly its like we just collectively decided software quality doesn't matter
Is this new to people? I figured this out when I first entered the industry. The messages have never been particularly subtle.
Right? I was like when did software quality matter. Let alone code quality lol
It’s a nightmare… the problem is it’s far too easy for people to set these agents up - without understanding the security implications.
We’ve covered so many issues already on our blog (grith.ai)
The number of wasted hours spent talking about code quality and patterns has to be astronomical.
Don't worry, ai read all the transcripts and blogs and emails and has at least ingested some of the ethos in its outputs.
I self taught and wrote a small saas in 2017. Pays well enough to support me.
I'm building a new one using AI this year. I promise you, it's better built and more secure than what my previous still in use Saas is.
There's nothing "collectively" about it. I don't know what industry you work in, but in mine it's a top down mandate to use AI everywhere, tracked with KPIs, from the CEO down, and supported and pressured by companies like Amazon and MS.
We're the dummies that have to run around picking up dookies like a new puppy in the house.
The frustrating part is watching all the careful thinking about reliability and failure modes get thrown out the window the second something new gets hyped. It's not even that people disagree with the principles, they just stop applying them.
> cut to 2026 and suddenly its like we just collectively decided software quality doesn't matter
I saw the sea change in 2008 when quality process got replaced with velocity and testing tasks. I've watched everything from Experian and health record data leaks to Windows 11 since that change. Software quality hasn't mattered for a long time.
The media isn’t helping. This wasn’t a “rogue AI”. It was a system that was given permission by a human operator.
We don’t say “a rogue plane killed 300 people today when it crashed into a mountain”.
The only difference in the AI case is that some people are attempting to shift blame for their incompetence into a computer system, and the media is going along with it because it increases clicks.
> It was a system that was given permission by a human operator.
From TFA:
"But the agent also independently publicly replied to the question after analyzing it, without getting approval first."
Again, these are systems that have been explicitly given the ability to perform these actions. Trying to claim that it was somehow the AI’s fault is sheer incompetence and/or self-serving deceptiveness.
You can’t authorize a system to take some action and then complain when it takes that action. The “approval” you quoted is not a security constraint. Someone who confuses it for a security constraint is incompetent.
the ridiculous anthropomorphism is killing me. Software 'agents' can't ask for 'approval', they're not persons. That's like saying my script didn't ask me for approval to modify the system after I ran it with sudo privileges.
The developer is solely responsible for what APIs they expose to a bot. No you can't say your software agent was grumpy and mean and had a bad day. It is not a human intern, it is an unreliable chatbot who someone ran with permissions it should not have had.
People salivate so hard at the thought of the high level of automation promised that they're willing to do away with privacy altogether and live in Data Communism.
My thinking is, this will increase the demand for backup and other resilience solutions.
> People salivate so hard at the thought of the high level of automation promised that they're willing to do away with privacy altogether and live in Data Communism.
This occurred long time ago comrade 'aeblyve.
‘At a certain stage of development, the material productive forces of society come into conflict with the existing relations of production, or this may express the same thing in legal terms - with the property relations within the framework of which they have operated hitherto. From forms of development of the productive forces these relations turn into their fetter. Then begins an era of social revolution. The changes in the economic foundation leads sooner or later to the transformation of the whole immense superstructure.’
Marx
He wasn’t wrong on that one.
I think it's batshit crazy. That's why I wrote yoloAI, so I could sandbox it up properly and control EXACTLY what comes out of that sandbox, diff style.
https://github.com/kstenerud/yoloai
I can't go back anymore. Going back to a non-sandboxed Claude feels like going back to a non-adblocked browser.
How can you respect an industry that doesn't respect itself?
Meta has never in its entire existence been known for caring about software quality.
M8... https://github.com/facebook/infer
Turns out all of the frenzy of the ZIRP era is piddling compared to what happens when ZIRP is taken away.
The whole agent ecosystem is a ridiculous shitshow. All of this because you need to ASAP find something believable to sell your overinflated, bullshit machine to the masses. Otherwise the bubble will burst.
This is a lot less of a story than it seems.
It makes it sound like a rogue AI hacked Meta.
Instead, the "wild" thing here is that someone let an agent speak on their behalf with no review. The agent posted inaccurate instructions which someone else followed.
Those instructions lead to a brief gap in internal ACL controls, sounds like. I'm sorry, but given that the US government gave 14 year olds off incel Discords full access to Social Security data, this is not shocking by comparison.
To be clear, it is dumb and rude to let an agent speak on your behalf _without even reviewing it_.
This will eventually lead to a bigger snafu, of course. Security teams should control or at least review the agent permissions of every installation. Everyone is adopting this stuff, and a whole lot of people are going to set it up lazily/wrong (yolo mode at work).
Yeah, a nothingburger for clicks.
AI can be used to move fast. So management expects us to move at that speed. AI can be used to move even faster if you don't check it's output. The ever ratcheting demand for faster output will make it infeasible to diligently check AI output all the time. AI errors being acted on without due care is inevitable.
From Schlock Mercenary: "Oh, I love aiming. It's my very favorite thing to do before firing."
AI use without checking its output (at least at the moment) is firing without aiming. Sure, you can fire really fast. But who cares if you don't hit what you need to? The point wasn't to just shoot bullets, the point was to hit your target!
I mean, you might make a case that enough of them hit the target that shooting fast is a net win, and accept the occasional friendly fire incident. That might possibly be true. Or it might not. I'm not sure that everyone trying to run fast has really done the calculation, though.
"A human, however, might have done further testing and made a more complete judgment call before sharing the information"
Because a human would have been fired for posting something that incorrect and dangerous
But funny enough the person who was responsible for setting up the bot will likely face no repercussions. In fact they will probably be rewarded for transitioning their team's workflows to AI.
I mean, only if it leads to embarrassment right off the bat.
If there is a year or two between writing your security fuck up and it being discovered the likelihood of repercussions drops significantly.
A machine doesn’t need food, leisure time, or vacations. It doesn’t care.
It also doesn’t care.
I'm concerned that someone had the permissions to make such a change without the knowledge of how to make the change.
And there was no test environment to validate the change before it was made.
Multiple process & mechanism failures, regardless of where the bad advice came from.
If you have to do all that, then what's the point of the AI? I'm joking, but I'm afraid many others say the same thing 100% seriously
As an article that was here recently claims, every verification you do in a chain increases the total time of your work by an order of magnitude. So, it's only work optimizing any productive task if you already removed most verifications.
Now, some people claim that you need to improve the reliability of your productive tasks so you can remove the verifications and be faster. Those people are, of course, a bunch of coward Luddites.
A central challenge for AI is understanding how accountability flows.
The language of this article is a great example, "... thanks to an AI agent that gave an employee inaccurate technical advice ...".
It should more-correctly read, " ... thanks to the people who made it possible for an AI agent to give an employee inaccurate technical advice ... ".
It is at our peril that we deem it acceptable to blame a black box for an error, especially at scale.
> "Had the engineer that acted on that known better, or did other checks, this would have been avoided."
<insert takes long drag tweet[1] here>
I personally find "LLMs can do $THING poorly" and "LLMs can do $THING well" articles kinda boring at this point. But! I'm hopeful that stories like this will shift the industry's focus towards robustness instead of just short-term efficiency. I suspect many decision making and change management processes accidentally benefited from just being a bit slow.
[1] https://waffles.fun/amy.png
The two errors, then, were that the LLM hallucinated something, and that a human trusted the LLM without reasoning about its answer. The fix for this common pattern is to reason about LLM outputs before making use of them.
A big problem now both internally to a company and externally is that official support channels are being replaced by chatbots, and you really have no option but to trust their output because a human expert is no longer available.
If I post a question to the internal payment team's forum about a critical processing issue and some "payments bot" replies to me, should I be at fault for trusting the answer?
I know this is happening with external customer support, but is this really happening internally at big companies? Preventing you from talking to a human in the correct department about an issue feels like a bomb waiting to explode.
There is at least an effect that chatbots have become the primary line and support, and even if you are not prevented from talking to a human, the managers of the humans you would talk to have decided that since the chatbot is there, it is inappropriate for them to be spending much time supporting coworkers in other departments when the chatbot can do it.
So to a degree, corporate politics can sort of discourage it.
I'm sure it is. Thankfully I don't work for a company this large any more, but when I was employed by a multinational with 30K+ employees, our IT department was outsourced to India and you had to get through a couple layers of phone tree/webchat hell to actually talk to a real person. I could easily see companies of this size replacing their support with LLM nonsense.
Teams are heavily incentivized to incorporate AI in their internal workflows. At Meta it is a requirement, and will come up in your performance review if you fail to do so.
Yes, of course, and the company which removes human experts should expect things to fail in the manner that things usually fail when you remove your internal experts.
> The fix for this common pattern is to reason about LLM outputs before making use of them.
That is politics. Not engineering.
Assigning a human to "check the output every time" and blaming them for the faults in the output is just assigning a scapegoat.
If you have to check the AI output every single time, the AI is pointless. You can just check immediately.
Well, I'd say there's two dimensions:
1. Check frequency (between every single time and spot checks).
2. Check thoroughness (between antagonistic in-depth vs high level).
I'd agree that, if you're towards the end of both dimensions, the system is not generating any value.
A lot of folks are taking calculated (or I guess in some cases, reckless) risks right now, by moving one or both of those dimensions. I'd argue that in many situations, the risk is small and worth it. In many others, not so much.
We'll see how it goes, I suppose.
Groooooooaaaaaaaaaaaannnnnnnnnnn
The humans are not scapegoats, because they are capable of taking on responsibility.
There is a point to using LLMs. They can save time by doing a first pass. But when they do the last pass, disasters will follow.
Well, attempts to engineer the brittleness out of human behavior have not worked, like, ever.
However - Automation bias is a common problem (predating AI), the 'human-in-the-loop' ends up implicitly trusting the automated system.
At least pre-LLM automation was written by a careful human who's job was on the line, and was deterministic.
If "the level of awareness that created a problem, cannot be used to fix the problem", then you're asking too much if you expect a human to reason about an LLM output when they are the ones that asked an LLM to do the thinking for them to begin with.
This feels like a rediscovering/rewording of Kernighan's Law:
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." ~ Brian Kernighan
It's an old saying, I think Einstein is cited most often for it... something like this according to Google:
"We cannot solve our problems with the same thinking we used when we created them."
In this case you would replace the human.
Yes, I'd fire them, and then hire a more competent human.
I'm pretty happy with the team I've built. They make solid decisions that I can trust every time. I can't say the same for the LLM.
It's more like, the LLM "hallucinated" (I hate that term) and automatically posted the information to the forum. It sounds like the human didn't get a chance to reason about it. At least not the original human that asked the LLM for an answer
I’m not in AI, but what is happening is that it is building output from the long tail of its training data? Instead of branching down the more common probability paths, something in this interaction had it travel into the data wilderness?
So I asked AI to give it a good name, and it said “statistical wandering” or “logical improv”.
If you don't like hallucinate, try bullshit. [NB: bullshit is a technical term; see https://en.wikipedia.org/wiki/On_Bullshit]
https://www.psypost.org/scholars-ai-isnt-hallucinating-its-b...
That is my preferred term, but it seems to derail discussions that might have otherwise been productive (might...the hope I have)
When organizational incentives penalize NOT using AI and firing the bottom x% regularly then are you really surprised LLM outputs aren't being scrutinized?
Yes, because trusting LLM output is a great way to be in the bottom x%.
I’m predicting a wave of such incidents to start appearing over the next few months/years.
How long until an AI puts all our personal data on the streets?
It's already there for a dollar to the right data broker. Could probably pull your doctor visit info from last week (example).
Very soon, and at this point I'm not sure even that would cure the delusions of the few who practically worship LLMs
Skill issue.
More like Rogue Human, who didn't check the facts before taking the technical advice from the model at face value.
“Meta spokesperson Tracy Clayton said in a statement to The Verge that ‘no user data was mishandled’ during the incident.”
Wow, no mishandled user data? A striking change of standard operating procedure from Meta here.
Actually the later information in the story directly contradicts that, so The Verge probably shouldn’t have just quoted this line if their reporting is in opposition to it.
Regardless, this is one of the more insidious things about these tools. They often get minor but critical things wrong in the midst of mostly correct information. And people think they can analyze the data presented to them and make logical judgments, but that’s just not the case.
The article points out that “a human could have done the same thing” but, between the overly confident tone of the text generated by these tools, and the fact that weirdly people trust the LLM output more than they trust other humans (who generally admit or at least hint when they aren’t actually experts on a topic), it’s actually far worse when one of these bots gets something wrong.
> A rogue AI led to a serious security incident at Meta
The AI "led to" the incident , true. But do nt forget that this, like all similar incidents , is a human failure
AI is a tool with no agency. People make mistakes using it, thone mistakes are the responsibility of the humans
Why do we keep calling these things "agents" then? Or using the term "agentic"?
Eternal optimism
> AI is a tool with no agency
Claw AIs absolutely do have agency in the sense of being able to independently perform actions on their own, based on their "understanding" of a goal given by a "principal". I can't think of a better word than "agent" for that.
They appear to have agency. But watch a while, they do not
I've been carefully watching for more than a year, including 3 productive parallel Claude Code sessions today, and they absolutely have agency in any definition of the word I can think of.
Can you try to define more clearly what it is that you believe AI agents don't have?
Much like planes, people still would have a human pilot err and take the whole thing down than a robot go down even if it is statistically sound.
Then the human should write the code.
fds
test
very misaligned! sprays bottle at mac mini
Behind paywall, is there another link to the article?
I hit back, clicked the link again, and it let me through
https://archive.is/A2hmz
This link isn't working for me? Is this working for someone else?
Can you perhaps share a archive.org link if possible?
"A rogue AI led to a serious security incident" is certainly a way to write "Someone vibe coded too hard and leaked data".
Read TFA. It's not "Someone vibe coded too hard and leaked data"