I have been working on a monitoring and prebunking system for digital manipulation and desinformation. We are focusing not on the content or narrative but on the psychological patterns and manipulation techniques that are used.
It's the most disturbing thing I have ever worked on, there is much more out there than moste people realize and a lot of it uses deceptive dark patterns.
If somebody is interested in talking more about this or is working on similar things, always welcome!
OK, so to start with you're saying that there's a small noisy pro- side, and a small noisy anti- side, and a moderate majority. But then suddenly:
> The Majority Goes Silent - When the majority of people looks at the feed and assumes they're outnumbered, people will often self-censor.
That's not the same thing, is it? Here the majority is, say, anti-, but they are being frightened by a noisy pro- minority. They're moderates in the sense that anti- is the conventional position to take. But they have opinions. (They could also be in the minority, and this fear of speaking up would still be a bad thing.)
Otherwise, if they're truly moderate, but are frightened into silence supposedly, what would they be saying if they dared? "Everybody listen to me, I have no strong opinion on this matter"?
This is amazing analysis, presentation, and has a call to action at the end. Some of this guys other stuff: https://tobias.cc/reading
The only point I'd add is that it's not handling time evolution in wicked problems quite right. Agree that the noisy room is distorting the world in exactly the ways described. But what if we've been in there so long, and the world has become so distorted.. that reality itself slides towards the once-extreme positions? Easiest to see this with climate-change controversy since that is the way that sort of thing happens, regardless of whether you think it's happened yet. Cascade, phase change, and collapse don't just call a truce.
So you have to anticipate that, acknowledging the pessimist is actually right, and that systems are a real bitch. Then you point out that if we're already doomed, we have nothing to lose nothing by trying. Systems are complex after all, that's the whole problem.. so if we miscalculated on the doom, then bothering to try actually saves us. Checkmate pessimists.
Regulating the algorithm is my favorite answer. Ban the recommendation engine on social media sites with > X people. Make it a chronological feed of who you follow. Make it boring. I don't know all the details, but something like this.
I agree, recommendation algorithms are a huge part of the problem. Consciously choosing what you interact with is a very important part of media consumption IMO and most social media sites give you very little tools to do that (no, having likes/dislikes affect your personalised feed is not enough, especially when that also becomes 'engagement' and boosts it everyone's feed in general). These algorithms should be dumber in all areas except spam prevention (and even then, if there's less stuff in your feed you didn't specifically choose to see, spam should be a much smaller problem anyway).
This is my question as well, especially about the "community check". How will it be ensured that the "community check" is not going to be dominated by bots pushing an agenda? How is that different from "just another comment section hidden behind a green button"?
This article is awesome but it doesn't acknowledge that the problem has been maliciously manufactured by social media companies. they do not have incentive to curb the distortion of extremism and therefore any attempt to do so in a grassroots way will likely not be effective. then there's the bot problem but that is probably easier to address if we actually committed to doing so.
The part that annoys me about the toxicity, or repetetive and annoying topics on reddit, HN, etc. is not that I am unaware that the content is produced by a small fraction. (I underestimated the count! I guessed 2%)
It's that people espouse it: They upvote and retweet it.
> Both sides develop wildly inaccurate beliefs about who the other side actually is.
That was a guess I had for a while. People have a strawman version of their out-groups in mind and quickly map people to that if an unknown person says something that indicates they might be part of the out-group.
> What percentage of the other side supports political violence?
It would be interesting to see the in-group statistic as well: "What percentage of your own side supports policical violence?", in my experience people also justify very shitty behavior as long as its from their in-group. (This plays heavily into the first point of espousing all kinds of shit)
---
It would be interesting to see if the community check actually changes anything. But the actual data seems to be only possibly for very generic topics - those we have the data on already. Something that would not be available for daily-fresh topics.
For my personal sanity I simply left reddit and stopped opening comments on certain HN posts - of course that does not help with the societal problems. Unfortunately.
New social trends and technologies frequently cause some level of moral panic. Moral panics of the past have been caused by all sorts of things, that now seem rather quaint: novels, bicycles, comics, television, videos, heavy metal, dungeons and dragons etc. But social media feels very different. It really does seem to be causing major societal disruption.
The current social panic always feels very different. But people literally believe social media is the sole cause of all of modern society's problems, that it's a mind-control platform and a cancer on society. I've seen people say they would welcome a fascist dictatorship if only it meant destroying social media. I've seen people say they want "algorithms" made illegal.
It's obvious from the hyperbole around the discourse alone that this moral panic has reached levels of derangement that far outclass any rational basis for judgement.
Does social media have negative consequences? Sure. Are people assholes on the internet? Always have been. Is social media the greatest and most existentially perilous evil ever conceived by humankind? No.
I think in ten years people will look back at this (on whatever strictly censored and regulated internet replaces this one) with the same bemused confusion as we do the Satanic Panic. And honestly in forty years, if technological civilization still exists, we'll find out how much of that was stoked by the CIA or other interests.
if you strip social media down to its essential parts it's clear that it can easily cause huge problems for a society. it's basically a never-ending 24/7 stream of information amplified out to anywhere in the world that is:
1. insanely low-effort to post
2. requires NO discernment, proof, credibility, or peer review to post
3. 'viral' in that opinions circulate because other people have interacted with them, not because they are right or meaningful. so bad news, good news, real news and fake news all travel at the same speed, lowering discernment even further
4. echo chambers are baked into the form. people are more likely to interact with content they agree with vs. content that is true or impactful. this creates circles of people agreeing with each other on increasingly niched-down topics.
it is extremely different from newspapers and television.
You: Other people are unhinged hyperbolists. Here, let me characterize them hyperbolically to prove it, therefore I am the calm rational one and by the way, civilization may collapse and the CIA might be behind this.
Every bit of hyperbole I mentioned is practically quoted verbatim from some thread or another here, it is what people believe, and you can't even bring yourself to approach me in good faith because I've committed wrongthink by defending the existence of social media even implicitly.
The CIA and other governments are running influence campaigns across social media. The links between the major social media platforms and intelligence agencies are well known and well documented. And civilization is threatened by numerous factors, such as our over-investment in AI and the mass deskilling and destabilization that will create, creeping fascism and increasing political violence in a multipolar world, climate change leading to mass famine, pandemics in a post-scientific age, etc.
But people want to destroy social media (and by extension, want to destroy the freedom of communication it allows) rather than bother to consider that the real problem is the same problem we've always had - government and corporate interests trying to control our lives and manufacture consent through fear and panic.
They ran the same playbook prior to social media but the process was so normalized because they controlled so much of the media and culture that no one really even noticed it. Now people notice but they can't distinguish between the symptom and the disease.
I was on social media since sharing Zynga game invites was majority of the posts. I've seen countless of magic bullets attempting to fix the polarization issues. Algorithm adjustments, fact checkers, community notes.
I feel like the real problem is the people. Many of us just want to be told what to think to blend in with society, some of us demonstrate Dunning-Kruger publicly and a few of us really want to drive the polarization for clout and attention.
Everyday I see people promote increasingly stupid ideas on both sides, further pushing my believe that the only solution is to severely limit what government can do, therefore making all this discussion pointless.
Fantastic presentation.
Unfortunately the conclusion is painfully naive and forgives the platforms too much.
>We Could Do This Now - Platforms already have a lot of these capabilities. They already survey users. They even know how to run sophisticated polls. There are a few technical details to work out (spec here), but this is not a hard problem to solve.
Why do you think something like this is not already implemented?
Platforms literally profit from this division, so why would they be incentivised to do anything?
What's needed is not a good gesture from the overly powerful platforms, is fast, hard and deep regulation.
The claim that this isn't a hard problem to solve seems very optimistic to me.
The tiny minority dominates the feeds because that's how the incentives for algorithmic driven social media are structured. Do we really expect Meta, X, TikTok to anything that could reduce engagement?
Good luck having any of the mainstream social media apps add the banner they propose.
Great article format with all the dynamic widgets in it. Will have to give this a good read. It is a very interesting topic given how much of (global) public opinion is formed through "social" media.
The magnitude of that number is a consequence of the effects being discussed in the post. And unless you find a way to solve the tyranny of the loudest, it's only going to continue to increase.
I have been working on a monitoring and prebunking system for digital manipulation and desinformation. We are focusing not on the content or narrative but on the psychological patterns and manipulation techniques that are used.
It's the most disturbing thing I have ever worked on, there is much more out there than moste people realize and a lot of it uses deceptive dark patterns.
If somebody is interested in talking more about this or is working on similar things, always welcome!
OK, so to start with you're saying that there's a small noisy pro- side, and a small noisy anti- side, and a moderate majority. But then suddenly:
> The Majority Goes Silent - When the majority of people looks at the feed and assumes they're outnumbered, people will often self-censor.
That's not the same thing, is it? Here the majority is, say, anti-, but they are being frightened by a noisy pro- minority. They're moderates in the sense that anti- is the conventional position to take. But they have opinions. (They could also be in the minority, and this fear of speaking up would still be a bad thing.)
Otherwise, if they're truly moderate, but are frightened into silence supposedly, what would they be saying if they dared? "Everybody listen to me, I have no strong opinion on this matter"?
This is amazing analysis, presentation, and has a call to action at the end. Some of this guys other stuff: https://tobias.cc/reading
The only point I'd add is that it's not handling time evolution in wicked problems quite right. Agree that the noisy room is distorting the world in exactly the ways described. But what if we've been in there so long, and the world has become so distorted.. that reality itself slides towards the once-extreme positions? Easiest to see this with climate-change controversy since that is the way that sort of thing happens, regardless of whether you think it's happened yet. Cascade, phase change, and collapse don't just call a truce.
So you have to anticipate that, acknowledging the pessimist is actually right, and that systems are a real bitch. Then you point out that if we're already doomed, we have nothing to lose nothing by trying. Systems are complex after all, that's the whole problem.. so if we miscalculated on the doom, then bothering to try actually saves us. Checkmate pessimists.
This seems like a great idea. Even without the linked surveys. Two questions I have:
- how you does this handle the fact that a lot of accounts on social media platforms are bots that maybe controlled by a small number of people.
- how do we actually get this implemented?
Regulating the algorithm is my favorite answer. Ban the recommendation engine on social media sites with > X people. Make it a chronological feed of who you follow. Make it boring. I don't know all the details, but something like this.
I agree, recommendation algorithms are a huge part of the problem. Consciously choosing what you interact with is a very important part of media consumption IMO and most social media sites give you very little tools to do that (no, having likes/dislikes affect your personalised feed is not enough, especially when that also becomes 'engagement' and boosts it everyone's feed in general). These algorithms should be dumber in all areas except spam prevention (and even then, if there's less stuff in your feed you didn't specifically choose to see, spam should be a much smaller problem anyway).
This is my question as well, especially about the "community check". How will it be ensured that the "community check" is not going to be dominated by bots pushing an agenda? How is that different from "just another comment section hidden behind a green button"?
This article is awesome but it doesn't acknowledge that the problem has been maliciously manufactured by social media companies. they do not have incentive to curb the distortion of extremism and therefore any attempt to do so in a grassroots way will likely not be effective. then there's the bot problem but that is probably easier to address if we actually committed to doing so.
> toxic tweets receive ~86% more retweets
The part that annoys me about the toxicity, or repetetive and annoying topics on reddit, HN, etc. is not that I am unaware that the content is produced by a small fraction. (I underestimated the count! I guessed 2%)
It's that people espouse it: They upvote and retweet it.
> Both sides develop wildly inaccurate beliefs about who the other side actually is.
That was a guess I had for a while. People have a strawman version of their out-groups in mind and quickly map people to that if an unknown person says something that indicates they might be part of the out-group.
> What percentage of the other side supports political violence?
It would be interesting to see the in-group statistic as well: "What percentage of your own side supports policical violence?", in my experience people also justify very shitty behavior as long as its from their in-group. (This plays heavily into the first point of espousing all kinds of shit)
---
It would be interesting to see if the community check actually changes anything. But the actual data seems to be only possibly for very generic topics - those we have the data on already. Something that would not be available for daily-fresh topics.
For my personal sanity I simply left reddit and stopped opening comments on certain HN posts - of course that does not help with the societal problems. Unfortunately.
New social trends and technologies frequently cause some level of moral panic. Moral panics of the past have been caused by all sorts of things, that now seem rather quaint: novels, bicycles, comics, television, videos, heavy metal, dungeons and dragons etc. But social media feels very different. It really does seem to be causing major societal disruption.
The current social panic always feels very different. But people literally believe social media is the sole cause of all of modern society's problems, that it's a mind-control platform and a cancer on society. I've seen people say they would welcome a fascist dictatorship if only it meant destroying social media. I've seen people say they want "algorithms" made illegal.
It's obvious from the hyperbole around the discourse alone that this moral panic has reached levels of derangement that far outclass any rational basis for judgement.
Does social media have negative consequences? Sure. Are people assholes on the internet? Always have been. Is social media the greatest and most existentially perilous evil ever conceived by humankind? No.
I think in ten years people will look back at this (on whatever strictly censored and regulated internet replaces this one) with the same bemused confusion as we do the Satanic Panic. And honestly in forty years, if technological civilization still exists, we'll find out how much of that was stoked by the CIA or other interests.
if you strip social media down to its essential parts it's clear that it can easily cause huge problems for a society. it's basically a never-ending 24/7 stream of information amplified out to anywhere in the world that is:
1. insanely low-effort to post 2. requires NO discernment, proof, credibility, or peer review to post 3. 'viral' in that opinions circulate because other people have interacted with them, not because they are right or meaningful. so bad news, good news, real news and fake news all travel at the same speed, lowering discernment even further 4. echo chambers are baked into the form. people are more likely to interact with content they agree with vs. content that is true or impactful. this creates circles of people agreeing with each other on increasingly niched-down topics.
it is extremely different from newspapers and television.
You: Other people are unhinged hyperbolists. Here, let me characterize them hyperbolically to prove it, therefore I am the calm rational one and by the way, civilization may collapse and the CIA might be behind this.
I mean wtf. Is this your parody account?
You see, you're doing the thing.
Every bit of hyperbole I mentioned is practically quoted verbatim from some thread or another here, it is what people believe, and you can't even bring yourself to approach me in good faith because I've committed wrongthink by defending the existence of social media even implicitly.
The CIA and other governments are running influence campaigns across social media. The links between the major social media platforms and intelligence agencies are well known and well documented. And civilization is threatened by numerous factors, such as our over-investment in AI and the mass deskilling and destabilization that will create, creeping fascism and increasing political violence in a multipolar world, climate change leading to mass famine, pandemics in a post-scientific age, etc.
But people want to destroy social media (and by extension, want to destroy the freedom of communication it allows) rather than bother to consider that the real problem is the same problem we've always had - government and corporate interests trying to control our lives and manufacture consent through fear and panic.
They ran the same playbook prior to social media but the process was so normalized because they controlled so much of the media and culture that no one really even noticed it. Now people notice but they can't distinguish between the symptom and the disease.
Case closed.
There's money in politics and money in social media.
And the money decides how to run the circus. Not for the benefit of all.
So it is a really hard problem.
I was on social media since sharing Zynga game invites was majority of the posts. I've seen countless of magic bullets attempting to fix the polarization issues. Algorithm adjustments, fact checkers, community notes.
I feel like the real problem is the people. Many of us just want to be told what to think to blend in with society, some of us demonstrate Dunning-Kruger publicly and a few of us really want to drive the polarization for clout and attention.
Everyday I see people promote increasingly stupid ideas on both sides, further pushing my believe that the only solution is to severely limit what government can do, therefore making all this discussion pointless.
Fantastic presentation. Unfortunately the conclusion is painfully naive and forgives the platforms too much.
>We Could Do This Now - Platforms already have a lot of these capabilities. They already survey users. They even know how to run sophisticated polls. There are a few technical details to work out (spec here), but this is not a hard problem to solve.
Why do you think something like this is not already implemented? Platforms literally profit from this division, so why would they be incentivised to do anything? What's needed is not a good gesture from the overly powerful platforms, is fast, hard and deep regulation.
The claim that this isn't a hard problem to solve seems very optimistic to me.
The tiny minority dominates the feeds because that's how the incentives for algorithmic driven social media are structured. Do we really expect Meta, X, TikTok to anything that could reduce engagement?
Good luck having any of the mainstream social media apps add the banner they propose.
“The nuts are always the loudest” has been an observation forever.
This is showing how in the social media system the dynamics play out.
Great article format with all the dynamic widgets in it. Will have to give this a good read. It is a very interesting topic given how much of (global) public opinion is formed through "social" media.
"What percentage of the other side supports political violence"
Both Democrats and Republicans estimated 30% but actually.. only 10% of both sides supported political violence
That number is crazy in so many ways and the post is overly nonchalant about it. The "distortion" isn't what's worrying here
The magnitude of that number is a consequence of the effects being discussed in the post. And unless you find a way to solve the tyranny of the loudest, it's only going to continue to increase.
I agree with the notion in the post, though I do not agree that users will feel the format is not being pushed top-down by the man
I just had an issue with the way that number was completely overlooked