I would be curious if this can do moderation with an arbitrary ruleset, or if it's just "that one moderation style" we already know from current big tech platforms.
The kind where malicious intent is okay if the words are nice.
___
Or, rephrased: How big is the space in which you can tune this model without retraining.
Is it just "we hate sex"/"we don't hate sex" "We hate violence"/"we don't hate violence" or is it _truly_ as flexible as claimed?
__
Maybe something like "Is this guy a corporate fraud that is going to waste my time with performative nonsense?"
That would be the true test for a moderation model and I would be immensely impressed if it could manage to pull that off.
___
Edit:
Looking at the paper though.. probably not.
I suppose this is useful for B2B, which seems to be mistrals whole thing. Question is just if it is also useful for society to hand the SV prefab morals down like that. Kinda like cultural imperialism but with an ethical spin.
Maybe opinions on those base datasets could occasionally differ more than the model can be steered.
Having grown a large healthcare review platform, I can attest to the success we had mapping specific policy violations to natural language is incredibly useful. At scale, patients having terrible situations and/days can write about in ways that can be deeply unhealthy for the community or the doctors reading/receiving the feedback and sometimes very threatening beyond that purposes for the community. We built a custom ML engine to handle our levels of traffic for reviews, which was among the largest in the US typical ranking top 3 on Google for the domain keywords. Back when BERT was the edge, a policy-adaptive model like this one from Mistral would have been an incredible cold-start solution. Most sites never have the massive volume nor budget needed nor skillset needed before you can train domain-specific models that outperform OOTB solutions. Generally, most people and site mean well and try to do well, so empowering those people with models like this can help the collective in my opinion, so I’m happy to see this released in this manner myself.
A bit of editorial and cultural note from a US native, the subsection of the original article “Teach discrimination, not memorization” is better worded as something like 'Differentiation' or 'Distinction'. In English, the word 'discrimination' can (and in this social context may) imply social prejudice or unfair treatment. I think this may have been a bit of carry over from the rather benign French translation of “Enseigner la discrimination" which I also see awkwardly translated in the paper as well.
They got a lot of hate for not keeping up with frontier model releases, but have managed to carve out a nice business that isn't even really niche.
Before the datacenter deals their revenue was higher than xAI's
There is a whole world out there of purpose built and hosted task specific vertical llms - especially with an emphasis on cost.
Mistral, Microsoft model releases and Thinking Machines are all over this, and it's smart. Scoop up all the tasks that don't require large and expensive frontier general-purpose llms.
> Question is just if it is also useful for society to hand the SV prefab morals down like that. Kinda like cultural imperialism but with an ethical spin.
Isn't Mistral a French company? Not that the French can't do cultural imperialism either, but they (the French) don't strike me as very SV.
It sounds like it is. You have a set of moderation policies and then you evaluate the model 1 time per policy if it is violating it. Then you combine the results into a score you use for taking actions off of.
Also I do like Mistral's seemingly newer strategy of focusing on smaller, more fine-tuned models for various use-cases, presumably the result of their large MoE models not competing effectively with the frontier models.
It's not that their strategy is to train smaller models, it's the only choice they have. Training SOTA takes anywhere from 1.5b to 150b. We don't know the real cost of training for the chinese models, but mistral neither has the compute nor money to do that.
Mistral has the capability of training such models. Take a look at Poolside[1], they are claiming to pre-train their Laguna series of models on 4,096 NVIDIA H200 GPUs[2].
Mistral has approximately 13,800 NVIDIA GB300 GPUs, which are nearly 2x more efficient for training.
The problem with Mistral is that they do not seem to have aligned incentives to train big open-weight models, even if the teams would like to.
As for use cases, obviously we can't fully rely on non-deterministic capability for sensitive things but a small model which can do a good job acts as a first defense and then a human can review later.
I've had dreams of building something in the image sharing or social platform realm, but stopped short of planning because of obvious content moderation responsibilities. This seems to be a realistic, cost effective solution to that one piece of the puzzle.
I am not sure how reliable it is in the real world. Also, in terms of liability, I don’t know how effective it would be to satisfy various regulations compared to a human moderator team.
I hear ya, but one could set different operating thresholds: auto-approve low-risk posts, hold ambiguous posts for review, and automatically reject very high-confidence violations. So HITL for sure, but MUCH less H in the L.
This model is way small for a proper assessment (imo). It should be very useful to study how big the real model must be for this purpose. Maybe merging it to a bigger one (adding it as expert style in moe) would be a solution!
Great job to Mistral team.
By this logic the Chinese should have just given up and let the American AI companies have the market because they were so far behind. I'm sure Europe has the capability to distill other people's frontier models to catch up if they wish to do so.
Distilling is unsafe from export control perspective - Chinese models are poisoned by US frontier distillation and a case can be made that the US won’t like distilling what they may consider transitively theirs, which they will the moment you’re anywhere near competitive.
US judges have already rules that output of an LLM can't be copyrighted so not sure what would prevent Chinese companies to use said output for distillation purposes.
By which other mechanism could American AI companies prevent this? Other companies don’t really care about EULAs and even if they needed to care it’s trivial to let third parties do it. Why would they? Almost nobody in the space cares about copyright and play fast and loose with laws and regulations.
What’s the mechanism that could today prevent other companies from using LLM outputs to train their models?
> already rules that output of an LLM can't be copyrighted
Mind sharing such cases? I'm not aware of any so far. There's the one with images, but that's commonly miss-understood, that case was ruled on a technicality (i.e. copyright needs to be attributed to a person, not a model)
> Distilling is unsafe from export control perspective
That is not the direction American judges are taking. Right now, they are saying that LLM output cannot be copyrighted. And if looting copyrighted works for training is fair game, I really don’t see how one could argue that learning from other LLMs is not.
Their web chat UI seems to be called “Vibe.” I think? I’m not sure if that’s the name of the product or just what they decided to label it in the browser. I wonder if -strap is just what they call the actual models, which are meant to be run “under the hood” anyway, so not really part of the branding.
But I wish they could commit to the bit fully and call everything -stral. It’s quirky and self aware to give your products silly names.
Someone should use this to do the exact opposite of the intention: filter for “offensive” content, and boost it or collate it into a newsletter/email blast for people of culture.
You have to give it to Mistral they do at least know what the market near them says they want right now. The great problem is in a few years of this that market won’t be worth anything.
Edit to add, you could also add this to an AI workflow so as to produce content that walks right up to the line but doesn’t trigger it.
> The great problem is in a few years of this that market won’t be worth anything.
To be fair, we don’t know how much resources they put into this and how much of a distraction it was. If it was quick enough to train or fine tune and it brings them valuable experience for the next models, it could well be worth it in the long run even if there is no direct successor.
Also, it sounds like the kind of thing that sells. Any company with a customer support chat is a potential user of this model, any large company may be interested in getting a mistral installed set-up for handling without needing to send client info over the web. Installing those kinds of local systems seems to be what butters Mistral's bread at the moment.
this just sounds like new age edgey mc lordy lord. I get censorship is bad but ya'll might want to concern yourselves with a bid less fascism before you worry about the bad words police.
> they do at least know what the market near them says they want right now
It does seem to be a very European approach to AI that their flagship AI lab is just making models that do nothing other than monitor and moderate internet content.
I guess they know that the EU AI Act, Chat Control, etc are going to cause a lot of companies to need this kind of compliance.
Some of the best social media is heavily moderate. This includes HN and r/credible defense . With a Quiet transparent and cheap LLM I imagine a social media website where you can have good discussion about everything around the world it would be a game changer and on my to-do list.
The bulk of moderation work is things which are easy and obvious.
The correct way to moderate is automation with certainty falling back to humans with discretion.
The new frontier of moderation should be blocking illiterate comments, as in the commenter is replying as though they didn't read or read and didn't understand.
> It does seem to be a very European approach to AI that their flagship AI lab is just making models that do nothing other than monitor and moderate internet content.
Well, first Mistral is French more than European. This might be a difficult distinction to make from the US but their approach is quite different from e.g. typical German companies.
Then, this is just a small model they release on the side. If that’s your benchmark, they released somewhat recently Voxtral, Voxtral transcribe, their OCR model, and Leanstral. I don’t think you can get much insight on their culture from this kind of release.
Ugh. There's nothing inherently European about Chat Control. It's a dumb proposal, and it's European. Any free society has a bunch of dumb proposals.
Nor is there anything inherently European about the AI Act. But that one I wouldn't even call dumb. At times misguided and confused, perhaps, but some of its core principles are valuable.
I would be curious if this can do moderation with an arbitrary ruleset, or if it's just "that one moderation style" we already know from current big tech platforms.
The kind where malicious intent is okay if the words are nice.
___
Or, rephrased: How big is the space in which you can tune this model without retraining.
Is it just "we hate sex"/"we don't hate sex" "We hate violence"/"we don't hate violence" or is it _truly_ as flexible as claimed?
__
Maybe something like "Is this guy a corporate fraud that is going to waste my time with performative nonsense?"
That would be the true test for a moderation model and I would be immensely impressed if it could manage to pull that off.
___
Edit: Looking at the paper though.. probably not.
I suppose this is useful for B2B, which seems to be mistrals whole thing. Question is just if it is also useful for society to hand the SV prefab morals down like that. Kinda like cultural imperialism but with an ethical spin.
Maybe opinions on those base datasets could occasionally differ more than the model can be steered.
Having grown a large healthcare review platform, I can attest to the success we had mapping specific policy violations to natural language is incredibly useful. At scale, patients having terrible situations and/days can write about in ways that can be deeply unhealthy for the community or the doctors reading/receiving the feedback and sometimes very threatening beyond that purposes for the community. We built a custom ML engine to handle our levels of traffic for reviews, which was among the largest in the US typical ranking top 3 on Google for the domain keywords. Back when BERT was the edge, a policy-adaptive model like this one from Mistral would have been an incredible cold-start solution. Most sites never have the massive volume nor budget needed nor skillset needed before you can train domain-specific models that outperform OOTB solutions. Generally, most people and site mean well and try to do well, so empowering those people with models like this can help the collective in my opinion, so I’m happy to see this released in this manner myself.
A bit of editorial and cultural note from a US native, the subsection of the original article “Teach discrimination, not memorization” is better worded as something like 'Differentiation' or 'Distinction'. In English, the word 'discrimination' can (and in this social context may) imply social prejudice or unfair treatment. I think this may have been a bit of carry over from the rather benign French translation of “Enseigner la discrimination" which I also see awkwardly translated in the paper as well.
> which seems to be mistrals whole thing
They got a lot of hate for not keeping up with frontier model releases, but have managed to carve out a nice business that isn't even really niche.
Before the datacenter deals their revenue was higher than xAI's
There is a whole world out there of purpose built and hosted task specific vertical llms - especially with an emphasis on cost.
Mistral, Microsoft model releases and Thinking Machines are all over this, and it's smart. Scoop up all the tasks that don't require large and expensive frontier general-purpose llms.
> Question is just if it is also useful for society to hand the SV prefab morals down like that. Kinda like cultural imperialism but with an ethical spin.
Isn't Mistral a French company? Not that the French can't do cultural imperialism either, but they (the French) don't strike me as very SV.
It sounds like it is. You have a set of moderation policies and then you evaluate the model 1 time per policy if it is violating it. Then you combine the results into a score you use for taking actions off of.
Should've called it Safestral.
Also I do like Mistral's seemingly newer strategy of focusing on smaller, more fine-tuned models for various use-cases, presumably the result of their large MoE models not competing effectively with the frontier models.
It's not that their strategy is to train smaller models, it's the only choice they have. Training SOTA takes anywhere from 1.5b to 150b. We don't know the real cost of training for the chinese models, but mistral neither has the compute nor money to do that.
Mistral has the capability of training such models. Take a look at Poolside[1], they are claiming to pre-train their Laguna series of models on 4,096 NVIDIA H200 GPUs[2]. Mistral has approximately 13,800 NVIDIA GB300 GPUs, which are nearly 2x more efficient for training.
The problem with Mistral is that they do not seem to have aligned incentives to train big open-weight models, even if the teams would like to.
[1]: https://poolside.ai/ [2]: https://poolside.ai/blog/introducing-laguna-s-2-1
Do you have a reference explaining these costs ? Part by part.
What, you don't want a model called the Shitstral-3B :D
Oh so THAT'S why the web app is so slow
Model: https://huggingface.co/mistralai/Shieldstral-1.0-3B
how does this compare with https://developers.openai.com/api/docs/models/omni-moderatio...
As for use cases, obviously we can't fully rely on non-deterministic capability for sensitive things but a small model which can do a good job acts as a first defense and then a human can review later.
I've had dreams of building something in the image sharing or social platform realm, but stopped short of planning because of obvious content moderation responsibilities. This seems to be a realistic, cost effective solution to that one piece of the puzzle.
I am not sure how reliable it is in the real world. Also, in terms of liability, I don’t know how effective it would be to satisfy various regulations compared to a human moderator team.
I hear ya, but one could set different operating thresholds: auto-approve low-risk posts, hold ambiguous posts for review, and automatically reject very high-confidence violations. So HITL for sure, but MUCH less H in the L.
This model is way small for a proper assessment (imo). It should be very useful to study how big the real model must be for this purpose. Maybe merging it to a bigger one (adding it as expert style in moe) would be a solution! Great job to Mistral team.
Thanks Mistral !
Finally an AI company besides DeepSeek taking economics into account.
I'd really like to see more conversation around Mistral's models. It's good to see Europe developing AI.
The problem is that their performance is too far away from the latest generation of Asian models.
They had kept up in the mid-range a few years ago. But this standing is sadly long gone.
If you need a fast Opensource'ed LLMs you can go for EU-hosted DeepSeek or Qwen.
By this logic the Chinese should have just given up and let the American AI companies have the market because they were so far behind. I'm sure Europe has the capability to distill other people's frontier models to catch up if they wish to do so.
Distilling is unsafe from export control perspective - Chinese models are poisoned by US frontier distillation and a case can be made that the US won’t like distilling what they may consider transitively theirs, which they will the moment you’re anywhere near competitive.
US judges have already rules that output of an LLM can't be copyrighted so not sure what would prevent Chinese companies to use said output for distillation purposes.
Note I didn't mention copyright
By which other mechanism could American AI companies prevent this? Other companies don’t really care about EULAs and even if they needed to care it’s trivial to let third parties do it. Why would they? Almost nobody in the space cares about copyright and play fast and loose with laws and regulations.
What’s the mechanism that could today prevent other companies from using LLM outputs to train their models?
> already rules that output of an LLM can't be copyrighted
Mind sharing such cases? I'm not aware of any so far. There's the one with images, but that's commonly miss-understood, that case was ruled on a technicality (i.e. copyright needs to be attributed to a person, not a model)
https://www.copyright.gov/newsnet/2025/1060.html
> Distilling is unsafe from export control perspective
That is not the direction American judges are taking. Right now, they are saying that LLM output cannot be copyrighted. And if looting copyrighted works for training is fair game, I really don’t see how one could argue that learning from other LLMs is not.
I'm from nor cal but always liked Mistral.
Mistral 7b is still one of the best free/open models you can run locally on a MacBook. So fast too.
I'm a bit doubtful that a black box approach like this to moderation will ever catch on.
Mistral needs to abandon their Everything-stral branding. Getting kind of lame.
"Shieldstral" is an awkward and bad name
That naming only works if you commit to the bit even when it doesn't make sense. That builds branding.
Their web chat UI seems to be called “Vibe.” I think? I’m not sure if that’s the name of the product or just what they decided to label it in the browser. I wonder if -strap is just what they call the actual models, which are meant to be run “under the hood” anyway, so not really part of the branding.
But I wish they could commit to the bit fully and call everything -stral. It’s quirky and self aware to give your products silly names.
People complain when a product use a familiar name that might collide and there's also people complaining when they invent new words altogether.
Naming things is hard.
Says who? I kind of like it.
There can be other valid perspectives than your own
Was this one the last stral for you?
The stral the broke the camel's back?
The shortest stral has been pulled for you
It seems like you're just clutching at strals now
Someone should use this to do the exact opposite of the intention: filter for “offensive” content, and boost it or collate it into a newsletter/email blast for people of culture.
You have to give it to Mistral they do at least know what the market near them says they want right now. The great problem is in a few years of this that market won’t be worth anything.
Edit to add, you could also add this to an AI workflow so as to produce content that walks right up to the line but doesn’t trigger it.
> filter for “offensive” content, and boost it or collate it into a newsletter/email blast for people of culture.
I think that's the main service that xAI provide for X.
> The great problem is in a few years of this that market won’t be worth anything.
To be fair, we don’t know how much resources they put into this and how much of a distraction it was. If it was quick enough to train or fine tune and it brings them valuable experience for the next models, it could well be worth it in the long run even if there is no direct successor.
Also, it sounds like the kind of thing that sells. Any company with a customer support chat is a potential user of this model, any large company may be interested in getting a mistral installed set-up for handling without needing to send client info over the web. Installing those kinds of local systems seems to be what butters Mistral's bread at the moment.
this just sounds like new age edgey mc lordy lord. I get censorship is bad but ya'll might want to concern yourselves with a bid less fascism before you worry about the bad words police.
The bad words police are the actual authoritarian nutjobs. That is the problem.
> they do at least know what the market near them says they want right now
It does seem to be a very European approach to AI that their flagship AI lab is just making models that do nothing other than monitor and moderate internet content.
I guess they know that the EU AI Act, Chat Control, etc are going to cause a lot of companies to need this kind of compliance.
Some of the best social media is heavily moderate. This includes HN and r/credible defense . With a Quiet transparent and cheap LLM I imagine a social media website where you can have good discussion about everything around the world it would be a game changer and on my to-do list.
> Some of the best social media is heavily moderate
Heavily moderated by humans with discretion.
Not AI chat bots following a rules engine.
Wouldn't discretion "just" be a really good rules engine?
No, they are moderated by arbitrary company moderation policies not humans with independent thoughts.
AI can do exactly that.
> Heavily moderated by humans with discretion.
Pretty sure both of the above have extensive automation in their moderation.
The bulk of moderation work is things which are easy and obvious.
The correct way to moderate is automation with certainty falling back to humans with discretion.
The new frontier of moderation should be blocking illiterate comments, as in the commenter is replying as though they didn't read or read and didn't understand.
I'd rather be censored by an AI than a reddit mod.
> It does seem to be a very European approach to AI that their flagship AI lab is just making models that do nothing other than monitor and moderate internet content.
Well, first Mistral is French more than European. This might be a difficult distinction to make from the US but their approach is quite different from e.g. typical German companies.
Then, this is just a small model they release on the side. If that’s your benchmark, they released somewhat recently Voxtral, Voxtral transcribe, their OCR model, and Leanstral. I don’t think you can get much insight on their culture from this kind of release.
Another commenter already mentioned that it's more likely a lack of compute and funding that forces their hand to focus on niche tasks.
Ugh. There's nothing inherently European about Chat Control. It's a dumb proposal, and it's European. Any free society has a bunch of dumb proposals.
Nor is there anything inherently European about the AI Act. But that one I wouldn't even call dumb. At times misguided and confused, perhaps, but some of its core principles are valuable.
So a censorship model
What else to expect from an EU company than regulation? More of moderation, more of censorship, more of limitation on anything.