I've been writing about the need for provable inference (and training) on and off for a number of years; this is welcome news and a good start, and a significant improvement on prior zk results.
The problems of inference proof for a closed model provider are more difficult than an open model provider -- you can get a probabilistic proof much more cheaply out of four independent inference providers running the same non-batched prompt than this 90-100x compute hit.
Anthropic and others also have a significant problem with these proofs in that they have a lot of incentive internally to have some portions of the pre-prompt secret; I haven't read this newest scheme closely enough to determine if they have a way to hide parts of the prompt with the proof. But if they don't, they are almost certainly not going to want to send these proofs out in the general case.
Last, I'll take this moment to soapbox and say for the researchers out there -- provable inference is nice. Provable training corpus information would be spectacular. It would be very, very, very nice to provide a set of model weights and a proof that they had only been trained on a dataset. To be clear, this is a really hard problem, but it would change fundamentally what levels of trust we could have about a given set of weights.
> It would be very, very, very nice to provide a set of model weights and a proof that they had only been trained on a dataset. To be clear, this is a really hard problem, but it would change fundamentally what levels of trust we could have about a given set of weights.
Is it hard? Seems like it isn't that hard if you use NVIDIA's remote attestation/enclave support instead of trying to do everything with ZKPs which have always been something of a dead end. CC overhead 2%, ZKP 100x+ and gigabyte sized proofs - just for inference. It's never been competitive.
I'm very skeptical anyone would care about this for training though. I mean, if there's someone out there who genuinely really really wants proofs of the dataset used to train model weights, let's talk? But it seems the value is mostly for open weights inference providers. They can prove what weights they're serving and they can prove they can't see people's prompts. It would take a lot of the risk out of using a no-name LLM cloud service.
Unfortunately, after many years of this stuff I have become old and cynical. The stated/revealed preference mismatch is huge. The majority of people say they want privacy in theory but in practice won't trade anything off to get it, and especially not convenience.
I agree ZKP is the wrong approach. I also agree most people don’t care.
I’d love to read a plan that lets someone trust a pile of weights that comes with some sort of provenance proof. I’ll happily call it and you brilliant even if it was really easy to write and come up with. All my ideas for it look really difficult.
Damnit Peter, now you nerd sniped me. What ideas were you looking at?
Here's the architecture I'd use. I'm going off of NVIDIA's current recommended stack / what seems most mature, if I understand correctly after a quick chat with GPT.
Basically I'd start with a bunch of Xeon 6s (Granite Rapids) connected to Blackwells. You could do it with AMD too but I trust their stack less. That gives you TDX as a starting point, which is pretty mature by now. I'd prepare a very minimal Linux image in which the OS boots straight into running a training server. You need the usual stuff like dm-verity or IMA to ensure the disk image is covered by the kernel image. The image also contains NVIDIA's driver and attestation SDK. Boot it all inside a TDX VM with GPU passthrough. I believe that currently uses encrypted bounce buffers so there'd be some perf loss, but supposedly there's some PCIe extension that can be used to eliminate that (I haven't investigated).
The trick is to combine NVIDIA's RA with Intel's. The best way to do that is just hash the evidence from the GPU attestation process into an intermediate data structure which also contains the usual stuff (e.g. ECC pubkey, nonce), that itself is hashed into the user slot of the TDX RA and then bundled all up together. After doing this you have a single über-data structure which can be used to prove that the given public key was generated inside a genuine Xeon TDX VM connected to a genuine NVIDIA GPU running in CC mode. I'd probably just use a zip file to collect all these bits of data together, for convenience when implementing the verifier. It might be worth including an RFC 3161 timestamp over all that as well. The RA protocols for something like this are always very tricky: I'd need to do some careful design work before committing to anything so don't over-read this comment.
Now the TDX RA covers the training server (indirectly, via the dm-verity root via the kernel args+hash). What is this server? It's training code (e.g. PyTorch) combined with a server that accepts document batches and - for more advanced training - maybe stuff like AllReduce traffic. Each time a document batch is accepted each doc is hashed into a sorted Merkle tree. The weight updates are then computed on the GPU and the results are hashed into a second accumulator.
Each node creates a blockchain-like log in which each record is signed by the static key embedded into the TDX RA. The records point at the hash of the previous records, and contain the current value of the Merkle root and weight accumulator. A verifier can rebuild the roots by iterating over the training set themselves in the right order and just hashing the text. I won't go into the details as you know how all this stuff works already.
I'm handwaving a lot of details around parallelism here. For a large scale training run with many nodes you'd need to solve a bunch of details like making the training servers verify each other's remote attestations before they join the cluster, etc. During the AllReduce step nodes would need to verify the data comes from a authenticated TDX VMs before accepting the gradients.
Anyway from time to time the weight updates have to be applied and the log also records the hash of the full sets of weights at that checkpoint. The final model consumer can be given the weights, along with the signed logs generated during the training run. They verify the chained RAs, verify the signatures on the logs, and if they wish, verify [a subset of] the hashes of the documents in the open training set.
Because the document Merkle tree of each batch is sorted, you can build both inclusion and exclusion proofs. Verifying an exclusion proof is more expensive as you'd have to check each batch versus a quick Merkle branch check for inclusion, but exclusion checks can be done in parallel. Normally in ML documents are shuffled rather than sorted, but the training server can sort the set for building the log entry Merkle tree, and then submit to the GPU in a shuffled order.
The bigger issue for exclusion proofs is that SHA256 is too rigid. You might need some sort of simhashing scheme to run along side it. Dunno, needs more tightly specified requirements.
Most do not even realize they have a choice and are not educated enough to make the one in their own best interest. Fast food wins as a staple for many people over a healthy home cooked meal.
Creating healthy and safe choices and making them the default is some of the most important work there is.
I have been addressing this problem head on aggressively; https://github.com/emberian/whatember is the toplevel index. Things like arithmetizing parser/other automata to write semantic predicates (incl graph rewrites!) over training sets, inference requests, etc.
> In both cases the question is the same: how can a company be sure of which weights are actually running on its servers?
> the resulting commitment is good for about 16 million requests, after which the provider has to commit again.
> The client downloads the commitment, ~6.5 MB in all, and then one proof per request. Checking a proof takes 17 seconds on 24 threads (and 21 seconds on a single thread), including the offline phase's check
I know they mentioned $ETH in the technical report, but this is exactly the mechanic for a crypto currency Proof-of-Work which is expensive by design.
When they announced it, I was hoping for a 5% overhead validation that the model that served you is really Fable and not Opus.
The quantization bit caught me out, because we saw the boundary crossing from good-context-size to bad is so sharp it feels like longer context requests are sent to quantized models to make more room for more KV-Cache. This would expose those sorts of tricks people can play to have the same GPU do better on short context benchmarks and gracefully degrade on long context.
At work, we did a bit of digging[1] into exactly why BaseTen, Fireworks and Together serve Inkling weights differently, we suspected the issue was a combination of how you do disaggregated prefill (+streaming) and quantization for longer context tasks.
No, that has nothing to do with PoW. PoW is computing something intrinsically useless but very hard just to prove you spent all that time, but trivial to check.
ZkPs are expensive to check and even more so to generate because the problem is intrinsically very compute heavy. The ecosystem would immediately jump into a faster way of doing thing, compute time are seen as a flaw, not a feature.
For open-weight models, provable inference is trivial: just redo the computation! This is much faster, cheaper, and doesn't require any fancy cryptography, just deterministic inference (which, if you read carefully, the ZKP approach needs too). See https://lukechampine.com/repligraphs
Of course, it's unlikely that we'll get open-weight models from OpenAI or Anthropic any time soon. That means we have to choose between a) fancy ZKPs, b) signed responses, or c) comparing results from multiple inference providers.
Whatever method ends up being used, it's clear to me that some form of provable inference is inevitable. People haven't caught on to its more significant applications, but they will. (This post touches upon one of them: model-driven audits.) Think about it: we can prove that a particular mind authored a particular artifact. That's never been possible before! You can't prove that Melville wrote Moby Dick, for example; but you can prove that Claude wrote a poem, or that Gemini approved a pull request, or that GPT got a particular score on some eval. This is a more powerful primitive than a cryptographic signature.
> The verifier runs on the client's CPU. It needs no GPU and only has access to a commitment to the weights. The client downloads the commitment, ~6.5 MB in all, and then one proof per request. Checking a proof takes 17 seconds on 24 threads (and 21 seconds on a single thread), including the offline phase's checks.
As a client that sounds like it might be much cheaper to verify than reprocessing.
As a client, you're still paying for the compute even if it happens on the service provider's side. Running the same computation twice (perhaps by another provider) is 2x the cost; but if you ask the provider to make this proof, that increases the cost 70-95 times according to the article.
With that level of overhead, all kinds of otherwise inefficient approaches of getting compute start to seem comparably cheap. This is progress, but it still needs like a hundredfold improvement in order to become practical; I don't see a usecase where the benefits of this proof would be worth 70x or even 7x increase in cost.
Since it's CPU vs GPU, it's not apples-to-apples. But in terms of wall-clock time, one inference pass is surely faster than 17 seconds. Plus there's the proof generation step, which the post describes as ~100x slower than generating the response itself.
21s on a single core for 1024 tokens, that’s ~50t/s to verify and a 6.5 meg download. I’m pretty sure you won’t hit that for a 400B model at 4 bits, but then maybe you would I don’t track that performance so much. You’d need an enormous amount of ram to handle it though.
Yes, the generation is slower but I think here the verification itself is more efficient than regenerating the output.
The bigger factor is that (IIUC) ZKP-based verification is constant-time in the size of the output, so it definitely beats regen-based verification (as long as the constant is reasonable). And the ability to verify on consumer hardware instead of a beefy GPU is important too.
I think there is a place for both. Deterministic inference has other benefits, like "compressing" an output into input+model, as well as more esoteric applications like solving coordination problems. But if we get fast ZKPs for real frontier models, I'll be overjoyed!
> Of course, it's unlikely that we'll get open-weight models from OpenAI or Anthropic any time soon
This could be worked around by signing the output now and releasing the model after an embargo period, paired with contract penalties if the output of the released model does not match or the cryptographic scheme is compromised in the meantime. However, such guarantees would be worth little, since they can be just moved to a bad bank that defaults on them.
At first I thought this was work on the far more important problem of proving that LLM output is provable and correct given some source of truth. But it's about the trivial problem of 'proving' which model is being used.
Who will find this useful? I can't really think of a use for it, because if you don't trust the provider of your models, it's game over really - you're already giving them all sorts of private data, you're relying on their service, you have to trust them to some extent and this doesn't really help because unless you are running the service yourself. Seems like more of an internal problem for Anthropic than something that would interest others.
1. it proves your models are running correctly. FP ops tend to deviate in small amounts between hardware providers and tensor ops even more so due to the relatively new and unstable standardisation environment atm. So it proves your HW configuration is delivering outputs correctly. i.e. it's a good test for what is currently a very non-trivial validation environment.
2. it proves your external provider is running the inference they said they are. i.e. it eliminates the concerns of model providers running quantized versions silently behind the scenes since quantization breaks the proof and you need to use a new, updated proof signature for the quantized version.
3. likewise for fine tuned or tampered models. Or outright substitution for another model (ex the accusation by OAI and Anthropic that Kimi and DS and co are periodically forwarding requests to their platforms to harvest responses for knowledge distillation).
All in all it's a nice chain of trust utility if it can evolve out of the toy stage and reach production viability (since rn it's limited to int-only weights and 1k context which is practically useless atm).
1. That sounds like an Anthropic concern, not a concern for most customers who don't run their own model.
2. This still requires trust in the provider, I imagine these proofs would change weekly as they tweak their models, and you have zero knowledge of what the changes are, you just know you're getting version 11.12.1999 - does that really help you if the provider is maliciously downgrading after release? All you know is you got the same version as everyone else, not what that version is.
3. Again, if you don't trust the provider, it's game over - you're sending them all of your data. Knowing a version number is not going to help you much.
I'm not convinced this is useful for problems most people face using models, it seems a very niche academic concern frankly and an incredibly complex system to prove not very much.
1. It's a provider concern and a developer concern. If your infra is serving broken proofs that might be justification for compensation for breaking SLA.
2. Models only change weekly for closed frontier models and even then they should be reporting model "sub-versions" if only for observability purposes.
3. This doesn't really hold true. Kimi and DS release their models on a monthly or bi-weekly cycle and while it's still DS-v4.1-Flash or whatever, under the hood it's DS-v4.1-Flash-1017 or some other date code identifier of which specific model it is and that is both exposed to the user and that model version can be requested explicitly by the user.
For example, there are Chinese resellers that promise to provide access to Claude (which isn't officially available in China) and defraud users by substituting something else. They're obviously sketchy in all sorts of ways, but in theory, this would prevent one kind of fraud.
It might also provide an additional guarantee for users of Open Router and Fireworks.
It's worth reading past the opening paragraph before asking these kinds of questions:
>When a user sends a request to an AI model, today they have no way of knowing for certain that the model they selected is the one that generated the response. For everyday uses, in which users mostly care about the output and not necessarily which specific model produced it, this is a less pressing concern. But there are several cases where such assurance would be useful. For example, independent evaluators who test AI models before and after release need to know that the model they are evaluating is the model being served. Clients who use a model through a third-party routing service want assurance that the tokens came from the model they paid for. And if a cyberattacker modifies the token stream while pretending to be a given model, a user has no way to detect it from the output alone.
This is from the first paragraph in the Introduction.
I didn't find any of it compelling, perhaps you do. That's what I was asking really, which of these use-cases is valuable for you?
For me if you use a routing service you have to trust it, it sees all your inputs and outputs.
If a cyberattacker can modify the token stream (i.e. man in the middle your traffic), you should be worried about a lot more than which model you're using!
If one were to listen to /r/claude or /r/codex where half the posts are complaining about models being "lobotomized", one could think that this is the most valuable thing there is!
In all seriousness, this is mostly just one of those defense in depth kind of things. Data exfiltration via a compromised provider is also a very different class of vulnerabilities than prompt injection via malicious tool calls.
But it still requires trust in whoever is running the server that tags version 'x' as x. All you have proof of is that they return the hash for a given set of weights which is the same as someone else's hash, and those tags/hashes will be changing all the time as the models evolve, so people are not going to be able to keep track of whether Anthropic say is dumbing down their model or not after release, because this is a zero knowledge proof, all it tells is that a certain set of weights was used (you have no idea what weights, how many weights etc etc).
So I don't really see this as useful, except in the very limited use case of using a third party proxy who is trying to cheat you (in which case why are you sending all your data to someone you can't trust?).
> and those tags/hashes will be changing all the time as the models evolve
I think that's where your assumption differs from stated claims.
According to Anthropics own docs[0], they don't alter the models at all after release. So for each model release you should be able to prove that a response was created with that exact model without dumbing it down.
Zero knowledge proofs also still allow you to have disclosed inputs. You can make one of the inputs be a publicly commited hash of the model weights, and have the ZKP validate that the inference was done with that committed set of model weights, in addition to all the inferences in the batch all coming from the same model.
If the releases change frequently though, this would be of little use to anyone save those who feel sticking with one model is best. The weights are not exposed so you have to trust Anthropic that changes they make are in your best interest.
None of these are directly valuable to me. But you asked "Who will find this useful?" and I gave you the answer to that question (provided by Anthropic in this post). Not everything being worked on will be useful for you. It's just bizarre that you'd be trying to make these arguments.
>If a cyberattacker can modify the token stream (i.e. man in the middle your traffic), you should be worried about a lot more than which model you're using!
Sure, so it'd be great if you could know if someone is modifying your token stream. If only there was some advancement in provable inference that could allow you to verify that what you're getting from a provider is what you expect to get from them.
Customers care about model output being correct, Anthropic cares about model output being generated by Anthropic model.
This solves for the latter with a bunch of handwaving for the former. The end goal is gonna be FUD for cheaper model routing because that obviously is at odds with their biz model.
You developed your agent implementation and more importantly, evaluated it, against a specific model.
You don't want all you eval work to go to waste just because the discount inference provider you're using decided AcmeModel4.23 is as good as the Anthropics GilgameshEpic7.2 that you used to validate your solution.
We already have trouble enough with model retirement.
Off the top of my head, it is useful when there are content-level proxies between the client and anthropic's API server, of which there are many these days.
> As models become capable of improving and deploying themselves ... how can a company be sure of which weights are actually running on its servers?
I find this a bit odd and backwards. Why do we choose not to make the deployment code to be robust, but instead to observe the model inference as proven? This moves unobservability layer to the system that collects the proofs. Given the assumption that code is fluid and LLMs are rewriting it often, how can we know it isn't being abused against the observer?
Goes without saying, it's also disappointing that this has nothing to do with making the model inference observable for consumers.
Anthropic here is solving one of their own problems, how to make sure you don't combine other company's models with their stuff for example, and make it sound as if they did it for you.
It makes the inference more expensive and helps controlling the users.
I think some model providers are implementing "eyes glaze over" where if you ask a boring simple question they route it to a cheap model but charge you for an expensive model, so consumers may want a proof of work certificate to know they're getting what they pay for.
I think a consumer facing system would be fantastic, but from how they're framing it in the article, it's all about internal company monitoring, with some fearmongering around LLM observability.
I assume this alludes to their prior work with models running on smart TVs being able to setup inference on additional smart TVs. The idea being that you can't be sure that, having gone through the whole deployment process, your hardware has not been subverted to run a different model. Any level of "robust" code you can imagine to prevent this in the traditional ways could be subverted on the fly by a capable model.
> This moves unobservability layer to the system that collects the proofs.
I'd say it's adding security through a secondary system, rather than displacing security to a secondary system. Security by layers.
I've read that providers like those you find on OpenRouter can sometimes route to other models; this would catch that.
But I've also heard, as complete rumour, sometimes some named models perform significantly worse on some hosts and the theory is they're running quantized, without that being published. Would this solve that? The post mentions proving a set of weights, but not how those are represented? What would stop a host using true weights for generating the signature, while running on quantized versions?
> We prove an exact integer version of the published weights, and a proof asserts that each checked token is that model’s greedy choice. A reply from a floating-point implementation, or sampled decoding, could depart at some position...
I think it would catch that, at least for open weights. From the repo docs [1] it looks like this type of attestation is even supported for open weight models: you commit with the public weights and publish the seed, so anyone can recompute the commitment from the Hugging Face files and check the provider is running the exact checkpoint and quantization they say they are. That's what they did with their Maverick demo. That said, it only works for greedy decoding and short contexts for now, and it's 70-95x the GPU time, so there may be cheaper ways to do this like TEEs.
Maybe it's possible to tell by looking at the size of that error?
> The integer version is generally expected and empirically tested to be within some small error rate of true floating-point inference generically (even taking into account non-determinism of floating-point computations), though this does not cover adversarially designed prompts.
Interesting, but I'm not sure the property it attests is one we care about that much. Or rather, it's one you'd expect Anthropic not to tell you it needs to prove. I mean it tells you that you're not being stiffed by Anthropic using a cheaper model and that they haven't literally been MiTMd, but that's kind of table stakes. Otherwise it's not telling you much useful. It's not telling you that your data hasn't been exfiltrated, and it's not telling you any useful property of the answer.
> I mean it tells you that you're not being stiffed by Anthropic using a cheaper model and that they haven't literally been MiTMd, but that's kind of table stakes.
That's literally the most common gripe I see from Anthropic users and, so far from being "table stakes" there no existing mechanism by which those users could verify it is or isn't happening. If Anthropic can figure this out then most of this "is it just me or is Opus hurr durr today" noise can evaporate.
We're working on provable inference at Attestable (https://attestable.com/) - our solution is one of the comparisons provided.
We've published initial performance metrics in August of 87 t/s on Muse Glimmer 30B (a dense model) - orders of magnitude faster than existing OS projects. Antropic's FlashProve supports only a fully integer model with a small context length (significantly hurting evals).
There is significant progress since then, and we'll publish more results soon.
This obviously an attempt at being able to say a response came from an anthropic model vs say some other model via a model router. As for me, I will keep using my model router, I don't care to be charged more for a name brand. I don't care about the brand at all, it's a commodity.
until your company gets sued for using Dangerous and Shoddy Foreign models and the boss says you can only use the Trustworthy and Premium Aligned labs' models.
It feels kinda yucky reading through a document that spends half the time saying how this great for our regulatory capture efforts and how they won't have to show anyone the wieghts. It's not even a bad thing, you want to know you are using what you think you are, but they are advertising the side effects as features.
My PhD is on auditability of AI systems, and we are exactly working on the engineering of ZKP in this context. I can confirm through our industry partner (AI Assurance and auditing), that black box auditing is indeed a huge problem for them, and snarks have shown their potential in this context. It's great to see Anthropic confirming our early results.
HTTPS makes sure you’re communicating on a secure channel with the API endpoint. ZKPs guarantees that the computation happened correctly behind the endpoint.
The work from Anthropic isn't a new idea, but they made the process more efficient, as inference verification is one of the biggest bottlenecks in the AI lifecycle. Beyond that, there's a lot more that can be proven, like showing a model was trained on a specific dataset with certain properties.
From an auditing perspective, this used to be handled either by showing the data (which is pretty rare), checking samples (which has been shown to be easily hackable), or similar workarounds. For inference, an auditor might want to interact with a model on demand and have absolute certainty that the model they're talking to is actually the one declared, and not a substitute.
I’m not sure what you mean by correctly here - do you mean with the model expected? That’s not related to correctness.
Pleased to hear somebody cares about this. What is it a bottleneck for though? Are people actually doing this right now in non-academic settings? I had the impression this is not in fact usable with larger models.
ZKPs verify "things" by encoding the verification in a computation, and verifying that the computation happened in the right way, means to assess it was computed correctly, i.e., with the right inputs, the right operations, etc.
So this is what I meant with "correctness".
The bottleneck has two sides: first, there's a technical bottleneck. ZKPs are quite expensive, and finding the right, production-ready protocol is challenging. The other side is a socio-technical bottleneck. There is always quite a large friction with adopting these kind of technologies, also due to the first bottleneck.
As you said, it is difficult to adopt because of how fast the complexity grows with big models. It's great indeed to see that Anthropic is tackling the challenge.
Well luckily not really, their work is more foundational-y, the idea is not new, but they made it more efficient and practical. On our side, we work more on the integration of these protocol in production systems, but we are pretty much on the frontier and is not very common to see such big companies experimenting with this technology. Microsoft already tried something [1] but I'm not sure if it ever went further than a research prototype.
I'm a little confused how this is even feasible to do in general. Most inference is batched and often implemented on heterogeneous hardware. In this environment, it's hard to get bitwise identical results, much less prove that the results came from a certain model. Is this actually practical? In the post near the end it doesn't seem like the authors think they have done anything for this and it seems pretty important for it to be used for any production environment.
This sort of result is about steady progress. It's not practical, yet. This system only allows for 1k context, not even enough for a system prompt. But the authors have contributed several orders of magnitude of speedup on the current state of the art.
I wager regulatory capture lies this way. Look for this becoming an in-hardware feature, with big fat patents on it, and then a push to demand all inference providers use "TPM chipped" hardware.
This is built on VerInf, work that I did as a MARS fellow. They ripped everything down to the choice of 400B Maverick from us with very little attribution. Cool, I guess...
https://github.com/jamespetrie/verinf
This is an exaggeration. The commitment scheme is related but different (Ligero vs Zook). Matmuls are different (Freivald vs Sumcheck), lookups are different (LogUp vs Grand Product), the packing is different.
What original ideas are in VerInf that are also in this work besides the choice of benchmark?
I'll concede that "ripped everything" was written as a strong emotional reaction. But you are also exaggerating; there were a lot of novel ideas that we worked through used here. I'd point you to our choice of UD-Q4 and rules for how to convert to integers as a starting point.
Claude being a official contributor here gives me the shivers.
This feels like setting up a case for: We have the rights for everything you do with Claude. Or rather what they do with Claude for you.
I'm just annoyed that they treat it as a person. Claude isn't a contributor more than that shovel I used to dig a hole in the ground is. Even if it's a really good shovel.
Sorry to hear. Just asking, why did you choose MIT license over something like GPL? It's not that they couldn't have copied you, but you would be in a different position vis-à-vis them now.
I would've chosen GPL personally, but a prototype repository already existed under MIT by the time I joined as a contributor through the fellowship. Alas.
It's actually fine to use MIT code and publish it under GPL, or to switch from MIT to GPL. You can still do that today on the repo, btw. It's the other direction that is thorny. Just fyi
...thus implying that you'd rather the preeminent safety-focused lab not have used your interpretibility research, as they're technically a for-profit company with various closed-source assets?
I am certainly biased by a passionate distaste for IP law in general, but I am honestly curious to hear the reasoning behind it in this case.
In case it helps: this is a "research prototype", so I'd assume that seeing one or more aspects used in a highly impactful paper would be about the best possible outcome. Is this about publish-or-perish stuff, perhaps (e.g. citation "importance" for the algos)? Or maybe you're okay with this in general, but are regretful because it's Anthropic in particular?
I think I misunderstood GPL per a comment above so it's all kinda moot anyway, but my original understanding was that it was a copyleft license, so wishing it was GPL would be equivelant to wishing Anthropic had not used that work at all.
Attribution is certainly polite, and I sorta doubt Anthropic of all groups is intentionally witholding it for h-score or prestige, but that's all a much lower-stakes conversation than potential license violations. AFAIU.
OP was pretty clear they wanted attribution, so you have hallucinated an entirely novel implicated desite for exclusion.
Seeing as the question posed out of "[honest curiosity]" to someone else is then immediately answered by yourself host without you even pausing to put on a sock, I think your honesty is very questionable.
Oh sorry, I must have misunderstood! My bad, honestly; I was under the impression that GPL vs. MIT was about copyleft stuff, which would seem to preclude Anthropic's use of that project altogether. Not 100% sure how it relates to attribution still but I'll have to look into it I guess.
I don't really understand what was "answered by [my]self host" nor what a "desite for exclusion" means but it seems beside the point I think. Also not sure what the alternative to curiosity is. Like, I'd be trying to entrap this prestigious researcher in some legal sense...?
Regarding attribution, I guess he is one of the copyright holders, so his name should be included in the new license. Unless we are talking about MIT-0.
Im not an expert on these things, but I think there is another type of attribution, that I guess is what the commenter is more concerned about, and that is academic attribution. It is part of academic culture that you attribute significant influences in the articles you write. There are no licenses as such for this, as far as I know. You do it because that is what you do in academic circles. If you dont, you are not in the club.
Protection and enforcement are kinda two sides of the same coin in most IP law. IP essentially grants you exclusivity for your works, and your protection comes from how well you can enforce that exclusivity. As an example, piracy exists largely because copyright holders simply can’t enforce their exclusive rights to distribution effectively on the vast, distributed Internet.
It’s still early days, true, and courts may find legal reasoning to address this... But I fear this stems from a fundamental yet little-known flaw in copyright for code, based on a legislative hack, which can only be made worse by caselaw. I ranted about it a few days ago: https://news.ycombinator.com/item?id=50320875
I wonder if this is partially letting agents.. agent, agentically and not know where ideas come form or how they come together, only whether it works or not.
I’m so tired of hearing stories like this. The biggest most well funded corps in the history of mankind and all they do is steal the shit out of everything. It’s pathetic.
I hope you don’t get that wrong, but maybe your best option is to see it positive. The top comment on a first page post on HN, gives you some publicity. Potentially more than if that would’ve not have happened.
You can also use this and create explanation video on it on Youtube, post on X, Bluesky and other in-detail stuff that only you and very little people are in the position to do.
I’m sure some are interested, and this could lead to a hire, too.
I know it’s always easy to say and I probably have less adversity faced as most people, but I really try to see them as energy enablers. Of the almost infinite options you have in life there is a big chunk of realities that are better than your current ones an adverse event can trigger emotions and therefore energy and push one to do things one wouldn’t have done in comfort.
I appreciate your positive energy here a lot. And I agree; all things can be dichotomies. This is how our entire team reacted to the news: it is good news and bad news both.
I agree with this. I often dedicate my life for months on end to prove novel and potentially valuable ideas just in the hope that one of the largest companies in the world can rip it off so I have the prestigious opportunity to make a youtube video about it. I can't believe that everyone doesn't do this.
Well I’m not saying to justify that, you can also start an activist movement to fight that, but I would rather move in the worlds that I can affect than in the one where I complain with no impact.
I’ve only done a bunch of "thinking" videos. Nothing special, so might not be the best advice giver, but I’ll share what I learnt.
Don’t try to achieve any goal with it, except doing it. When doing it you’ll eventually get better in making videos.
The first video takes way longer than you think it takes, but the second probably takes half the time and it will be tremendously faster over time.
Try to reduce the friction on recording. My first training video on my app took me almost a full day to set up and plan with teleprompter and everything. After a couple of time I could do 3 in a row.
Making videos is still not "my thing" that I enjoy most about, but I’m still happy that I did it, if that’s what I need to I’ll do it. Right now coding has more leverage for me.
A technical write-up or a blog is an alternative. Or even reaching out to Labs. You could reach out to OpenAi/Anthropic/Mistral or your 'poison' of choice, or companies that do something similar. But this is now on your CV.
i guess the idea here is that if the end user cannot discriminate based on capability above some threshold, the premium that you can charge for more capable models either disappears entirely (if the end user is the principal source of truth on quality) or gets arbitraged down (if the end user is merely an agent of the principal source of truth on quality). to maintain that premium in perpetuity, you just need (a) trusted benchmarking services that can distinguish between premium and generic, and then (b) a way to prove to the end user that a given output is premium.
> The goal of provable inference (PI) is to enable a user with limited computational resources and without any information about the weights to verify that a response came from the requested model
Isn't this already solved by watermarking?
This page said they would provide an API to check whether a given text has the watermark for a Claude model:
Not to the degree needed for actually verifying the things this verifies. You could probably quite easily produce a non-Claude model that generates text that conforms to Claude watermarking.
In the page I linked above, Anthropic explains that they use the techniques from SynthID-Text, where a secret watermarking key is involved. Only Anthropic knows this key, so only Anthropic can produce and detect the watermark. (But they also promised to provide an API through which other parties can check the watermark.)
- it only gives a high probability that the text was outputted from the model, whereas a crytographic proof is computationally infeasible to forge
- it cannot be traced back to the input. You could "wash" the output from one model into another and it would be watermarked by the latter. With the crytographic proof, if you have the input, output, and "commitment", you can verify your exact input was processed by the model (or so the model provider says).
Very cool. Now prove you are unable to read chat transcripts with provable end to end encryption down to the CPU registers and TPM endorsement chain of the software involved.
Provable privacy via remote attestation effectively gives you provable inference for free.
What are the tradeoffs of an approach like this vs confidential computing primitives and attesting everything from untampered hardware all the way up to the served request? It seems more performant. Is it simply that they would have to reveal too much?
More so that attestation requires the model to be running on secure hardware, which is hard to do across all models that a company like Anthropic serves.
In part because many of the accelerators they use may not support attestation, and even with ones that do, such as NVIDIA Blackwell, support is restricted to a single node (8 GPUs). Vera Rubin supports rack-scale encryption, but that is not broadly deployed yet and will not be the only accelerator they use to serve inference.
> A user querying a model today cannot check that the responses came from the specific model that a developer claims is responding
Indeed, I wonder what kind of developer would make users worry about that...? Funny PR stunt coming from a company that was in the news for silently routing ML-related requests to an inferior model.
> Unlike Fable’s other restrictions, such as around cybersecurity and biology, which openly redirect users to a less powerful model with a visible notification, the system card emphasized that this is “not visible to the user.” The model still responds, but uses “interventions to limit Claude’s effectiveness” without telling the user it’s doing so.
"The commitment to the model, and subsequent inference proofs, hide the weights of the model; hiding the architecture of the model is not a goal of this system."
> Some proposals to regulate and pace AI development rely on monitoring hardware and how it is used. For example, a provider might be required to devote its hardware to inference rather than to training a powerful new model. The provider might try to cheat by serving cheaper models or using compute in an unexpected way, freeing up resources for training. How can a regulator reliably ensure that the hardware is used for inference?
Is this even a real problem? Forcing a company to only do inference?
I've been writing about the need for provable inference (and training) on and off for a number of years; this is welcome news and a good start, and a significant improvement on prior zk results.
The problems of inference proof for a closed model provider are more difficult than an open model provider -- you can get a probabilistic proof much more cheaply out of four independent inference providers running the same non-batched prompt than this 90-100x compute hit.
Anthropic and others also have a significant problem with these proofs in that they have a lot of incentive internally to have some portions of the pre-prompt secret; I haven't read this newest scheme closely enough to determine if they have a way to hide parts of the prompt with the proof. But if they don't, they are almost certainly not going to want to send these proofs out in the general case.
Last, I'll take this moment to soapbox and say for the researchers out there -- provable inference is nice. Provable training corpus information would be spectacular. It would be very, very, very nice to provide a set of model weights and a proof that they had only been trained on a dataset. To be clear, this is a really hard problem, but it would change fundamentally what levels of trust we could have about a given set of weights.
> It would be very, very, very nice to provide a set of model weights and a proof that they had only been trained on a dataset. To be clear, this is a really hard problem, but it would change fundamentally what levels of trust we could have about a given set of weights.
Is it hard? Seems like it isn't that hard if you use NVIDIA's remote attestation/enclave support instead of trying to do everything with ZKPs which have always been something of a dead end. CC overhead 2%, ZKP 100x+ and gigabyte sized proofs - just for inference. It's never been competitive.
I'm very skeptical anyone would care about this for training though. I mean, if there's someone out there who genuinely really really wants proofs of the dataset used to train model weights, let's talk? But it seems the value is mostly for open weights inference providers. They can prove what weights they're serving and they can prove they can't see people's prompts. It would take a lot of the risk out of using a no-name LLM cloud service.
Unfortunately, after many years of this stuff I have become old and cynical. The stated/revealed preference mismatch is huge. The majority of people say they want privacy in theory but in practice won't trade anything off to get it, and especially not convenience.
I agree ZKP is the wrong approach. I also agree most people don’t care.
I’d love to read a plan that lets someone trust a pile of weights that comes with some sort of provenance proof. I’ll happily call it and you brilliant even if it was really easy to write and come up with. All my ideas for it look really difficult.
Damnit Peter, now you nerd sniped me. What ideas were you looking at?
Here's the architecture I'd use. I'm going off of NVIDIA's current recommended stack / what seems most mature, if I understand correctly after a quick chat with GPT.
Basically I'd start with a bunch of Xeon 6s (Granite Rapids) connected to Blackwells. You could do it with AMD too but I trust their stack less. That gives you TDX as a starting point, which is pretty mature by now. I'd prepare a very minimal Linux image in which the OS boots straight into running a training server. You need the usual stuff like dm-verity or IMA to ensure the disk image is covered by the kernel image. The image also contains NVIDIA's driver and attestation SDK. Boot it all inside a TDX VM with GPU passthrough. I believe that currently uses encrypted bounce buffers so there'd be some perf loss, but supposedly there's some PCIe extension that can be used to eliminate that (I haven't investigated).
The trick is to combine NVIDIA's RA with Intel's. The best way to do that is just hash the evidence from the GPU attestation process into an intermediate data structure which also contains the usual stuff (e.g. ECC pubkey, nonce), that itself is hashed into the user slot of the TDX RA and then bundled all up together. After doing this you have a single über-data structure which can be used to prove that the given public key was generated inside a genuine Xeon TDX VM connected to a genuine NVIDIA GPU running in CC mode. I'd probably just use a zip file to collect all these bits of data together, for convenience when implementing the verifier. It might be worth including an RFC 3161 timestamp over all that as well. The RA protocols for something like this are always very tricky: I'd need to do some careful design work before committing to anything so don't over-read this comment.
Now the TDX RA covers the training server (indirectly, via the dm-verity root via the kernel args+hash). What is this server? It's training code (e.g. PyTorch) combined with a server that accepts document batches and - for more advanced training - maybe stuff like AllReduce traffic. Each time a document batch is accepted each doc is hashed into a sorted Merkle tree. The weight updates are then computed on the GPU and the results are hashed into a second accumulator.
Each node creates a blockchain-like log in which each record is signed by the static key embedded into the TDX RA. The records point at the hash of the previous records, and contain the current value of the Merkle root and weight accumulator. A verifier can rebuild the roots by iterating over the training set themselves in the right order and just hashing the text. I won't go into the details as you know how all this stuff works already.
I'm handwaving a lot of details around parallelism here. For a large scale training run with many nodes you'd need to solve a bunch of details like making the training servers verify each other's remote attestations before they join the cluster, etc. During the AllReduce step nodes would need to verify the data comes from a authenticated TDX VMs before accepting the gradients.
Anyway from time to time the weight updates have to be applied and the log also records the hash of the full sets of weights at that checkpoint. The final model consumer can be given the weights, along with the signed logs generated during the training run. They verify the chained RAs, verify the signatures on the logs, and if they wish, verify [a subset of] the hashes of the documents in the open training set.
Because the document Merkle tree of each batch is sorted, you can build both inclusion and exclusion proofs. Verifying an exclusion proof is more expensive as you'd have to check each batch versus a quick Merkle branch check for inclusion, but exclusion checks can be done in parallel. Normally in ML documents are shuffled rather than sorted, but the training server can sort the set for building the log entry Merkle tree, and then submit to the GPU in a shuffled order.
The bigger issue for exclusion proofs is that SHA256 is too rigid. You might need some sort of simhashing scheme to run along side it. Dunno, needs more tightly specified requirements.
> The majority of people
Most do not even realize they have a choice and are not educated enough to make the one in their own best interest. Fast food wins as a staple for many people over a healthy home cooked meal.
Creating healthy and safe choices and making them the default is some of the most important work there is.
It would be fantastic to compare benchmark values. The current trust-me-bro economics leave little other choice than interpreting those numbers as PR.
I have been addressing this problem head on aggressively; https://github.com/emberian/whatember is the toplevel index. Things like arithmetizing parser/other automata to write semantic predicates (incl graph rewrites!) over training sets, inference requests, etc.
Thank you! I will review. Super interesting.
> In both cases the question is the same: how can a company be sure of which weights are actually running on its servers?
> the resulting commitment is good for about 16 million requests, after which the provider has to commit again.
> The client downloads the commitment, ~6.5 MB in all, and then one proof per request. Checking a proof takes 17 seconds on 24 threads (and 21 seconds on a single thread), including the offline phase's check
I know they mentioned $ETH in the technical report, but this is exactly the mechanic for a crypto currency Proof-of-Work which is expensive by design.
When they announced it, I was hoping for a 5% overhead validation that the model that served you is really Fable and not Opus.
The quantization bit caught me out, because we saw the boundary crossing from good-context-size to bad is so sharp it feels like longer context requests are sent to quantized models to make more room for more KV-Cache. This would expose those sorts of tricks people can play to have the same GPU do better on short context benchmarks and gracefully degrade on long context.
At work, we did a bit of digging[1] into exactly why BaseTen, Fireworks and Together serve Inkling weights differently, we suspected the issue was a combination of how you do disaggregated prefill (+streaming) and quantization for longer context tasks.
[1] - https://isotopes.ai/blog/everyone-is-selling-the-same-model-...
No, that has nothing to do with PoW. PoW is computing something intrinsically useless but very hard just to prove you spent all that time, but trivial to check.
ZkPs are expensive to check and even more so to generate because the problem is intrinsically very compute heavy. The ecosystem would immediately jump into a faster way of doing thing, compute time are seen as a flaw, not a feature.
For open-weight models, provable inference is trivial: just redo the computation! This is much faster, cheaper, and doesn't require any fancy cryptography, just deterministic inference (which, if you read carefully, the ZKP approach needs too). See https://lukechampine.com/repligraphs
Of course, it's unlikely that we'll get open-weight models from OpenAI or Anthropic any time soon. That means we have to choose between a) fancy ZKPs, b) signed responses, or c) comparing results from multiple inference providers.
Whatever method ends up being used, it's clear to me that some form of provable inference is inevitable. People haven't caught on to its more significant applications, but they will. (This post touches upon one of them: model-driven audits.) Think about it: we can prove that a particular mind authored a particular artifact. That's never been possible before! You can't prove that Melville wrote Moby Dick, for example; but you can prove that Claude wrote a poem, or that Gemini approved a pull request, or that GPT got a particular score on some eval. This is a more powerful primitive than a cryptographic signature.
Is it cheaper to rerun it?
> The verifier runs on the client's CPU. It needs no GPU and only has access to a commitment to the weights. The client downloads the commitment, ~6.5 MB in all, and then one proof per request. Checking a proof takes 17 seconds on 24 threads (and 21 seconds on a single thread), including the offline phase's checks.
As a client that sounds like it might be much cheaper to verify than reprocessing.
As a client, you're still paying for the compute even if it happens on the service provider's side. Running the same computation twice (perhaps by another provider) is 2x the cost; but if you ask the provider to make this proof, that increases the cost 70-95 times according to the article.
With that level of overhead, all kinds of otherwise inefficient approaches of getting compute start to seem comparably cheap. This is progress, but it still needs like a hundredfold improvement in order to become practical; I don't see a usecase where the benefits of this proof would be worth 70x or even 7x increase in cost.
Since it's CPU vs GPU, it's not apples-to-apples. But in terms of wall-clock time, one inference pass is surely faster than 17 seconds. Plus there's the proof generation step, which the post describes as ~100x slower than generating the response itself.
21s on a single core for 1024 tokens, that’s ~50t/s to verify and a 6.5 meg download. I’m pretty sure you won’t hit that for a 400B model at 4 bits, but then maybe you would I don’t track that performance so much. You’d need an enormous amount of ram to handle it though.
Yes, the generation is slower but I think here the verification itself is more efficient than regenerating the output.
The bigger factor is that (IIUC) ZKP-based verification is constant-time in the size of the output, so it definitely beats regen-based verification (as long as the constant is reasonable). And the ability to verify on consumer hardware instead of a beefy GPU is important too.
I think there is a place for both. Deterministic inference has other benefits, like "compressing" an output into input+model, as well as more esoteric applications like solving coordination problems. But if we get fast ZKPs for real frontier models, I'll be overjoyed!
> Of course, it's unlikely that we'll get open-weight models from OpenAI or Anthropic any time soon
This could be worked around by signing the output now and releasing the model after an embargo period, paired with contract penalties if the output of the released model does not match or the cryptographic scheme is compromised in the meantime. However, such guarantees would be worth little, since they can be just moved to a bad bank that defaults on them.
Good luck rerunning the computation done on 8xB100 unless you aren't a cloud provider yourself
At first I thought this was work on the far more important problem of proving that LLM output is provable and correct given some source of truth. But it's about the trivial problem of 'proving' which model is being used.
Who will find this useful? I can't really think of a use for it, because if you don't trust the provider of your models, it's game over really - you're already giving them all sorts of private data, you're relying on their service, you have to trust them to some extent and this doesn't really help because unless you are running the service yourself. Seems like more of an internal problem for Anthropic than something that would interest others.
The main benefits are:
1. it proves your models are running correctly. FP ops tend to deviate in small amounts between hardware providers and tensor ops even more so due to the relatively new and unstable standardisation environment atm. So it proves your HW configuration is delivering outputs correctly. i.e. it's a good test for what is currently a very non-trivial validation environment.
2. it proves your external provider is running the inference they said they are. i.e. it eliminates the concerns of model providers running quantized versions silently behind the scenes since quantization breaks the proof and you need to use a new, updated proof signature for the quantized version.
3. likewise for fine tuned or tampered models. Or outright substitution for another model (ex the accusation by OAI and Anthropic that Kimi and DS and co are periodically forwarding requests to their platforms to harvest responses for knowledge distillation).
All in all it's a nice chain of trust utility if it can evolve out of the toy stage and reach production viability (since rn it's limited to int-only weights and 1k context which is practically useless atm).
1. That sounds like an Anthropic concern, not a concern for most customers who don't run their own model. 2. This still requires trust in the provider, I imagine these proofs would change weekly as they tweak their models, and you have zero knowledge of what the changes are, you just know you're getting version 11.12.1999 - does that really help you if the provider is maliciously downgrading after release? All you know is you got the same version as everyone else, not what that version is. 3. Again, if you don't trust the provider, it's game over - you're sending them all of your data. Knowing a version number is not going to help you much.
I'm not convinced this is useful for problems most people face using models, it seems a very niche academic concern frankly and an incredibly complex system to prove not very much.
1. It's a provider concern and a developer concern. If your infra is serving broken proofs that might be justification for compensation for breaking SLA.
2. Models only change weekly for closed frontier models and even then they should be reporting model "sub-versions" if only for observability purposes.
3. This doesn't really hold true. Kimi and DS release their models on a monthly or bi-weekly cycle and while it's still DS-v4.1-Flash or whatever, under the hood it's DS-v4.1-Flash-1017 or some other date code identifier of which specific model it is and that is both exposed to the user and that model version can be requested explicitly by the user.
>1. That sounds like an Anthropic concern, not a concern for most customers who don't run their own model.
You're talking about this as if a bug or defect in their inference stack has no impact on you as the customer.
Yeah a provider serving botched inference probably would count against SLA if it was at all detectable.
Also useful to audit in case we pause the development of AI and we need to check inference is being done with known models.
For example, there are Chinese resellers that promise to provide access to Claude (which isn't officially available in China) and defraud users by substituting something else. They're obviously sketchy in all sorts of ways, but in theory, this would prevent one kind of fraud.
It might also provide an additional guarantee for users of Open Router and Fireworks.
But if you’re using them they’re probably already stealing your data, fake-models are the least of your problems.
>Who will find this useful?
It's worth reading past the opening paragraph before asking these kinds of questions:
>When a user sends a request to an AI model, today they have no way of knowing for certain that the model they selected is the one that generated the response. For everyday uses, in which users mostly care about the output and not necessarily which specific model produced it, this is a less pressing concern. But there are several cases where such assurance would be useful. For example, independent evaluators who test AI models before and after release need to know that the model they are evaluating is the model being served. Clients who use a model through a third-party routing service want assurance that the tokens came from the model they paid for. And if a cyberattacker modifies the token stream while pretending to be a given model, a user has no way to detect it from the output alone.
This is from the first paragraph in the Introduction.
I didn't find any of it compelling, perhaps you do. That's what I was asking really, which of these use-cases is valuable for you?
For me if you use a routing service you have to trust it, it sees all your inputs and outputs.
If a cyberattacker can modify the token stream (i.e. man in the middle your traffic), you should be worried about a lot more than which model you're using!
If one were to listen to /r/claude or /r/codex where half the posts are complaining about models being "lobotomized", one could think that this is the most valuable thing there is!
In all seriousness, this is mostly just one of those defense in depth kind of things. Data exfiltration via a compromised provider is also a very different class of vulnerabilities than prompt injection via malicious tool calls.
But it still requires trust in whoever is running the server that tags version 'x' as x. All you have proof of is that they return the hash for a given set of weights which is the same as someone else's hash, and those tags/hashes will be changing all the time as the models evolve, so people are not going to be able to keep track of whether Anthropic say is dumbing down their model or not after release, because this is a zero knowledge proof, all it tells is that a certain set of weights was used (you have no idea what weights, how many weights etc etc).
So I don't really see this as useful, except in the very limited use case of using a third party proxy who is trying to cheat you (in which case why are you sending all your data to someone you can't trust?).
> and those tags/hashes will be changing all the time as the models evolve
I think that's where your assumption differs from stated claims.
According to Anthropics own docs[0], they don't alter the models at all after release. So for each model release you should be able to prove that a response was created with that exact model without dumbing it down.
Zero knowledge proofs also still allow you to have disclosed inputs. You can make one of the inputs be a publicly commited hash of the model weights, and have the ZKP validate that the inference was done with that committed set of model weights, in addition to all the inferences in the batch all coming from the same model.
[0]: https://platform.claude.com/docs/en/about-claude/models/mode...
If the releases change frequently though, this would be of little use to anyone save those who feel sticking with one model is best. The weights are not exposed so you have to trust Anthropic that changes they make are in your best interest.
>which of these use-cases is valuable for you?
None of these are directly valuable to me. But you asked "Who will find this useful?" and I gave you the answer to that question (provided by Anthropic in this post). Not everything being worked on will be useful for you. It's just bizarre that you'd be trying to make these arguments.
>If a cyberattacker can modify the token stream (i.e. man in the middle your traffic), you should be worried about a lot more than which model you're using!
Sure, so it'd be great if you could know if someone is modifying your token stream. If only there was some advancement in provable inference that could allow you to verify that what you're getting from a provider is what you expect to get from them.
If only the internet had some way to encrypt data in transit between two parties.
This is a solved problem if you trust your counterparty and if you don’t it’s the least of your problems.
All this provides is a tag saying 'x' version (which your counterparty defines anyway) is being used.
Customers care about model output being correct, Anthropic cares about model output being generated by Anthropic model.
This solves for the latter with a bunch of handwaving for the former. The end goal is gonna be FUD for cheaper model routing because that obviously is at odds with their biz model.
You developed your agent implementation and more importantly, evaluated it, against a specific model.
You don't want all you eval work to go to waste just because the discount inference provider you're using decided AcmeModel4.23 is as good as the Anthropics GilgameshEpic7.2 that you used to validate your solution.
We already have trouble enough with model retirement.
Off the top of my head, it is useful when there are content-level proxies between the client and anthropic's API server, of which there are many these days.
> As models become capable of improving and deploying themselves ... how can a company be sure of which weights are actually running on its servers?
I find this a bit odd and backwards. Why do we choose not to make the deployment code to be robust, but instead to observe the model inference as proven? This moves unobservability layer to the system that collects the proofs. Given the assumption that code is fluid and LLMs are rewriting it often, how can we know it isn't being abused against the observer?
Goes without saying, it's also disappointing that this has nothing to do with making the model inference observable for consumers.
Anthropic here is solving one of their own problems, how to make sure you don't combine other company's models with their stuff for example, and make it sound as if they did it for you.
It makes the inference more expensive and helps controlling the users.
I think some model providers are implementing "eyes glaze over" where if you ask a boring simple question they route it to a cheap model but charge you for an expensive model, so consumers may want a proof of work certificate to know they're getting what they pay for.
I think a consumer facing system would be fantastic, but from how they're framing it in the article, it's all about internal company monitoring, with some fearmongering around LLM observability.
I assume this alludes to their prior work with models running on smart TVs being able to setup inference on additional smart TVs. The idea being that you can't be sure that, having gone through the whole deployment process, your hardware has not been subverted to run a different model. Any level of "robust" code you can imagine to prevent this in the traditional ways could be subverted on the fly by a capable model.
> This moves unobservability layer to the system that collects the proofs.
I'd say it's adding security through a secondary system, rather than displacing security to a secondary system. Security by layers.
I've read that providers like those you find on OpenRouter can sometimes route to other models; this would catch that.
But I've also heard, as complete rumour, sometimes some named models perform significantly worse on some hosts and the theory is they're running quantized, without that being published. Would this solve that? The post mentions proving a set of weights, but not how those are represented? What would stop a host using true weights for generating the signature, while running on quantized versions?
> We prove an exact integer version of the published weights, and a proof asserts that each checked token is that model’s greedy choice. A reply from a floating-point implementation, or sampled decoding, could depart at some position...
So, no...?
I think it would catch that, at least for open weights. From the repo docs [1] it looks like this type of attestation is even supported for open weight models: you commit with the public weights and publish the seed, so anyone can recompute the commitment from the Hugging Face files and check the provider is running the exact checkpoint and quantization they say they are. That's what they did with their Maverick demo. That said, it only works for greedy decoding and short contexts for now, and it's 70-95x the GPU time, so there may be cheaper ways to do this like TEEs.
[1] https://github.com/anthropics/FlashProve/blob/main/docs/what...
Maybe it's possible to tell by looking at the size of that error?
> The integer version is generally expected and empirically tested to be within some small error rate of true floating-point inference generically (even taking into account non-determinism of floating-point computations), though this does not cover adversarially designed prompts.
Interesting, but I'm not sure the property it attests is one we care about that much. Or rather, it's one you'd expect Anthropic not to tell you it needs to prove. I mean it tells you that you're not being stiffed by Anthropic using a cheaper model and that they haven't literally been MiTMd, but that's kind of table stakes. Otherwise it's not telling you much useful. It's not telling you that your data hasn't been exfiltrated, and it's not telling you any useful property of the answer.
> I mean it tells you that you're not being stiffed by Anthropic using a cheaper model and that they haven't literally been MiTMd, but that's kind of table stakes.
That's literally the most common gripe I see from Anthropic users and, so far from being "table stakes" there no existing mechanism by which those users could verify it is or isn't happening. If Anthropic can figure this out then most of this "is it just me or is Opus hurr durr today" noise can evaporate.
We're working on provable inference at Attestable (https://attestable.com/) - our solution is one of the comparisons provided.
We've published initial performance metrics in August of 87 t/s on Muse Glimmer 30B (a dense model) - orders of magnitude faster than existing OS projects. Antropic's FlashProve supports only a fully integer model with a small context length (significantly hurting evals).
There is significant progress since then, and we'll publish more results soon.
This obviously an attempt at being able to say a response came from an anthropic model vs say some other model via a model router. As for me, I will keep using my model router, I don't care to be charged more for a name brand. I don't care about the brand at all, it's a commodity.
until your company gets sued for using Dangerous and Shoddy Foreign models and the boss says you can only use the Trustworthy and Premium Aligned labs' models.
It feels kinda yucky reading through a document that spends half the time saying how this great for our regulatory capture efforts and how they won't have to show anyone the wieghts. It's not even a bad thing, you want to know you are using what you think you are, but they are advertising the side effects as features.
In modern sv-wallstreet corporateness you need to be bashful over any attempt to do good for good’s sake.
Are they doing that because they are scared that Moonshot might route queries to Claude?
My PhD is on auditability of AI systems, and we are exactly working on the engineering of ZKP in this context. I can confirm through our industry partner (AI Assurance and auditing), that black box auditing is indeed a huge problem for them, and snarks have shown their potential in this context. It's great to see Anthropic confirming our early results.
I'm blind to the problem being solved here. Can you motivate it? Why do I need more than normal HTTPS on the token API?
HTTPS makes sure you’re communicating on a secure channel with the API endpoint. ZKPs guarantees that the computation happened correctly behind the endpoint.
The work from Anthropic isn't a new idea, but they made the process more efficient, as inference verification is one of the biggest bottlenecks in the AI lifecycle. Beyond that, there's a lot more that can be proven, like showing a model was trained on a specific dataset with certain properties.
From an auditing perspective, this used to be handled either by showing the data (which is pretty rare), checking samples (which has been shown to be easily hackable), or similar workarounds. For inference, an auditor might want to interact with a model on demand and have absolute certainty that the model they're talking to is actually the one declared, and not a substitute.
I’m not sure what you mean by correctly here - do you mean with the model expected? That’s not related to correctness.
Pleased to hear somebody cares about this. What is it a bottleneck for though? Are people actually doing this right now in non-academic settings? I had the impression this is not in fact usable with larger models.
ZKPs verify "things" by encoding the verification in a computation, and verifying that the computation happened in the right way, means to assess it was computed correctly, i.e., with the right inputs, the right operations, etc. So this is what I meant with "correctness".
The bottleneck has two sides: first, there's a technical bottleneck. ZKPs are quite expensive, and finding the right, production-ready protocol is challenging. The other side is a socio-technical bottleneck. There is always quite a large friction with adopting these kind of technologies, also due to the first bottleneck.
As you said, it is difficult to adopt because of how fast the complexity grows with big models. It's great indeed to see that Anthropic is tackling the challenge.
> guarantees that the computation happened correctly behind the endpoint
but why do I care about that? can you some example "user stories"?
So, you got beaten to the punch?
Well luckily not really, their work is more foundational-y, the idea is not new, but they made it more efficient and practical. On our side, we work more on the integration of these protocol in production systems, but we are pretty much on the frontier and is not very common to see such big companies experimenting with this technology. Microsoft already tried something [1] but I'm not sure if it ever went further than a research prototype.
[1] https://arxiv.org/pdf/2402.02675
So can we get proof that the inference was done on the model we actually paid-for and not some nerfed version.
What’s the point of a SOTA model if, by 3 weeks later, it’s dumber than the previous generation at launch.
(If you did this random substitution with physical goods it would probably be illegal or very frowned upon…)
I'm a little confused how this is even feasible to do in general. Most inference is batched and often implemented on heterogeneous hardware. In this environment, it's hard to get bitwise identical results, much less prove that the results came from a certain model. Is this actually practical? In the post near the end it doesn't seem like the authors think they have done anything for this and it seems pretty important for it to be used for any production environment.
This sort of result is about steady progress. It's not practical, yet. This system only allows for 1k context, not even enough for a system prompt. But the authors have contributed several orders of magnitude of speedup on the current state of the art.
I wager regulatory capture lies this way. Look for this becoming an in-hardware feature, with big fat patents on it, and then a push to demand all inference providers use "TPM chipped" hardware.
Attestable does great work on the subject: https://attestable.com/blog/proving-llms-scale
This is built on VerInf, work that I did as a MARS fellow. They ripped everything down to the choice of 400B Maverick from us with very little attribution. Cool, I guess... https://github.com/jamespetrie/verinf
Not sure if this is sufficient attribution, but TFA at least mentions VerInf:
> VerInf proved Llama 4 Maverick (the same 400B-parameter MoE model we prove) at ≈0.02 tok/s.
This is an exaggeration. The commitment scheme is related but different (Ligero vs Zook). Matmuls are different (Freivald vs Sumcheck), lookups are different (LogUp vs Grand Product), the packing is different.
What original ideas are in VerInf that are also in this work besides the choice of benchmark?
I'll concede that "ripped everything" was written as a strong emotional reaction. But you are also exaggerating; there were a lot of novel ideas that we worked through used here. I'd point you to our choice of UD-Q4 and rules for how to convert to integers as a starting point.
You should have used Bedrock...no sharing of inputs with model providers ;-)
Claude being a official contributor here gives me the shivers. This feels like setting up a case for: We have the rights for everything you do with Claude. Or rather what they do with Claude for you.
In that case they should cite every author of every source used to train their model that enabled this result.
I'm just annoyed that they treat it as a person. Claude isn't a contributor more than that shovel I used to dig a hole in the ground is. Even if it's a really good shovel.
Sorry to hear. Just asking, why did you choose MIT license over something like GPL? It's not that they couldn't have copied you, but you would be in a different position vis-à-vis them now.
I would've chosen GPL personally, but a prototype repository already existed under MIT by the time I joined as a contributor through the fellowship. Alas.
It's actually fine to use MIT code and publish it under GPL, or to switch from MIT to GPL. You can still do that today on the repo, btw. It's the other direction that is thorny. Just fyi
...thus implying that you'd rather the preeminent safety-focused lab not have used your interpretibility research, as they're technically a for-profit company with various closed-source assets?
I am certainly biased by a passionate distaste for IP law in general, but I am honestly curious to hear the reasoning behind it in this case.
In case it helps: this is a "research prototype", so I'd assume that seeing one or more aspects used in a highly impactful paper would be about the best possible outcome. Is this about publish-or-perish stuff, perhaps (e.g. citation "importance" for the algos)? Or maybe you're okay with this in general, but are regretful because it's Anthropic in particular?
If they are closed source and for profit, can they really be described as safety focused…
Why can't Anthropic give credit though?
I think I misunderstood GPL per a comment above so it's all kinda moot anyway, but my original understanding was that it was a copyleft license, so wishing it was GPL would be equivelant to wishing Anthropic had not used that work at all.
Attribution is certainly polite, and I sorta doubt Anthropic of all groups is intentionally witholding it for h-score or prestige, but that's all a much lower-stakes conversation than potential license violations. AFAIU.
Who said they didn't? Not sure what "very little" attribution means, but presumably it doesn't mean "none."
OP was pretty clear they wanted attribution, so you have hallucinated an entirely novel implicated desite for exclusion.
Seeing as the question posed out of "[honest curiosity]" to someone else is then immediately answered by yourself host without you even pausing to put on a sock, I think your honesty is very questionable.
Oh sorry, I must have misunderstood! My bad, honestly; I was under the impression that GPL vs. MIT was about copyleft stuff, which would seem to preclude Anthropic's use of that project altogether. Not 100% sure how it relates to attribution still but I'll have to look into it I guess.
I don't really understand what was "answered by [my]self host" nor what a "desite for exclusion" means but it seems beside the point I think. Also not sure what the alternative to curiosity is. Like, I'd be trying to entrap this prestigious researcher in some legal sense...?
Doesn’t matter. AI bros don’t care. AI wash it and release it closed. What are you going to do? Sue them?
Regarding attribution, I guess he is one of the copyright holders, so his name should be included in the new license. Unless we are talking about MIT-0.
Im not an expert on these things, but I think there is another type of attribution, that I guess is what the commenter is more concerned about, and that is academic attribution. It is part of academic culture that you attribute significant influences in the articles you write. There are no licenses as such for this, as far as I know. You do it because that is what you do in academic circles. If you dont, you are not in the club.
No, unfortunately. In this era of AI rewriting entire projects from scratch, copyright protection for software is effectively dead.
The question is less about protection than enforcement, I'd say. Courts may come to very different conclusions from what tech people would expect.
Protection and enforcement are kinda two sides of the same coin in most IP law. IP essentially grants you exclusivity for your works, and your protection comes from how well you can enforce that exclusivity. As an example, piracy exists largely because copyright holders simply can’t enforce their exclusive rights to distribution effectively on the vast, distributed Internet.
It’s still early days, true, and courts may find legal reasoning to address this... But I fear this stems from a fundamental yet little-known flaw in copyright for code, based on a legislative hack, which can only be made worse by caselaw. I ranted about it a few days ago: https://news.ycombinator.com/item?id=50320875
A small academic lab is going to sue a trillion dollar company with any sort of success?
Yeah, enforcement is the much bigger issue. If the IP got bought out by a giant lab, the situation could change quite substantially.
The claude contributeth and the claude taketh away.
> MIT license
Live by the sword, die by the sword...
I wonder if this is partially letting agents.. agent, agentically and not know where ideas come form or how they come together, only whether it works or not.
I’m so tired of hearing stories like this. The biggest most well funded corps in the history of mankind and all they do is steal the shit out of everything. It’s pathetic.
I hope you don’t get that wrong, but maybe your best option is to see it positive. The top comment on a first page post on HN, gives you some publicity. Potentially more than if that would’ve not have happened.
You can also use this and create explanation video on it on Youtube, post on X, Bluesky and other in-detail stuff that only you and very little people are in the position to do.
I’m sure some are interested, and this could lead to a hire, too.
I know it’s always easy to say and I probably have less adversity faced as most people, but I really try to see them as energy enablers. Of the almost infinite options you have in life there is a big chunk of realities that are better than your current ones an adverse event can trigger emotions and therefore energy and push one to do things one wouldn’t have done in comfort.
Thanks for your work!
I appreciate your positive energy here a lot. And I agree; all things can be dichotomies. This is how our entire team reacted to the news: it is good news and bad news both.
I agree with this. I often dedicate my life for months on end to prove novel and potentially valuable ideas just in the hope that one of the largest companies in the world can rip it off so I have the prestigious opportunity to make a youtube video about it. I can't believe that everyone doesn't do this.
Well I’m not saying to justify that, you can also start an activist movement to fight that, but I would rather move in the worlds that I can affect than in the one where I complain with no impact.
Guess I'm making a video about the system! My younger self would be proud about this... any tips?
I’ve only done a bunch of "thinking" videos. Nothing special, so might not be the best advice giver, but I’ll share what I learnt.
Don’t try to achieve any goal with it, except doing it. When doing it you’ll eventually get better in making videos.
The first video takes way longer than you think it takes, but the second probably takes half the time and it will be tremendously faster over time.
Try to reduce the friction on recording. My first training video on my app took me almost a full day to set up and plan with teleprompter and everything. After a couple of time I could do 3 in a row.
Making videos is still not "my thing" that I enjoy most about, but I’m still happy that I did it, if that’s what I need to I’ll do it. Right now coding has more leverage for me.
A technical write-up or a blog is an alternative. Or even reaching out to Labs. You could reach out to OpenAi/Anthropic/Mistral or your 'poison' of choice, or companies that do something similar. But this is now on your CV.
I guess if you’re “motivated enough”, you can [try to] put a positive spin on almost anything.
That improved my wellbeing
Sure like any other method akin to burying your head in the sand, this could be “helpful”.
i guess the idea here is that if the end user cannot discriminate based on capability above some threshold, the premium that you can charge for more capable models either disappears entirely (if the end user is the principal source of truth on quality) or gets arbitraged down (if the end user is merely an agent of the principal source of truth on quality). to maintain that premium in perpetuity, you just need (a) trusted benchmarking services that can distinguish between premium and generic, and then (b) a way to prove to the end user that a given output is premium.
so this solves for (b).
I really don't like that the first more elaborated real-world case is to stop people from training their own models. (Challenge 1)
> The goal of provable inference (PI) is to enable a user with limited computational resources and without any information about the weights to verify that a response came from the requested model
Isn't this already solved by watermarking?
This page said they would provide an API to check whether a given text has the watermark for a Claude model:
https://www.anthropic.com/news/claude-text-watermark
That's two entirely different types of things that are being verified.
Watermarking checks whether a _specific text_ came from a provider.
PI checks that the text (that you already know came from the provider) was produced with _a specific set of weights_.
PI is mainly useful for AI supply-chain security.
Not to the degree needed for actually verifying the things this verifies. You could probably quite easily produce a non-Claude model that generates text that conforms to Claude watermarking.
In the page I linked above, Anthropic explains that they use the techniques from SynthID-Text, where a secret watermarking key is involved. Only Anthropic knows this key, so only Anthropic can produce and detect the watermark. (But they also promised to provide an API through which other parties can check the watermark.)
SynthID-Text paper: https://www.nature.com/articles/s41586-024-08025-4
I think there are two gaps with the watermark:
- it only gives a high probability that the text was outputted from the model, whereas a crytographic proof is computationally infeasible to forge
- it cannot be traced back to the input. You could "wash" the output from one model into another and it would be watermarked by the latter. With the crytographic proof, if you have the input, output, and "commitment", you can verify your exact input was processed by the model (or so the model provider says).
Very cool. Now prove you are unable to read chat transcripts with provable end to end encryption down to the CPU registers and TPM endorsement chain of the software involved.
Provable privacy via remote attestation effectively gives you provable inference for free.
What are the tradeoffs of an approach like this vs confidential computing primitives and attesting everything from untampered hardware all the way up to the served request? It seems more performant. Is it simply that they would have to reveal too much?
More so that attestation requires the model to be running on secure hardware, which is hard to do across all models that a company like Anthropic serves.
In part because many of the accelerators they use may not support attestation, and even with ones that do, such as NVIDIA Blackwell, support is restricted to a single node (8 GPUs). Vera Rubin supports rack-scale encryption, but that is not broadly deployed yet and will not be the only accelerator they use to serve inference.
> A user querying a model today cannot check that the responses came from the specific model that a developer claims is responding
Indeed, I wonder what kind of developer would make users worry about that...? Funny PR stunt coming from a company that was in the news for silently routing ML-related requests to an inferior model.
https://fortune.com/2026/06/10/anthropic-accu-claude-fable-5...
> Unlike Fable’s other restrictions, such as around cybersecurity and biology, which openly redirect users to a less powerful model with a visible notification, the system card emphasized that this is “not visible to the user.” The model still responds, but uses “interventions to limit Claude’s effectiveness” without telling the user it’s doing so.
"The commitment to the model, and subsequent inference proofs, hide the weights of the model; hiding the architecture of the model is not a goal of this system."
Your Claude is showing.
isn't model substitution a vendor accountability thing? and token tampering, is that not infra security?
do either of these need 70-95x the GPU time in "zksnark" proofs?
already solved provable inference with encrypted chats on trustedrouter (using TEEs which most people don’t know about)
What a great way to enforce a (still hypothetical) open models ban
> Some proposals to regulate and pace AI development rely on monitoring hardware and how it is used. For example, a provider might be required to devote its hardware to inference rather than to training a powerful new model. The provider might try to cheat by serving cheaper models or using compute in an unexpected way, freeing up resources for training. How can a regulator reliably ensure that the hardware is used for inference?
Is this even a real problem? Forcing a company to only do inference?
Sounds like Anthropic's dream scenario