The attack is a ROME-style edit (Meng et al., 2022 - the method behind PoisonGPT). It makes a small, targeted change to one weight matrix so the model becomes wrong about a single chosen fact while staying accurate at everything else. What makes it worth writing up is where we ran it: not as a pre-edited checkpoint you could scan, but at runtime, from a malicious Python import that patches PyTorch and applies the edit once the model is loaded on the GPU.
The consequence for defenders is that none of the usual evidence exists. The checkpoint on disk is byte-identical before and after. No process spawns, no file is written, no network connection opens, there isn’t even an extra ioctl or mmap - the edit is ordinary CUDA work issued by the same python3 process on the same context as inference. Host EDR, file integrity monitoring, and network detection all watch layers where, in this attack, nothing happened.
So the post is really about one question: when the malicious operation is numerically indistinguishable from normal inference and leaves no artifact, what signal is left?
We go at it two ways:
1. The first is tying each CUDA kernel launch back to the Python call stack that issued it - the same kernel is benign from one caller and suspicious from another, and the edit also fires kernels (an LU solve, a GEMM) that a clean inference run never launches.
2. The second is hooking where PyTorch records in-place tensor writes, so a weight change carries the call stack behind it and a loader write is distinguishable from a mid-run edit.
One honest note on maturity: the kernel-launch/caller-stack detection is something we have working today; the weight-mutation work is earlier-stage research we’re sharing rather than anything finished. The demo also uses a small transformer - the mechanics don’t change with model size, but I don’t want anyone thinking we benchmarked this on a 70B.
The part I’d most like feedback on is the hard edge of the second approach: plenty of legitimate code writes weights in place (load-time quantization, LoRA merges, optimizer steps). The call stack separates a load from an edit cleanly in our examples, but I’m interested in where that heuristic breaks in real serving stacks. Happy to answer anything.
Curious.
This feels like a tool to add to an LLM, being able to modify it's own weights, to be able to relearn, refine its own understanding.
The attack is a ROME-style edit (Meng et al., 2022 - the method behind PoisonGPT). It makes a small, targeted change to one weight matrix so the model becomes wrong about a single chosen fact while staying accurate at everything else. What makes it worth writing up is where we ran it: not as a pre-edited checkpoint you could scan, but at runtime, from a malicious Python import that patches PyTorch and applies the edit once the model is loaded on the GPU.
The consequence for defenders is that none of the usual evidence exists. The checkpoint on disk is byte-identical before and after. No process spawns, no file is written, no network connection opens, there isn’t even an extra ioctl or mmap - the edit is ordinary CUDA work issued by the same python3 process on the same context as inference. Host EDR, file integrity monitoring, and network detection all watch layers where, in this attack, nothing happened.
So the post is really about one question: when the malicious operation is numerically indistinguishable from normal inference and leaves no artifact, what signal is left?
We go at it two ways:
1. The first is tying each CUDA kernel launch back to the Python call stack that issued it - the same kernel is benign from one caller and suspicious from another, and the edit also fires kernels (an LU solve, a GEMM) that a clean inference run never launches.
2. The second is hooking where PyTorch records in-place tensor writes, so a weight change carries the call stack behind it and a loader write is distinguishable from a mid-run edit.
One honest note on maturity: the kernel-launch/caller-stack detection is something we have working today; the weight-mutation work is earlier-stage research we’re sharing rather than anything finished. The demo also uses a small transformer - the mechanics don’t change with model size, but I don’t want anyone thinking we benchmarked this on a 70B.
The part I’d most like feedback on is the hard edge of the second approach: plenty of legitimate code writes weights in place (load-time quantization, LoRA merges, optimizer steps). The call stack separates a load from an edit cleanly in our examples, but I’m interested in where that heuristic breaks in real serving stacks. Happy to answer anything.