Until ~2000, "cellphone user in public" equated with "smug asshole." We're at that icky rejection stage with AI.
Two years ago, that writeup would have been viewed as a generous gift of time to the community. Now we can't be bothered to read through it because we know where it came from, and that it's just worth $0.06 in tokens. Another reason (I think) is because we know what it portends.
In a few years, digging manually through bugs will probably be a last resort. We'll go from sneering to yawning at the reports, as another AI agent reads them and verifies the fix. Just like we ignore compiler-generated assembly code and people on their cellphones.
I get why people don't bother replacing the default allocator from musl all the time (it's there, convenient). But in an application whose purpose is to be FAST, I find it weird they haven't bothered replacing it with another more performant one.
mallocng is bad at dealing with contention during multithreading. I've had applications that usually were I/O bound suddenly become "malloc" bound when building with musl in multithreaded scenarios (and only just 8 threads). Switching to mimalloc improved performance by 20x, very close to what glibc offers by default, and just a bit under a glibc + mimalloc configuration.
I get that there's a real issue there and it's interesting (to some) to address it, but it should have never surfaced this way in the first place.
That's good! But I wonder why it wasn't then enabled for that configuration or why the override wasn't "global" enough and the default allocator was still partly used.
Rust doesn't get to override musl internals. Ripgrep uses opendir, a POSIX library feature implemented in musl to look at er, directories. The stack trace suggests we blew up when Rust's std::sys::fs::unix::readdir internal detail called opendir, and it in turn allocated.
On Linux it would be possible for ripgrep to talk directly to the kernel via documented system calls without libc, but that wouldn't work on any other popular OS.
Last time I checked (a while ago), it wasn't. You needed to compile musl and exclude a few files from the archive to then provide your own allocator.
Depending on the toolchain, I'm assuming you could use some tricks or hacks to make it work better, but the tooling I used (Bazel) did that automatically and I haven't bothered to look at the internals yet .
I assume it already doesn't work on Windows. At some point you have to define your compatibility boundary. And high performance often coincides with mediocre compatibility.
You can call into the kernel just fine in Windows. The fact you use a function call wrapper instead of a raw syscall is not really relevant.
Most Windows NT kernel functions take a buffer to use rather than allocating their own memory. Even many Win32 functions don't allocate (although there you have to be much more careful).
A good example is go. On linux you can use a from scratch image fairly easily because it only uses syscalls. But for windows or mac the moving target wasn't maintainable so they link against shared objects. Linus enforcing the don't break userspace rule is what made that possible. That definitely has tradeoffs. At some point relibc or something similar will allow the same (stably) for rust. But using posix as that compatibility boundary gets you a much larger set of OS and only occasionally has a performance penalty. Often, a posix api tuned to the kernel is more performant. Musl is an exception precisely because it makes an openbsd-esque trade of performance for simple small attack surface.
The Linux kernel defines syscalls as its stable ABI (note: there are also non-kernel ABIs on Linux, such as Wayland) while Windows defines the DLL calls as its stable ABI.
One isn't better than the other. Linux's approach allows binaries to be fully statically linked, which is a more predictable environment for binaries, but Windows's approach composes better, as every process loads DLLs and this allows for things like graphics drivers and COM to work more reliably. As things stand on Linux you can't use the GPU in a portable statically linked app, because the kernel doesn't define the semantics of dynamic linking.
Notably, Linux has this too, but it's a special mechanism (vDSO) while on Windows it's just a normal function call with no difference from any other function.
If you look at the sigsegv stack, the allocation comes from opendir which is in musl libc as well. The allocation override mechanism used in Rust doesn’t replace the allocator process-wide; it merely replaces the allocator Rust code talks to.
It does seem like ripgrep should probably avoid using opendir from libc if it allocates using an allocator with a global lock though.
It’s a kernel bug. While I agree libc allocators suck for no good reason, it seems like this work of equally likely hit other application code including mimalloc and glibc.
Sure, but the problem isn't that the bug exists, is that it has surfaced in a performance application using musl and exerting code paths in a slow allocator.
it's a tradeoff; mallocng does have benefits as well -- for example it uses less than half the amount of RAM compared to mimalloc and glibc for certain Python workloads. While yes it is a bit slower, I prefer mallocng in many cases.
I imagine there is no free lunch in the allocator space, but a series of trade-offs. Pick your poison on implementation which is going to be sub-optimal for some subset of use cases.
Without a hugely compelling reason to switch, going with the default is reasonable.
ripgrep is there to be FAST. Trading any speed to improve memory efficiency over a longer period of time for a process with a short life-time doesn't make sense in this case.
It's also a development tool. If your development machine is having RAM issues because it's doing a grep, you have bigger problems to solve.
As for other workloads that might use less RAM with mallocng compared to other performant ones, I'm curious to know about the magnitudes we're talking about. From 10MB to 20MB or from 100MB to 2GB? How was the speed of the program? Was there any multithreading involved?
For context, the Python workload was a general-purpose fileserver with file indexing and image thumbnailing, largely IO-bound. Switching from mallocng to mimalloc resulted in a slight speedup (150% of baseline), but over twice the memory usage, going from 250 to 670 MiB, primarily for thumbnailing. Some pathological cases (libvips calling imagemagick to decode heif) was a 10x multiplier.
There was multithreading, and yes, mallocng visibly became a bottleneck beyond 5 threads. However already at 3 threads there was diminishing returns for both allocators, so this was not an issue in this case.
The speed gain was hardly noticeable in practice since the program was already plenty fast, but the additional memory usage was a very real inconvenience.
Is that memory increase seen in peak usage or average/idle usage?
For file servers, I could be willing to give up 350 MB of memory temporarily for large-scale indexing and thumbnailing operations if it doesn't happen often and makes the file serving/browsing more responsive.
On the other hand, plenty of NAS/file server systems are going to have paltry CPU and memory. Running slower with a more modest memory foot print might be the only way to keep the system stable.
The way most programs achieve being fast is by re-using allocations. You don't need a fast allocator if you don't allocate. Nothing of what ripgrep does inherently requires frequent allocations.
The ISO C definition of opendir() requires an allocation in practice because it returns a DIR* and it's bad practice for the library to arbitrarily limit how many of those an application has at a time.
Maybe a C library could preallocate several DIRs and only use the heap when those are exhausted, but this ripgrep use case (lots of threads running in parallel on a large tree) would still be likely to trigger that.
There is no ISO C definition of opendir(), that's a POSIX thing.
But yeah, those DIR objects which opendir() returns a pointer to are probably some kind of heap allocated. But we're talking about a system call involving the filesystem here. The time taken by the allocator is gonna be dwarfed by the time spent in the syscall even with the slowest allocator.
Ripgrep reads through every byte of most files and matches it against a regex. That is the tight inner loop where you want to avoid allocations.
Not that you even need to call the C functions. I don't think ripgrep would gain anything from it, but the syscall to read directory contents just needs a file descriptor.
I think ripgrep does avoid allocations in that innermost loop. That's why this report says all the crashes are associated with opendir() and its memory allocation: the opendir() call is outside of ripgrep's innermost loop but still runs often enough to trigger the race condition.
That's to be properly measured. Assuming that the allocator will not cause issues there is to be proven.
The application I recently improved by changing the allocator had a similar profile (a C++ include scanner) and thread parallel I/O functions had terrible performance originally with mallocng. Adding threads almost had negative value because of the contention.
- The musl allocator is only slow with multi-threading.
- Almost all other allocators have trouble reclaiming memory when using multi-threading. This often results in multiples more RSS than single threaded or musl's allocator.
Agree with you on mimalloc. It can even be configured to be aggressive in memory reclaim at the cost of performance.
Anyone running ripgrep on a an HPC cluster against a large cluster filesystem needs to stop and redesign their workflow. This generates high amounts of small I/O which is the Achilles heel of any large cluster filesystem. You are exporting your workload onto the metadata mechanisms of the filesystem rather keeping it within the higher bandwidth capable memory subsystem on your cluster. It doesn't take but a couple users running these types of jobs simultaneously to bring a high-bandwidth filesystem to its knees. Just stop it already.
Similarly, I've often wondered if this is the root cause of why GitHub has become fragile of late. All of a sudden you have operations on billions of tiny files that are amplified by AI usage. And object graphs are so fragmented by nature that it's hard to e.g. prefetch a page into memory and ensure that common git operations mainly touch that page's set of objects.
> All of a sudden you have operations on billions of tiny files that are amplified by AI usage. And object graphs are so fragmented by nature that it's hard to e.g. prefetch a page into memory and ensure that common git operations mainly touch that page's set of objects.
In theory, that's what git's packfiles are for: loose objects get packed together into a file, and if the file is well-arranged then objects that tend to get read together will also end up getting paged-in together (see also: https://www.kernel.org/pub/software/scm/git/docs/technical/p...).
>Headline. The crash is real and reproducible. With musl instrumentation we pin the in-process mechanism precisely: a thread's own store to a freshly- faulted anonymous page becomes invisible to that same thread's reload ~10 instructions later, because the page's backing is replaced mid-function. A pagemap read at the instant of the fault shows the backing is the kernel's zero page. A captured core dump confirms the crash site with matched virtual addresses. The mechanism localizes to the interaction between the per-VMA-lock anonymous-fault fast path and a concurrent munmap's TLB shootdown. A source-level review of Linux 7.0.12 identifies a specific race in that interaction, and a git comparison across v6.19/v7.0/v7.1/mainline identifies the v7.0-introduced change on the munmap-teardown side that widens it.
"a thread's own store to a freshly- faulted anonymous page becomes invisible to that same thread's reload ~10 instructions later, because the page's backing is replaced mid-function."
-> ??? What's a "backing" of a page? Freshly-faulted? Fresh fries?
"~10 instructions later"?
"A pagemap read at the instant of the fault shows the backing is the kernel's zero page."
-> Backing?
"The mechanism localizes to the interaction between the per-VMA-lock anonymous-fault fast path and a concurrent munmap's TLB shootdown."
-> How can a mechanism "localize"?
And more.
This is... words strung together. Nothing more. I wonder how people read and make sense of this.
I'm becoming increasingly allergic to the performative writing of LLMs.
The worst part is how compulsive LLMs are at writing like this. Like the system prompt instructs it to "reason" and like a college Sophomore it pontificates and quotes Nietzsche to fein intelligence and orginal thought.
I love LLMs for technical stuff, but their writing style is horrendous. I also think it's bad form to waste fellow humans' time with walls of generated text.
I understand that it's probably a hard technical problem to get models to conform to a style without sacrificing performance in other areas. But I really wish AI companies would spend more effort getting them to write in a neutral, concise way and not like a middle manager on 80mg of Adderall.
Definitely easier. It's the same for humans. You can tell that someone has little of substance say when they cloak everything in big words and elaborate metaphors.
This is technical writing for kernel developers who already know how memory management works. If you want to understand it better, you should study up on the x86 MMU and the kernel memory management subsystem. I'm not a kernel expert but I found it pretty understandable, so I'll try to explain it.
MMU: The memory management unit, part of the CPU designed to allow an OS kernel to flexibly control how memory addresses used by a program map to physical RAM.
Page: A 4k region of memory addresses.
Anonymous: The page is just regular old memory used by a single process for general purposes, it's not part of e.g. a memory-mapped file.
Backing: The memory corresponding to a page. The kernel can direct the MMU to map any page to any part of RAM, or delete the mapping entirely.
Zero page: The kernel keeps a single 4 KB page filled with zeros (the "zero page").
Page fault: An error that occurs when you try to write a page that's not present, or write to a read-only page. Some page faults are normal and expected by the kernel, they support features like swapping, memory-mapped files, etc. Such "expected" page faults are invisible to your program. (On the other hand, an unexpected page fault occurs when your program just straight-up accesses an address it's not supposed to, which will end your program with a SIGSEGV signal -- an all-too-familiar experience for a C programmer.)
When you allocate say 4 MB of memory, the kernel simply sets it to 1k read-only copies of the 4 KB zero page. Then when your program tries to write to a page, it faults. The kernel handles this expected fault by assigning that page an individualized memory region. If a program allocates a lot of memory, it's only assigned memory for what it actually uses, when it first uses it. This is an optimization: Every program that allocates more memory than it needs is wasteful, but the kernel can recover that waste by not backing the allocation with memory -- the program has to prove each page is needed by writing to it.
Store to a freshly-faulted anonymous page: The program stored data to its own memory, which caused an expected page fault.
> becomes invisible to that same thread's reload ~10 instructions later
The thread fairly quickly tried to read the data from the same place in memory it had just written to. It doesn't get the same data back. This is a major problem that causes the program to malfunction and crash.
> A pagemap read at the instant of the fault shows the backing is the kernel's zero page
The expected fault occurred because the program tried to write to an address that was mapped to the zero page, probably because it's the first write to freshly allocated memory. (This rules out other reasons it might be faulted, e.g. the page had been swapped to disk.)
> concurrent munmap's TLB shootdown
The page table is big and slow so the CPU caches parts of it in an area called the TLB (translation lookaside buffer). munmap is a kernel function that removes the backing for an address region. Since munmap changes the mapping, it has to inform the MMU the relevant part of the TLB is no longer valid.
> per-VMA-lock anonymous-fault fast path
This is what the kernel does when the program first writes to a zero page. Saying it's the "fast path" implies there's some (slow) special case handling in the kernel code, but the bug doesn't trigger any of the special cases, we're in the most common case, which the kernel code tries to handle as quickly as possible because it has measurable impact on performance. I would guess per-VMA-lock has to do with how the kernel synchronizes this code (a "lock" is a basic synchronization primitive, you should be familiar with it if you do any kind of multithreaded programming).
> The mechanism localizes to the interaction between the per-VMA-lock anonymous-fault fast path and a concurrent munmap's TLB shootdown. A source-level review of Linux 7.0.12 identifies a specific race in that interaction
The mechanism: What's actually happening to cause the program not to be able to load data it just stored in memory.
Localizes: The cause of a problem like this is a needle in a haystack -- it could be caused by the program, the kernel or the hardware. The analysis has narrowed down the haystack to a specific part of the kernel code.
Race in that interaction: Two parts of the kernel code, (1) The kernel code for munmap and (2) the kernel code to handle a program's first write to the read-only zero page for a fresh allocation. These two parts work fine individually but the problem occurs when they both try to change the page tables / TLB at exactly the same time. This is quite a small, specific haystack compared to "somewhere in the program, kernel or hardware" we started with.
fwiw "backing" is standard jargon in this context, and "freshly faulted" is phrasing you'll see in Linux source comments. The writeup as a whole is indeed slop, though.
This is a pretty clear analysis if you are a kernel developer.
> a thread's own store to a freshly- faulted anonymous page becomes invisible to that same thread's reload ~10 instructions later, because the page's backing is replaced mid-function
A result of a machine instruction to store something in RAM got erased, when the same thread attempted to load it ~10 machine instructions later. The reason is that the physical RAM page backing that particular virutal page got replaced while the thread was running.
The fault happens in the fast path for anonymous (i.e. not backed by files on disk) pages, in the granular per-VMA (virtual memory area) locks.
> "A pagemap read at the instant of the fault shows the backing is the kernel's zero page." -> Backing?
Yes, it's typical kernel terminology. "A backing page" or a "backing file".
> This is... words strung together. Nothing more. I wonder how people read and make sense of this.
There's nothing wrong with this description. It's just written for kernel developers. It needs to be expanded and explained for people who are not.
"backing" is a Virtual-Memory related term. With virtual memory, you can have memory areas (called "pages") that are not really there but only present in the metadata (the "book-keeping", so to say). The first time, someone actually tries to access this page, that access is interrupted ("faulted") and the kernel gets a say in what should be done to that piece of memory (i.e. load it from disk somewhere, reserve actual physical memory and fill it with zeroes, or crash the process).
The "backing" refers to the physical memory that might or might not be present for every "virtual" piece of memory that your program has allocated.
"Freshly faulted" means that a page of (virtual) memory has just received a "backing" by the process above and is, thus, very fresh in physical memory (even though the virtual memory might have been allocated much earlier)
"~10 instructions later" refers to the assembly- (machine-) code, which, contrary to a high-level language like C, usually has long(ish) sequences of rather simple "instructions". 10 instructions is a rather short interval in assembly code.
As for the "localize", the term used is actually "localizes to" which I read as "turns out to be located in" (probably just a bad English translation by the original author)
While this whole summary reads a bit weird, I don't think it is necessarily the result of an LLM, it's probably just that someone who is rather inexperienced at writing up technical summaries did it.
So the theory is that the page is faulted in, then somehow evicted within 10 instructions, then re-faulted in somewhere else, resulting in writes not making it to the page? That would need a context switch and another page fault to happen in short succession. But the context switch to evict the page back out would have necessitated pending writes to have finished. Nothing actually makes sense with that explanation.
The way I read it was that it wrote to a page and then did a read on that page within 10 instructions and that was not enough time for the memory system to have gone through the process of creating the page backed with real memory and that the timing bug widened in Linux 7.0 such that more things like this exposed the bug. Or there was flapping in the TLB for some reason, or the locking wasn’t correct, or whatever else that the kernel devs will figure out was the cause.
I’m guessing it wrote to a page and read back zero page as the memory was in the mist of being allocated.
But the write should have triggered a page fault immediately in that case, and should have been blocked on the page being actually allocated before resuming.
Sounded like a race condition causing a correct new mapping in the TLB to be cleared away by mistake, reverting back to the zero page mapping it was before.
I understand all the words in it and follow the argument, but it's still bullshit in the technical sense of being written by someone who wants to sound authoritative but doesn't ultimately care if they are right or wrong.
The write-up should have been about five paragraphs: (1) "The script at https://... generates a tree with X million files taking 20 GB total. Running ripgrep 1.2.34, compiled with musl, on my Threadripper 9876 as 'rg ...' crashes about a fourth of the time. With glibc, it does not crash." (2) "This appears to be a bug introduced by commit abcdef1234 in some_vm_call() that allows a race between [...] such that user space can see a corrupt whatever." (3) "The sequence of operations that causes this is: (show two or more kernel threads with lines interleaved to explain the bug)". (4) and (5) as needed to elaborate on those three core points.
But the actual post is long on (fluent) speculation and short on reference to actual code. That's a large part of why it's offensive slop.
Thanks for the explanation. I should have looked up each term a bit more.
So "backing" is the underlying physical memory the page maps to.
So if I were to make sense of that piece of slop, is this what it would be?
"In one thread, a page fault happens during a store operation. The physical memory address is the proper address, which is immediately (around 10 assembly instructions later) replaced by the 0x00000000 memory address by another thread due to race condition, resulting in a crash when the original thread tries to read that page again."
The first thread is the user program, the other thread is the kernel doing things on another core.
Reading from the zero page is legal, that doesn’t cause it to crash. The crash is because of a logic “bug” in the program where the zero it reads back causes some other issue in the program.
I say “bug” because it’s clearly impossible without the kernel messing up. It just stored a non-zero value there.
Nope. It's more complicated. The store operation does not need to trigger the fault, it can happen for other reasons.
What happens here is a bug in munmap() that happens in another thread. It corrupts the TLB, for a brief instant replacing the correct page.
So that the virtual memory page that the crashed thread tried to read its RAM, it gets replaced by the special zero-filled physical page. The kernel uses this zero page as an optimization when it needs to create a region of data filled with zeroes, so it can just map one physical page over the entire range.
It's a really low-level bug that just requires a lot of specialized knowledge just to explain. A better write-up is certainly possible but needs to have a lot of explanations to make it understandable.
I don't care if the cat is black or white, so long as it catches mice.
Translation: if this nails-on-chalkboard LLM spew identifies a kernel bug, which is then patched, its aesthetic qualities do not interest me in the slightest.
Unfortunately, the kernel bug would need to be understood to be patched, and the developed patch itself would need to be reviewed before being accepted, which seems... very hard, to put it mildly, using the nails-on-chalkboard LLM spew report.
The only people who have a need to judge the quality of the report (not aesthetically, I mean) are those few who know enough about the subsystems of the kernel it implicates to do something about it.
My point: does it lead to a bugfix? Good. No? Bad.
Do you have an answer to that question? I believe it's a bit early, no?
The overflow would still overflow; the use-after-free would still use after free; a musl mask race would still race.
Hilarious. Apart from the computer poetry, the conclusion seems to be “it’s something in Linux 7.0 + musl 1.2.5”, although the only reproduction is still on the same physical Threadripper CPU and only sometimes when heavily exercised, so it hasn’t really ruled out a hardware issue.
> It’s the kernel. Nothing a user space library can do should ever be able to call this.
I don't follow. An application might see this kind of crash if it has a bug causing it to access a page while another thread is mapping or unmapping that page. That would be a bug in mallocng, musl or ripgrep. Or, as someone else mentioned, it could be bug in the processor's virtual memory logic that has the same effect. Why do you say it can only be a kernel bug?
I looked at the explanation, and it seems to be at very least plausible. The kernel version bisection also lines up. We'll see if the proposed patch fixes this.
I mean, we are deep into reading tea leaves at this point, but what it seems to have identified is the source file that changed such that this issue can now occur. My gut instinct is that the changed code is still valid and correct, but it exercises the hardware in a different or a more intense way.
The worst thing is that I know there's something interesting in there but I'm too arsed to take the time and decode what data the author must have used to generate that text.
Exactly, it would be so much better if they shared the prompts instead of the output, at least then we'd be able to make some sense of what they were thinking about.
Unfortunately the prompt may have been something like "investigate this bug and file an issue when you think you understand what's going on". The "better just to share the prompt" heuristic falls down when the actual underlying intellectual work was also done by the machine.
(Feel free to pretend that I used some phrase other than "intellectual work" if you dislike seeing it used to describe something done by AI.)
Yeah in that case just closing it and moving on would be better IMO. A sufficiently large fraction of the time the issues with vibeslopped code and/or analysis are bad enough once you start digging into it that it's not worth spending time on. My heuristic would be to simply ignore contributions derived from lazy prompts. I don't think there are very many healthy babies in that bath water.
So having the prompt available would be useful even in the degenerate case.
Maybe it's not going to get any better. LLMs were trained to do two things:
- Mimic human language (and logic, since that's part of what we express with language). This includes code written by humans.
- Write code to solve problems. The figure of merit here is solving the problem, not reproducing anything human.
I'm not an expert on LLMs, but it's not obvious that the human reasoning they are fitting in the first case is going to be anything like the problem solving required in the second case. It was trained to get itself out of a problem, not to do it in a way that a human would relate to.
And we're already running out of human data to train on, while synthetic data has no limit. Bug reports in the future might amount to "fix this because then I'll get a cookie".
A lot of it is that AI can talk with different "voice" but that requires giving it cues, and that costs money and expands context, so people that don't know, don't, and those that do have no incentive to
Nice sleuthing, nonsense explanation. An extra TLB flush is never an error. (The CPU is free to flush whenever it feels like doing so.) The error seems to be that somehow a zero-page PTE was present when it shouldn’t have been.
This sounds to me like either (a) a complex race involving a CPU migration at an awkward time or (b) a bug in the zap path transiently exposing wrong PTEs.
Also, I don’t think the zero page has pfn zero.
If I had to throw a dart, I would guess that direct page table zapping is allowing a CPU to read through a higher-level-paging-structure cached entry to a table that has been freed and reused. Yuck. I’ve debugged one of these before.
The part that doesn't make sense to me is that trying to read through a zeroed PTE should be an immediate segfault, not something that reads back zeroes (regardless of whether the zero page is pfn 0 (it isn't)).
I think the broad strokes are that, due to bad locking in the kernel, mmap() returns a VA that a concurrent munmap() is still in the middle of unmapping. The specifics beyond that seem murky/speculative/inconsistent.
Alternatively, it could be a hardware bug, since afaict it's only been repro'd on one hardware config. (I know there are a bunch of ARM cores with erratas around TLB invalidation)
After more digging, I bet it's a paging-structure-cache flush bug. I've debugged these before, and they're nasty, hardware dependent, hard-to-reproduce issues. Looks like the code that the AI flagged might actually be wrong, but not for the reason that the AI thought.
> Alternatively, it could be a hardware bug, since afaict it's only been repro'd on one hardware config. (I know there are a bunch of ARM cores with erratas around TLB invalidation)
Which raises the question of whether any HN readers have successfully reproduced this?
I've just tried (using his file generation script and the official rg binary he links) on a Ryzen 7 5800X / 7.1.5-arch1-2 with sufficient free ram as the github issue suggests, and still no segfaults after 10 minutes.
I have never, in my entire personal history of kernel development, successfully reproduced a paging structure cache bug. I have stared at these sorts of bugs and puzzled them out.
With this particular issue, if my theory is right, it will depend on all manner of microarchitectural issues, possibly address spare randomization, context switch and migration timing, random speculative accesses that pull things into cache, etc. Yuck.
Not really. It's typical rambling LLM slop-analysis. It may be a correct analysis (I couldn't stomach reading it in detail), but it's a pain in the ass to read. A human doing the same analysis would have written something 1/5th the length.
But it goes beyond this. The whole 2020s era is about cheap ways to do anything and be able to stream it to billions of people. Yet I feel no density in the work because there was no energy spent on it.
Yeah but the thing is that someone tries to waste time of humans.
This is why I hate "interacting" with bots, scripts or AI/LLMs. It just wastes my time, again and again and again. Oddly enough not all humans understand that. About two months ago, a german developer involved with ffmpeg, spam-slopped their mailing list with AI (it was an AI proposal for some change to ffmpeg in the future). He still does not understand why that is a problem.
I think it was GZDoom where the absentee project owner suddenly came back, pushed a bunch of nonsense AI comments, so the entire actual development community just forked it again and left him to play in his sandbox? Now it's UZDoom.
Something similar happened with PolyMC to PrismLauncher but that wasn't about AI, it was an absentee owner who came back and deleted everything he said was woke.
In the history of Linux that I'm aware of (and I've been nose-deep in this stuff on and off for quite a while), these issues are almost never debugged because they're reproduced -- they're debugged because someone who actually understands the messy interactions of the code and hardware in question thinks about them.
A reproducer might not actually be useful because there is basically no way short of fancy hardware tracing to figure out what the reproducer is doing.
What will you do when the prompt was "Figure out the bug and write a report for me". Not saying it was in this particular case, but I think at least in other cases, it will be.
> What will you do when the prompt was "Figure out the bug and write a report for me".
The rational choice would be to cut your losses and stop reading at that point. Once you realize zero effort went into the prompt, there's no longer any reason to read the output. The age old truism still applies: garbage in, garbage out.
If you think it's worthwhile, close the issue with a comment: "please rework this and show your work next time". Otherwise just close it without commenting and move on.
It didn't figure out the bug: the report is largely nonsense. It merely did a bad impression of having figured out the bug, by stringing together several observations into something superficially resembling a narrative. Providing those observations without the gibberish framing might be useful.
The parent's point still stands. In many cases it will have figured out the bug. In many cases it will have produced a small, clean, deterministic reproducer.
In my daily work I see these cases. It does help that the bugs that are filed contain a test case and some analysis by the agent.
It would not help to get ten identical bug reports all saying "I asked my agent to find a bug by prompting it with "find a bug and produce a test case". It found a nasty bug and a really nice reproducer. I'm not including its output here. Good luck!"
If it's an actual bug, I hope the Linux devs will. It impacted a human, and it's a bug regardless of human or non-human touch. Of course, looking at a bug is a matter of priority, always has been.
Why is it unreadable? I actually find LLM bug reports/breakdowns to be far more detailed and concise that classical human written ones. If you read the linked repo it clearly goes it depth where the bug was found, how to reproduce it (and in depth). Most disclosures that are human written don't do this at all, they barely even tell you _how_ to reproduce the bug. Just look at the "3.3 The self-store tear", the LLM clearly describes exactly what went wrong, so you can verify it by hand.
I'm reasonably confident that the author has a reproduction case on their hands. That's easy to directly verify. I'm way less confident in that long and rambling analysis. As fwlr observes, it's not clear that this has been reproduced off the original machine: https://news.ycombinator.com/item?id=49134357
If it is the case that the original machine has an intermittent hardware fault, that analysis is exactly what I'd expect of an current AI. Voluminous and confidently wrong. Also interesting that the AI itself doesn't note this... having been biased by the conversation to consider kernel bugs, in the "cross-machine data" [1] it blames the kernel and doesn't appear to consider alternate explanations.
It doesn't get mentioned much around here but one major issue with AIs today is that they fall into tunnel vision very easily and often need some help getting out of it. There's a structural reason for tunnel vision built into their context limits, and there's also situations like this, where if you bias them in a certain direction they can get monomaniacally stuck on it. You can also see it where they do things like try to do something on your system like view a certain file, then if for some reason they fail due to permission issues, if you don't stop them they'll go absolutely insane trying different ways to access the file, rather than just stopping and saying "Hey, I need help". Even if you prompt them directly to do that, it only helps, it doesn't fix the issue. I suspect the RHLF training they get to be better coding agents reinforces this behavior because in the coding quality tests they're rewarded for one-shotting all the various benchmarks, where that "bull in a china shop" approach works better than giving up. But I'd actually like them to give up a bit more often. I've had the same problem with some particularly go-getting junior devs, too... I appreciate the ambition and I look forward to harnessing it in other situations but I'd rather you didn't spend five hours to create a terrible work around to something I could have gotten you in two minutes. For the junior dev, it's OK, they take the feedback and adjust... the AIs never adjust.
That reminds me of one ticket I had to handle. It was generated by an agent from a customer report and the different hypotheses were wrong. Also a huge wall of text.
The issue was alongside the API boundary (timezone shenanigans) but the agent invented a lot of reasons with reasonable looking arguments because of the tunnel vision that there should be something wrong inside the project.
Because any writing needs a core intent they need to convey, which you can summarize down to according to the audience and why it should be important to them. Kinda like the same idea of elevator pitches, “explain like I’m five” and tactical reports when there’s a time constraints.
You got none of that here. It’s just realms of text.
Maybe the person writing the report isn't an expert in this domain or doesn't have the time to commit to it? From my point of view as long as the information is accurate and reproducible, it's valuable.
The other part is that almost no one would have the energy to spend hours instrumenting and rebuilding musl, reading the kernel mm code, and thinking hard about how they might interact.
Claude is probably not right about the root cause here and is probably bsing, I agree. But it's collected enough raw data to point some expert humans at the right interaction. I'd take this bs bug report and start asking Claude some questions that would guide it to a more plausible actual answer, but that requires a little more experience in kernel development.
> The other part is that almost no one would have the energy to spend hours instrumenting and rebuilding musl, reading the kernel mm code, and thinking hard about how they might interact.
Maybe because it isn’t needed. That’s the reason we have experts and professionals, because it’s more economical to use them than for everyone to start from scratch. It’s easier to go to a mechanic shops to fix my engine block than to try to do it myself. I’ve heard that a lot of shops charge more if you said an amateur try to fix things first.
It’s a lot of stuff, but that’s not how you experiment for an hypothesis.
I've been a systems dev for a long time and have debugged many such tricky issues before. Even on teams of experts, this type of issue would take someone really capable weeks to track down, especially because the repro environment is hard to obtain. And the opportunity cost of doing this type of work for a rare bug is high, since you may never get to an answer and have burned a lot of time. So in practice I solved most of this type of issue early in my career before I had a family and before I had higher level work to do.
The ai doesn't have the same incentives as a human. It can also be applied to the repro where it exists.
Your mechanic analogy is a bit inapt. Sure they're experts and can replace your engine much better than you can. But tracking down a non obvious system level bug like this one is much less repeatable and much harder because the problem has already been filtered through a lot of reviews and testing. As a system becomes mature, the bugs become harder and harder individually to track down.
> Even on teams of experts, this type of issue would take someone really capable weeks to track down, especially because the repro environment is hard to obtain.
> As a system becomes mature, the bugs become harder and harder individually to track down.
What I'm talking about is that experts knows the system mechanism more than amateurs, so any hypothesis and experimental setup will be more focused and thus more economical than any amateurish one. And there's the matter of knowledge not present in some docs or other forms, such as past experiences.
Even with LLM tooling, we've seen the rises of harnesses and helper tools instead of relying on generation for everything. Who would you trust to build such harness, a domain experts or some random guy off the street?
> From my point of view as long as the information is accurate
That's the trillion-dollar catch, isn't it. LLMs love to write 30 paragraphs about some plausibly-correct-sounding explanation that is just as likely to be completely fucking wrong as it is accurate. The bug might be real, but that doesn't mean this analysis is accurate, and trying to figure out where the LLM went off the rails can be a nightmare. If you can actually understand the bug, it doesn't take 30 paragraphs to explain it. I would throw this bug report into my junk bin if I were on the receiving end of it, and I say that as someone who will spend days troubleshooting any issue a user will help me diagnose even if it only happens on their machine.
I'd rather read walls of AI slop than soulful meatbag bickering. Just because you might have a soul and intent doesn't mean you do anything useful with it. All I see is the intentful invention of more reasons to fight over arbitrary crap.
What? it is mostly literal nonsense. Moreover the last paragraph where they say it only reproduced on one machine just does not justify the pseudo-deep analysis in the whole document.
Probably because musl's allocator exposes single newly faulted pages directly to the app. Other allocators tend to pre-allocate multiple pages at a time, so that the race window is narrower.
Codex bundles a ripgrep binary that is linked against musl. This is noted in the bug report: "I originally encountered this bug in the rg bundled with OpenAI Codex. That binary is byte-for-byte identical with the one in https://github.com/BurntSushi/ripgrep/releases/download/15.2... [...]"
Heh, from the kernel patch: https://lore.kernel.org/all/CALCETrXbj__SFQMzPZhES5y6-sh4np-...
> I saw a fun bug report in ripgrep and a studious but pretty bad AI-generated analysis
Referring to https://github.com/dfoxfranke/ripgrep-3494-analysis which I indeed thought "that's an awful lot written to have been written by a human."
Looks like that thread is from...today!
That was a tough read, but in all the verbiage I do not see it identifying the same code or area as the real genuine human on lore.kernel.
For those who can understand Claude better than me: did it anywhere actually identify the issue?
Assuming Andy (the mailing list post) is right, no it didn't.
I know this because he said so here in this comment section: https://news.ycombinator.com/item?id=49134550
That said the AI was sort of directionality right... it identified roughly the right code.
Thanks for the fun read.
Until ~2000, "cellphone user in public" equated with "smug asshole." We're at that icky rejection stage with AI.
Two years ago, that writeup would have been viewed as a generous gift of time to the community. Now we can't be bothered to read through it because we know where it came from, and that it's just worth $0.06 in tokens. Another reason (I think) is because we know what it portends.
In a few years, digging manually through bugs will probably be a last resort. We'll go from sneering to yawning at the reports, as another AI agent reads them and verifies the fix. Just like we ignore compiler-generated assembly code and people on their cellphones.
I get why people don't bother replacing the default allocator from musl all the time (it's there, convenient). But in an application whose purpose is to be FAST, I find it weird they haven't bothered replacing it with another more performant one.
mallocng is bad at dealing with contention during multithreading. I've had applications that usually were I/O bound suddenly become "malloc" bound when building with musl in multithreaded scenarios (and only just 8 threads). Switching to mimalloc improved performance by 20x, very close to what glibc offers by default, and just a bit under a glibc + mimalloc configuration.
I get that there's a real issue there and it's interesting (to some) to address it, but it should have never surfaced this way in the first place.
> I get why people don't bother replacing the default allocator from musl all the time (it's there, convenient).
ripgrep actually sets jemalloc as global allocator when built for 64b musl: https://github.com/BurntSushi/ripgrep/blob/435f59fc4b43af3ab...
That's good! But I wonder why it wasn't then enabled for that configuration or why the override wasn't "global" enough and the default allocator was still partly used.
Rust doesn't get to override musl internals. Ripgrep uses opendir, a POSIX library feature implemented in musl to look at er, directories. The stack trace suggests we blew up when Rust's std::sys::fs::unix::readdir internal detail called opendir, and it in turn allocated.
On Linux it would be possible for ripgrep to talk directly to the kernel via documented system calls without libc, but that wouldn't work on any other popular OS.
> Rust doesn't get to override musl internals.
Is malloc not a weak symbol in musl? I would expect it to be overridable like it is with glibc. Or does ripgrep only override Rust's global allocator?
> Or does ripgrep only override Rust's global allocator?
That one, as you can see in the code linked above, it only does rust-level overriding as that was where musl's allocator was found to impact ripgrep (https://github.com/BurntSushi/ripgrep/commit/03bf37ff4a29361...)
Last time I checked (a while ago), it wasn't. You needed to compile musl and exclude a few files from the archive to then provide your own allocator.
Depending on the toolchain, I'm assuming you could use some tricks or hacks to make it work better, but the tooling I used (Bazel) did that automatically and I haven't bothered to look at the internals yet .
I assume it already doesn't work on Windows. At some point you have to define your compatibility boundary. And high performance often coincides with mediocre compatibility.
You can call into the kernel just fine in Windows. The fact you use a function call wrapper instead of a raw syscall is not really relevant.
Most Windows NT kernel functions take a buffer to use rather than allocating their own memory. Even many Win32 functions don't allocate (although there you have to be much more careful).
A good example is go. On linux you can use a from scratch image fairly easily because it only uses syscalls. But for windows or mac the moving target wasn't maintainable so they link against shared objects. Linus enforcing the don't break userspace rule is what made that possible. That definitely has tradeoffs. At some point relibc or something similar will allow the same (stably) for rust. But using posix as that compatibility boundary gets you a much larger set of OS and only occasionally has a performance penalty. Often, a posix api tuned to the kernel is more performant. Musl is an exception precisely because it makes an openbsd-esque trade of performance for simple small attack surface.
The Linux kernel defines syscalls as its stable ABI (note: there are also non-kernel ABIs on Linux, such as Wayland) while Windows defines the DLL calls as its stable ABI.
One isn't better than the other. Linux's approach allows binaries to be fully statically linked, which is a more predictable environment for binaries, but Windows's approach composes better, as every process loads DLLs and this allows for things like graphics drivers and COM to work more reliably. As things stand on Linux you can't use the GPU in a portable statically linked app, because the kernel doesn't define the semantics of dynamic linking.
FYI, Windows's approach is better in some respects, like letting you avoid syscall overhead in some cases.
Notably, Linux has this too, but it's a special mechanism (vDSO) while on Windows it's just a normal function call with no difference from any other function.
If you look at the sigsegv stack, the allocation comes from opendir which is in musl libc as well. The allocation override mechanism used in Rust doesn’t replace the allocator process-wide; it merely replaces the allocator Rust code talks to.
It does seem like ripgrep should probably avoid using opendir from libc if it allocates using an allocator with a global lock though.
It’s a kernel bug. While I agree libc allocators suck for no good reason, it seems like this work of equally likely hit other application code including mimalloc and glibc.
Sure, but the problem isn't that the bug exists, is that it has surfaced in a performance application using musl and exerting code paths in a slow allocator.
mallocng should not be used at all.
Ah, no, the problem is that the bug exists.
Another, separate, problem, noticed by people investigating this problem, is that ripgrep on musl uses a serializing allocator in the hot path.
it's a tradeoff; mallocng does have benefits as well -- for example it uses less than half the amount of RAM compared to mimalloc and glibc for certain Python workloads. While yes it is a bit slower, I prefer mallocng in many cases.
I imagine there is no free lunch in the allocator space, but a series of trade-offs. Pick your poison on implementation which is going to be sub-optimal for some subset of use cases.
Without a hugely compelling reason to switch, going with the default is reasonable.
ripgrep is there to be FAST. Trading any speed to improve memory efficiency over a longer period of time for a process with a short life-time doesn't make sense in this case.
It's also a development tool. If your development machine is having RAM issues because it's doing a grep, you have bigger problems to solve.
As for other workloads that might use less RAM with mallocng compared to other performant ones, I'm curious to know about the magnitudes we're talking about. From 10MB to 20MB or from 100MB to 2GB? How was the speed of the program? Was there any multithreading involved?
For context, the Python workload was a general-purpose fileserver with file indexing and image thumbnailing, largely IO-bound. Switching from mallocng to mimalloc resulted in a slight speedup (150% of baseline), but over twice the memory usage, going from 250 to 670 MiB, primarily for thumbnailing. Some pathological cases (libvips calling imagemagick to decode heif) was a 10x multiplier.
There was multithreading, and yes, mallocng visibly became a bottleneck beyond 5 threads. However already at 3 threads there was diminishing returns for both allocators, so this was not an issue in this case.
The speed gain was hardly noticeable in practice since the program was already plenty fast, but the additional memory usage was a very real inconvenience.
Is that memory increase seen in peak usage or average/idle usage?
For file servers, I could be willing to give up 350 MB of memory temporarily for large-scale indexing and thumbnailing operations if it doesn't happen often and makes the file serving/browsing more responsive.
On the other hand, plenty of NAS/file server systems are going to have paltry CPU and memory. Running slower with a more modest memory foot print might be the only way to keep the system stable.
libvips shouldn't be calling imagemagick for heif decode, it has a nice one built in. Unless you were using a very old libvips!
I've found jemalloc works best for long running libvips processes, fwiw.
It's maybe a kernel bug.
The way most programs achieve being fast is by re-using allocations. You don't need a fast allocator if you don't allocate. Nothing of what ripgrep does inherently requires frequent allocations.
The ISO C definition of opendir() requires an allocation in practice because it returns a DIR* and it's bad practice for the library to arbitrarily limit how many of those an application has at a time.
Maybe a C library could preallocate several DIRs and only use the heap when those are exhausted, but this ripgrep use case (lots of threads running in parallel on a large tree) would still be likely to trigger that.
There is no ISO C definition of opendir(), that's a POSIX thing.
But yeah, those DIR objects which opendir() returns a pointer to are probably some kind of heap allocated. But we're talking about a system call involving the filesystem here. The time taken by the allocator is gonna be dwarfed by the time spent in the syscall even with the slowest allocator.
Ripgrep reads through every byte of most files and matches it against a regex. That is the tight inner loop where you want to avoid allocations.
Not that you even need to call the C functions. I don't think ripgrep would gain anything from it, but the syscall to read directory contents just needs a file descriptor.
I think ripgrep does avoid allocations in that innermost loop. That's why this report says all the crashes are associated with opendir() and its memory allocation: the opendir() call is outside of ripgrep's innermost loop but still runs often enough to trigger the race condition.
And that's why musl's allocator's performance isn't a huge concern for ripgrep
That's to be properly measured. Assuming that the allocator will not cause issues there is to be proven.
The application I recently improved by changing the allocator had a similar profile (a C++ include scanner) and thread parallel I/O functions had terrible performance originally with mallocng. Adding threads almost had negative value because of the contention.
We use musl+mimalloc by default for our entire production operating systems: https://stagex.tools
There is a real tradeoff:
- The musl allocator is only slow with multi-threading.
- Almost all other allocators have trouble reclaiming memory when using multi-threading. This often results in multiples more RSS than single threaded or musl's allocator.
Agree with you on mimalloc. It can even be configured to be aggressive in memory reclaim at the cost of performance.
Anyone running ripgrep on a an HPC cluster against a large cluster filesystem needs to stop and redesign their workflow. This generates high amounts of small I/O which is the Achilles heel of any large cluster filesystem. You are exporting your workload onto the metadata mechanisms of the filesystem rather keeping it within the higher bandwidth capable memory subsystem on your cluster. It doesn't take but a couple users running these types of jobs simultaneously to bring a high-bandwidth filesystem to its knees. Just stop it already.
Similarly, I've often wondered if this is the root cause of why GitHub has become fragile of late. All of a sudden you have operations on billions of tiny files that are amplified by AI usage. And object graphs are so fragmented by nature that it's hard to e.g. prefetch a page into memory and ensure that common git operations mainly touch that page's set of objects.
And if https://isolveproblems.substack.com/p/how-microsoft-vaporize... is to be believed even in the slightest, any unoptimized paths in Azure's filesystem abstractions could cause usage spikes to have massive splash radius.
> All of a sudden you have operations on billions of tiny files that are amplified by AI usage. And object graphs are so fragmented by nature that it's hard to e.g. prefetch a page into memory and ensure that common git operations mainly touch that page's set of objects.
In theory, that's what git's packfiles are for: loose objects get packed together into a file, and if the file is well-arranged then objects that tend to get read together will also end up getting paged-in together (see also: https://www.kernel.org/pub/software/scm/git/docs/technical/p...).
GitHub in particular has spent a lot of time and effort on packing things well (e.g. https://github.blog/open-source/git/scaling-monorepo-mainten...), so I doubt that the "billions of tiny files" problem is at the root of their recent instability.
The analysis of the kernel bug may be a better thing to link to: https://github.com/dfoxfranke/ripgrep-3494-analysis.
I tried reading and gave up at the "Headline"...
Quoting from the bug analysis:
>Headline. The crash is real and reproducible. With musl instrumentation we pin the in-process mechanism precisely: a thread's own store to a freshly- faulted anonymous page becomes invisible to that same thread's reload ~10 instructions later, because the page's backing is replaced mid-function. A pagemap read at the instant of the fault shows the backing is the kernel's zero page. A captured core dump confirms the crash site with matched virtual addresses. The mechanism localizes to the interaction between the per-VMA-lock anonymous-fault fast path and a concurrent munmap's TLB shootdown. A source-level review of Linux 7.0.12 identifies a specific race in that interaction, and a git comparison across v6.19/v7.0/v7.1/mainline identifies the v7.0-introduced change on the munmap-teardown side that widens it.
"a thread's own store to a freshly- faulted anonymous page becomes invisible to that same thread's reload ~10 instructions later, because the page's backing is replaced mid-function." -> ??? What's a "backing" of a page? Freshly-faulted? Fresh fries? "~10 instructions later"?
"A pagemap read at the instant of the fault shows the backing is the kernel's zero page." -> Backing?
"The mechanism localizes to the interaction between the per-VMA-lock anonymous-fault fast path and a concurrent munmap's TLB shootdown." -> How can a mechanism "localize"?
And more.
This is... words strung together. Nothing more. I wonder how people read and make sense of this.
In case people have forgotten what real technical writing looks like, here's a sample (I am not the author): https://yifan.lu/2019/01/11/the-first-f00d-exploit/
I'm becoming increasingly allergic to the performative writing of LLMs.
The worst part is how compulsive LLMs are at writing like this. Like the system prompt instructs it to "reason" and like a college Sophomore it pontificates and quotes Nietzsche to fein intelligence and orginal thought.
I love LLMs for technical stuff, but their writing style is horrendous. I also think it's bad form to waste fellow humans' time with walls of generated text.
I understand that it's probably a hard technical problem to get models to conform to a style without sacrificing performance in other areas. But I really wish AI companies would spend more effort getting them to write in a neutral, concise way and not like a middle manager on 80mg of Adderall.
> But I really wish AI companies would spend more effort getting them to write in a neutral, concise way
I wonder if this would make it harder or easier to detect whether the underlying meaning of the writing is bullshit.
Definitely easier. It's the same for humans. You can tell that someone has little of substance say when they cloak everything in big words and elaborate metaphors.
Good old sesquipedalian loquaciousness.
This is technical writing for kernel developers who already know how memory management works. If you want to understand it better, you should study up on the x86 MMU and the kernel memory management subsystem. I'm not a kernel expert but I found it pretty understandable, so I'll try to explain it.
MMU: The memory management unit, part of the CPU designed to allow an OS kernel to flexibly control how memory addresses used by a program map to physical RAM.
Page: A 4k region of memory addresses.
Anonymous: The page is just regular old memory used by a single process for general purposes, it's not part of e.g. a memory-mapped file.
Backing: The memory corresponding to a page. The kernel can direct the MMU to map any page to any part of RAM, or delete the mapping entirely.
Zero page: The kernel keeps a single 4 KB page filled with zeros (the "zero page").
Page fault: An error that occurs when you try to write a page that's not present, or write to a read-only page. Some page faults are normal and expected by the kernel, they support features like swapping, memory-mapped files, etc. Such "expected" page faults are invisible to your program. (On the other hand, an unexpected page fault occurs when your program just straight-up accesses an address it's not supposed to, which will end your program with a SIGSEGV signal -- an all-too-familiar experience for a C programmer.)
When you allocate say 4 MB of memory, the kernel simply sets it to 1k read-only copies of the 4 KB zero page. Then when your program tries to write to a page, it faults. The kernel handles this expected fault by assigning that page an individualized memory region. If a program allocates a lot of memory, it's only assigned memory for what it actually uses, when it first uses it. This is an optimization: Every program that allocates more memory than it needs is wasteful, but the kernel can recover that waste by not backing the allocation with memory -- the program has to prove each page is needed by writing to it.
Store to a freshly-faulted anonymous page: The program stored data to its own memory, which caused an expected page fault.
> becomes invisible to that same thread's reload ~10 instructions later
The thread fairly quickly tried to read the data from the same place in memory it had just written to. It doesn't get the same data back. This is a major problem that causes the program to malfunction and crash.
> A pagemap read at the instant of the fault shows the backing is the kernel's zero page
The expected fault occurred because the program tried to write to an address that was mapped to the zero page, probably because it's the first write to freshly allocated memory. (This rules out other reasons it might be faulted, e.g. the page had been swapped to disk.)
> concurrent munmap's TLB shootdown
The page table is big and slow so the CPU caches parts of it in an area called the TLB (translation lookaside buffer). munmap is a kernel function that removes the backing for an address region. Since munmap changes the mapping, it has to inform the MMU the relevant part of the TLB is no longer valid.
> per-VMA-lock anonymous-fault fast path
This is what the kernel does when the program first writes to a zero page. Saying it's the "fast path" implies there's some (slow) special case handling in the kernel code, but the bug doesn't trigger any of the special cases, we're in the most common case, which the kernel code tries to handle as quickly as possible because it has measurable impact on performance. I would guess per-VMA-lock has to do with how the kernel synchronizes this code (a "lock" is a basic synchronization primitive, you should be familiar with it if you do any kind of multithreaded programming).
> The mechanism localizes to the interaction between the per-VMA-lock anonymous-fault fast path and a concurrent munmap's TLB shootdown. A source-level review of Linux 7.0.12 identifies a specific race in that interaction
The mechanism: What's actually happening to cause the program not to be able to load data it just stored in memory.
Localizes: The cause of a problem like this is a needle in a haystack -- it could be caused by the program, the kernel or the hardware. The analysis has narrowed down the haystack to a specific part of the kernel code.
Race in that interaction: Two parts of the kernel code, (1) The kernel code for munmap and (2) the kernel code to handle a program's first write to the read-only zero page for a fresh allocation. These two parts work fine individually but the problem occurs when they both try to change the page tables / TLB at exactly the same time. This is quite a small, specific haystack compared to "somewhere in the program, kernel or hardware" we started with.
I understood the jargon just fine. The writing is still terrible and way too verbose.
Excellent explanation.
I’ve been reading the LWN kernel page for >20 years so I’ve picked up almost all the jargon and could understand it pretty decently.
You’re dead on in saying it was written for that specific audience, not anyone more “normal”.
Thank you for taking the time to write that out. I found it incredibly helpful, just wanted to let you know I appreciated it!
Agreed, the article you linked is a great example of clear, understandable, detailed technical writing.
fwiw "backing" is standard jargon in this context, and "freshly faulted" is phrasing you'll see in Linux source comments. The writeup as a whole is indeed slop, though.
This is a pretty clear analysis if you are a kernel developer.
> a thread's own store to a freshly- faulted anonymous page becomes invisible to that same thread's reload ~10 instructions later, because the page's backing is replaced mid-function
A result of a machine instruction to store something in RAM got erased, when the same thread attempted to load it ~10 machine instructions later. The reason is that the physical RAM page backing that particular virutal page got replaced while the thread was running.
The fault happens in the fast path for anonymous (i.e. not backed by files on disk) pages, in the granular per-VMA (virtual memory area) locks.
> "A pagemap read at the instant of the fault shows the backing is the kernel's zero page." -> Backing?
Yes, it's typical kernel terminology. "A backing page" or a "backing file".
> This is... words strung together. Nothing more. I wonder how people read and make sense of this.
There's nothing wrong with this description. It's just written for kernel developers. It needs to be expanded and explained for people who are not.
"backing" is a Virtual-Memory related term. With virtual memory, you can have memory areas (called "pages") that are not really there but only present in the metadata (the "book-keeping", so to say). The first time, someone actually tries to access this page, that access is interrupted ("faulted") and the kernel gets a say in what should be done to that piece of memory (i.e. load it from disk somewhere, reserve actual physical memory and fill it with zeroes, or crash the process).
The "backing" refers to the physical memory that might or might not be present for every "virtual" piece of memory that your program has allocated.
"Freshly faulted" means that a page of (virtual) memory has just received a "backing" by the process above and is, thus, very fresh in physical memory (even though the virtual memory might have been allocated much earlier)
"~10 instructions later" refers to the assembly- (machine-) code, which, contrary to a high-level language like C, usually has long(ish) sequences of rather simple "instructions". 10 instructions is a rather short interval in assembly code.
As for the "localize", the term used is actually "localizes to" which I read as "turns out to be located in" (probably just a bad English translation by the original author)
While this whole summary reads a bit weird, I don't think it is necessarily the result of an LLM, it's probably just that someone who is rather inexperienced at writing up technical summaries did it.
So the theory is that the page is faulted in, then somehow evicted within 10 instructions, then re-faulted in somewhere else, resulting in writes not making it to the page? That would need a context switch and another page fault to happen in short succession. But the context switch to evict the page back out would have necessitated pending writes to have finished. Nothing actually makes sense with that explanation.
The way I read it was that it wrote to a page and then did a read on that page within 10 instructions and that was not enough time for the memory system to have gone through the process of creating the page backed with real memory and that the timing bug widened in Linux 7.0 such that more things like this exposed the bug. Or there was flapping in the TLB for some reason, or the locking wasn’t correct, or whatever else that the kernel devs will figure out was the cause.
I’m guessing it wrote to a page and read back zero page as the memory was in the mist of being allocated.
But the write should have triggered a page fault immediately in that case, and should have been blocked on the page being actually allocated before resuming.
Sounded like a race condition causing a correct new mapping in the TLB to be cleared away by mistake, reverting back to the zero page mapping it was before.
Except incorrectly flushing the TLB would merely be a performance bug.
I understand all the words in it and follow the argument, but it's still bullshit in the technical sense of being written by someone who wants to sound authoritative but doesn't ultimately care if they are right or wrong.
The write-up should have been about five paragraphs: (1) "The script at https://... generates a tree with X million files taking 20 GB total. Running ripgrep 1.2.34, compiled with musl, on my Threadripper 9876 as 'rg ...' crashes about a fourth of the time. With glibc, it does not crash." (2) "This appears to be a bug introduced by commit abcdef1234 in some_vm_call() that allows a race between [...] such that user space can see a corrupt whatever." (3) "The sequence of operations that causes this is: (show two or more kernel threads with lines interleaved to explain the bug)". (4) and (5) as needed to elaborate on those three core points.
But the actual post is long on (fluent) speculation and short on reference to actual code. That's a large part of why it's offensive slop.
Thanks for the explanation. I should have looked up each term a bit more.
So "backing" is the underlying physical memory the page maps to.
So if I were to make sense of that piece of slop, is this what it would be?
"In one thread, a page fault happens during a store operation. The physical memory address is the proper address, which is immediately (around 10 assembly instructions later) replaced by the 0x00000000 memory address by another thread due to race condition, resulting in a crash when the original thread tries to read that page again."
The first thread is the user program, the other thread is the kernel doing things on another core.
Reading from the zero page is legal, that doesn’t cause it to crash. The crash is because of a logic “bug” in the program where the zero it reads back causes some other issue in the program.
I say “bug” because it’s clearly impossible without the kernel messing up. It just stored a non-zero value there.
Nope. It's more complicated. The store operation does not need to trigger the fault, it can happen for other reasons.
What happens here is a bug in munmap() that happens in another thread. It corrupts the TLB, for a brief instant replacing the correct page.
So that the virtual memory page that the crashed thread tried to read its RAM, it gets replaced by the special zero-filled physical page. The kernel uses this zero page as an optimization when it needs to create a region of data filled with zeroes, so it can just map one physical page over the entire range.
It's a really low-level bug that just requires a lot of specialized knowledge just to explain. A better write-up is certainly possible but needs to have a lot of explanations to make it understandable.
I don't care if the cat is black or white, so long as it catches mice.
Translation: if this nails-on-chalkboard LLM spew identifies a kernel bug, which is then patched, its aesthetic qualities do not interest me in the slightest.
Unfortunately, the kernel bug would need to be understood to be patched, and the developed patch itself would need to be reviewed before being accepted, which seems... very hard, to put it mildly, using the nails-on-chalkboard LLM spew report.
At least it is a report. It says hey, there's a bug vaguely like this in this general area. Not too dissimilar from the average user report.
Yes, the word 'if' was in that sentence.
The only people who have a need to judge the quality of the report (not aesthetically, I mean) are those few who know enough about the subsystems of the kernel it implicates to do something about it.
My point: does it lead to a bugfix? Good. No? Bad.
Do you have an answer to that question? I believe it's a bit early, no?
The computer seems to have identified source code to blame though? Unless of course it hallucinated which there’s always a non zero chance of
It’s the kernel. Nothing a user space library can do should ever be able to call this.
Just happens to be that the musl code is able to hit this and other code isn’t for some reason.
> It’s the kernel. Nothing a user space library can do should ever be able to call this.
I don't follow. An application might see this kind of crash if it has a bug causing it to access a page while another thread is mapping or unmapping that page. That would be a bug in mallocng, musl or ripgrep. Or, as someone else mentioned, it could be bug in the processor's virtual memory logic that has the same effect. Why do you say it can only be a kernel bug?
It got traced to a race condition in munmap() in the kernel. The inefficient musl allocator simply triggers it more easily.
It got arm-waved to a race condition in munmap(). Claude (or a distill of Claude) didn't identify a race, it just convinced itself that was the cause.
Andy Lutomirski seems to think that it is a bug: https://lore.kernel.org/all/CALCETrXbj__SFQMzPZhES5y6-sh4np-...
I looked at the explanation, and it seems to be at very least plausible. The kernel version bisection also lines up. We'll see if the proposed patch fixes this.
User code calls kernel code all the time - just not by its address.
I mean, we are deep into reading tea leaves at this point, but what it seems to have identified is the source file that changed such that this issue can now occur. My gut instinct is that the changed code is still valid and correct, but it exercises the hardware in a different or a more intense way.
Reading an AI-generated bug report is awful.
Stopped after
> Headline. The crash is real and reproducible.
I really hated that kind of language, feels like speaking to a motivator
anyway it's slop, can sense it even before i started reading
The worst thing is that I know there's something interesting in there but I'm too arsed to take the time and decode what data the author must have used to generate that text.
Exactly, it would be so much better if they shared the prompts instead of the output, at least then we'd be able to make some sense of what they were thinking about.
Unfortunately the prompt may have been something like "investigate this bug and file an issue when you think you understand what's going on". The "better just to share the prompt" heuristic falls down when the actual underlying intellectual work was also done by the machine.
(Feel free to pretend that I used some phrase other than "intellectual work" if you dislike seeing it used to describe something done by AI.)
How so? We can give the same starting point to the same AI and ask it to investigate too.
Yeah in that case just closing it and moving on would be better IMO. A sufficiently large fraction of the time the issues with vibeslopped code and/or analysis are bad enough once you start digging into it that it's not worth spending time on. My heuristic would be to simply ignore contributions derived from lazy prompts. I don't think there are very many healthy babies in that bath water.
So having the prompt available would be useful even in the degenerate case.
Get your own AI to give you the tldr :/
How does that help anything?
Makes the stock market go up
Maybe it's not going to get any better. LLMs were trained to do two things:
- Mimic human language (and logic, since that's part of what we express with language). This includes code written by humans.
- Write code to solve problems. The figure of merit here is solving the problem, not reproducing anything human.
I'm not an expert on LLMs, but it's not obvious that the human reasoning they are fitting in the first case is going to be anything like the problem solving required in the second case. It was trained to get itself out of a problem, not to do it in a way that a human would relate to.
And we're already running out of human data to train on, while synthetic data has no limit. Bug reports in the future might amount to "fix this because then I'll get a cookie".
A lot of it is that AI can talk with different "voice" but that requires giving it cues, and that costs money and expands context, so people that don't know, don't, and those that do have no incentive to
Nice sleuthing, nonsense explanation. An extra TLB flush is never an error. (The CPU is free to flush whenever it feels like doing so.) The error seems to be that somehow a zero-page PTE was present when it shouldn’t have been.
This sounds to me like either (a) a complex race involving a CPU migration at an awkward time or (b) a bug in the zap path transiently exposing wrong PTEs.
Also, I don’t think the zero page has pfn zero.
If I had to throw a dart, I would guess that direct page table zapping is allowing a CPU to read through a higher-level-paging-structure cached entry to a table that has been freed and reused. Yuck. I’ve debugged one of these before.
The part that doesn't make sense to me is that trying to read through a zeroed PTE should be an immediate segfault, not something that reads back zeroes (regardless of whether the zero page is pfn 0 (it isn't)).
I think the broad strokes are that, due to bad locking in the kernel, mmap() returns a VA that a concurrent munmap() is still in the middle of unmapping. The specifics beyond that seem murky/speculative/inconsistent.
Alternatively, it could be a hardware bug, since afaict it's only been repro'd on one hardware config. (I know there are a bunch of ARM cores with erratas around TLB invalidation)
After more digging, I bet it's a paging-structure-cache flush bug. I've debugged these before, and they're nasty, hardware dependent, hard-to-reproduce issues. Looks like the code that the AI flagged might actually be wrong, but not for the reason that the AI thought.
https://lore.kernel.org/all/CALCETrXbj__SFQMzPZhES5y6-sh4np-...
> Alternatively, it could be a hardware bug, since afaict it's only been repro'd on one hardware config. (I know there are a bunch of ARM cores with erratas around TLB invalidation)
Which raises the question of whether any HN readers have successfully reproduced this?
I've just tried (using his file generation script and the official rg binary he links) on a Ryzen 7 5800X / 7.1.5-arch1-2 with sufficient free ram as the github issue suggests, and still no segfaults after 10 minutes.
I have never, in my entire personal history of kernel development, successfully reproduced a paging structure cache bug. I have stared at these sorts of bugs and puzzled them out.
With this particular issue, if my theory is right, it will depend on all manner of microarchitectural issues, possibly address spare randomization, context switch and migration timing, random speculative accesses that pull things into cache, etc. Yuck.
The explanation is obviously written by an AI agent.
Not really. It's typical rambling LLM slop-analysis. It may be a correct analysis (I couldn't stomach reading it in detail), but it's a pain in the ass to read. A human doing the same analysis would have written something 1/5th the length.
soulless unreadable AI slop
I can read it but not sure if worth spending energy on it.
It doesn't make sense for the reader to spend more energy than the writer spent on creating it.
> It doesn't make sense for the reader to spend more energy than the writer spent on creating it.
Great way to summarize cultural "economics"
Couldn't put the words on this pattern but sometimes all I care about is that someone cared about.
I usually go for "If I wanted LLM answer I'd ask LLM instead of reading your answer/article/content"
But it goes beyond this. The whole 2020s era is about cheap ways to do anything and be able to stream it to billions of people. Yet I feel no density in the work because there was no energy spent on it.
energy might be the wrong metric, plenty of energy was wasted spinning up that LLM.
Yeah but the thing is that someone tries to waste time of humans.
This is why I hate "interacting" with bots, scripts or AI/LLMs. It just wastes my time, again and again and again. Oddly enough not all humans understand that. About two months ago, a german developer involved with ffmpeg, spam-slopped their mailing list with AI (it was an AI proposal for some change to ffmpeg in the future). He still does not understand why that is a problem.
I think it was GZDoom where the absentee project owner suddenly came back, pushed a bunch of nonsense AI comments, so the entire actual development community just forked it again and left him to play in his sandbox? Now it's UZDoom.
Something similar happened with PolyMC to PrismLauncher but that wasn't about AI, it was an absentee owner who came back and deleted everything he said was woke.
PoC||GTFO, if it reproduces, it's valuable
In the history of Linux that I'm aware of (and I've been nose-deep in this stuff on and off for quite a while), these issues are almost never debugged because they're reproduced -- they're debugged because someone who actually understands the messy interactions of the code and hardware in question thinks about them.
A reproducer might not actually be useful because there is basically no way short of fancy hardware tracing to figure out what the reproducer is doing.
Then post whatever notes were fed into the AI instead. The verbosity and self-congratulating add negative value.
What will you do when the prompt was "Figure out the bug and write a report for me". Not saying it was in this particular case, but I think at least in other cases, it will be.
"it will be"
Trying to figure out what to do based on possible future scenarios has a place, but not here when we are talking about a concrete present problem.
Prompt an AI to figure out the bug and write a report, of course. If that is actually useful.
> What will you do when the prompt was "Figure out the bug and write a report for me".
The rational choice would be to cut your losses and stop reading at that point. Once you realize zero effort went into the prompt, there's no longer any reason to read the output. The age old truism still applies: garbage in, garbage out.
If you think it's worthwhile, close the issue with a comment: "please rework this and show your work next time". Otherwise just close it without commenting and move on.
It didn't figure out the bug: the report is largely nonsense. It merely did a bad impression of having figured out the bug, by stringing together several observations into something superficially resembling a narrative. Providing those observations without the gibberish framing might be useful.
The parent's point still stands. In many cases it will have figured out the bug. In many cases it will have produced a small, clean, deterministic reproducer.
In my daily work I see these cases. It does help that the bugs that are filed contain a test case and some analysis by the agent.
It would not help to get ten identical bug reports all saying "I asked my agent to find a bug by prompting it with "find a bug and produce a test case". It found a nasty bug and a really nice reproducer. I'm not including its output here. Good luck!"
That I agree with (not instead, in addition)
They have a PoC, we're arguing about the LLM-generated slop explanation of the PoC.
Is it actually reproducible though and do I want to chase and reproduce something that had no human touch? Do you?
If it's an actual bug, I hope the Linux devs will. It impacted a human, and it's a bug regardless of human or non-human touch. Of course, looking at a bug is a matter of priority, always has been.
Why is it unreadable? I actually find LLM bug reports/breakdowns to be far more detailed and concise that classical human written ones. If you read the linked repo it clearly goes it depth where the bug was found, how to reproduce it (and in depth). Most disclosures that are human written don't do this at all, they barely even tell you _how_ to reproduce the bug. Just look at the "3.3 The self-store tear", the LLM clearly describes exactly what went wrong, so you can verify it by hand.
I'm reasonably confident that the author has a reproduction case on their hands. That's easy to directly verify. I'm way less confident in that long and rambling analysis. As fwlr observes, it's not clear that this has been reproduced off the original machine: https://news.ycombinator.com/item?id=49134357
If it is the case that the original machine has an intermittent hardware fault, that analysis is exactly what I'd expect of an current AI. Voluminous and confidently wrong. Also interesting that the AI itself doesn't note this... having been biased by the conversation to consider kernel bugs, in the "cross-machine data" [1] it blames the kernel and doesn't appear to consider alternate explanations.
It doesn't get mentioned much around here but one major issue with AIs today is that they fall into tunnel vision very easily and often need some help getting out of it. There's a structural reason for tunnel vision built into their context limits, and there's also situations like this, where if you bias them in a certain direction they can get monomaniacally stuck on it. You can also see it where they do things like try to do something on your system like view a certain file, then if for some reason they fail due to permission issues, if you don't stop them they'll go absolutely insane trying different ways to access the file, rather than just stopping and saying "Hey, I need help". Even if you prompt them directly to do that, it only helps, it doesn't fix the issue. I suspect the RHLF training they get to be better coding agents reinforces this behavior because in the coding quality tests they're rewarded for one-shotting all the various benchmarks, where that "bull in a china shop" approach works better than giving up. But I'd actually like them to give up a bit more often. I've had the same problem with some particularly go-getting junior devs, too... I appreciate the ambition and I look forward to harnessing it in other situations but I'd rather you didn't spend five hours to create a terrible work around to something I could have gotten you in two minutes. For the junior dev, it's OK, they take the feedback and adjust... the AIs never adjust.
[1]: https://github.com/dfoxfranke/ripgrep-3494-analysis#6-cross-...
Not enough people are going to see this comment. It's so true!
That reminds me of one ticket I had to handle. It was generated by an agent from a customer report and the different hypotheses were wrong. Also a huge wall of text.
The issue was alongside the API boundary (timezone shenanigans) but the agent invented a lot of reasons with reasonable looking arguments because of the tunnel vision that there should be something wrong inside the project.
Because any writing needs a core intent they need to convey, which you can summarize down to according to the audience and why it should be important to them. Kinda like the same idea of elevator pitches, “explain like I’m five” and tactical reports when there’s a time constraints.
You got none of that here. It’s just realms of text.
Maybe the person writing the report isn't an expert in this domain or doesn't have the time to commit to it? From my point of view as long as the information is accurate and reproducible, it's valuable.
> Maybe the person writing the report isn't an expert in this domain or doesn't have the time to commit to it?
Then maybe that person should not do it? At least until they find the time?
The other part is that almost no one would have the energy to spend hours instrumenting and rebuilding musl, reading the kernel mm code, and thinking hard about how they might interact.
Claude is probably not right about the root cause here and is probably bsing, I agree. But it's collected enough raw data to point some expert humans at the right interaction. I'd take this bs bug report and start asking Claude some questions that would guide it to a more plausible actual answer, but that requires a little more experience in kernel development.
> The other part is that almost no one would have the energy to spend hours instrumenting and rebuilding musl, reading the kernel mm code, and thinking hard about how they might interact.
Maybe because it isn’t needed. That’s the reason we have experts and professionals, because it’s more economical to use them than for everyone to start from scratch. It’s easier to go to a mechanic shops to fix my engine block than to try to do it myself. I’ve heard that a lot of shops charge more if you said an amateur try to fix things first.
It’s a lot of stuff, but that’s not how you experiment for an hypothesis.
I've been a systems dev for a long time and have debugged many such tricky issues before. Even on teams of experts, this type of issue would take someone really capable weeks to track down, especially because the repro environment is hard to obtain. And the opportunity cost of doing this type of work for a rare bug is high, since you may never get to an answer and have burned a lot of time. So in practice I solved most of this type of issue early in my career before I had a family and before I had higher level work to do.
The ai doesn't have the same incentives as a human. It can also be applied to the repro where it exists.
Your mechanic analogy is a bit inapt. Sure they're experts and can replace your engine much better than you can. But tracking down a non obvious system level bug like this one is much less repeatable and much harder because the problem has already been filtered through a lot of reviews and testing. As a system becomes mature, the bugs become harder and harder individually to track down.
> Even on teams of experts, this type of issue would take someone really capable weeks to track down, especially because the repro environment is hard to obtain.
> As a system becomes mature, the bugs become harder and harder individually to track down.
What I'm talking about is that experts knows the system mechanism more than amateurs, so any hypothesis and experimental setup will be more focused and thus more economical than any amateurish one. And there's the matter of knowledge not present in some docs or other forms, such as past experiences.
Even with LLM tooling, we've seen the rises of harnesses and helper tools instead of relying on generation for everything. Who would you trust to build such harness, a domain experts or some random guy off the street?
> From my point of view as long as the information is accurate
That's the trillion-dollar catch, isn't it. LLMs love to write 30 paragraphs about some plausibly-correct-sounding explanation that is just as likely to be completely fucking wrong as it is accurate. The bug might be real, but that doesn't mean this analysis is accurate, and trying to figure out where the LLM went off the rails can be a nightmare. If you can actually understand the bug, it doesn't take 30 paragraphs to explain it. I would throw this bug report into my junk bin if I were on the receiving end of it, and I say that as someone who will spend days troubleshooting any issue a user will help me diagnose even if it only happens on their machine.
Then where’s the lie in “soulless AI slop”?
I'd rather read walls of AI slop than soulful meatbag bickering. Just because you might have a soul and intent doesn't mean you do anything useful with it. All I see is the intentful invention of more reasons to fight over arbitrary crap.
I had the opposite reaction. Clear, detailed, well-organized. Pretty close to the ideal writeup.
Could you explain it in your own words? (I tried myself, and quickly discovered that it is incoherent)
Perhaps you are talking about the ripgrep issue itself, which is fine. The others are talking about the linked analysis, which is not.
Hi Claude! :wave:
Really? It's chock full of poorly written and irrelevant chain-of-thought rambling.
What? it is mostly literal nonsense. Moreover the last paragraph where they say it only reproduced on one machine just does not justify the pseudo-deep analysis in the whole document.
So, why the bug triggers only with muslc and not other libc ?
Coincidence. It also only triggers on one machine.
Probably because musl's allocator exposes single newly faulted pages directly to the app. Other allocators tend to pre-allocate multiple pages at a time, so that the race window is narrower.
I would normally suspect musl's thread stack size. Is the kernel bug confirmed?
wow, everything is broken lol
No wonder search in codex is so a$$
What does one have to do with the other?
Codex uses musl?
Codex bundles a ripgrep binary that is linked against musl. This is noted in the bug report: "I originally encountered this bug in the rg bundled with OpenAI Codex. That binary is byte-for-byte identical with the one in https://github.com/BurntSushi/ripgrep/releases/download/15.2... [...]"