I did this manually back in Feb 2024[1], but only got "connector A" mapped out and a partially designed SXM2-to-PCIe interposer designed before getting distracted.
During the process, I discovered Facebook's Big Basin JBOG schematics/specs (from Open Compute), which contained a design derived from NVIDIA's DGX-1 reference implementation. Looks like this clears up some gaps in the current reference. It also explains some of the design choices, like the form factor and similar pinout to OpenPOWER.
Nice, I did a similar thing with OCP documents and an SXM2 server schematic set I found, but ultimately lost interest in the project after I rented a server with 8x V100s and realized it was kind of crap for LLM inference.
V100 is a beast if you feed it the right format. A lot of these older cards don’t do well with lower quants. So instead of using Q4, try Q8 or FP16.
If you’re still getting poor performance, point an Astra agent at it and ask it to benchmark and find the pressure points.
Also speculative decoding has changed the game completely. I’ve managed to 2-3x’d the theoretical max for my setup. Apple Silicon is a bit more limited here, and cannot deal with large DFlash block sizes. But AMD and Intel chips are game.
I have 4 of these now of the V100S variety. They are aging but solid. If you are not going above 4 cards you probably don’t need SXM2, but the packaging can be easier to deal with. I have PCI variants and network them with 25Gbps and do RDMA.
I have a brain damaged quant of DeepSeek V4.1 (IQ2_XXS) running at about 80-100 t/s and 950 pps. It’s really not that bad, but not as smart as the full model. I have Qwen Flash Next 3.8 on UD4 K XL at about 200 t/s single, 450-550 aggregate t/s and 3-4k pps. Hard to say which is smarter honestly between the two models.
It’s a great model for local LLM. I wrote my own engine and I havent seem any 4 card benchmarks faster than it yet so these cards definitely have more ceiling than most people thought a few months ago. They don’t have a lot of compute but they have just enough to keep their VRAM busy most of the time :)
It's 2xTP 2xPP. Each system runs 2xTP cards, then split and pipelined. This works because each server is doing TP only over its local PCIe and only the split traffic, which is much less, goes over the 25Gbps RDMA link so there is virtually no penalty for doing it this way, but it also does not give any single user speedup. It does free up more room per card to easily host 8x262K KV in VRAM and the vision tower. So it is pretty nice and I am very happy with this setup. I mean, behold: https://stealthgoat.com/fnfl/pelicans6.html the evolution of the pelican :)
Thanks, super information. I am considering a TP4 PP2 system using 2 nodes with 4x 5060ti 16gb on each. Already have 6 such GPUs, with 4 of them running 27B at the moment. Hoping to get similar performance. Ideally with stock vLLM to keep on a well maintained standard software.
You should be fine with 5060 on vLLM. The support for V100 is limited but getting much better. You should be able to get good performance. TP4 will hurt across any sort of network link, but you can see, the 5060 has a lot of compute power so the delays won't hurt as much.
That should be solid. Just splitting / pipeline over the network link is not bad. Tp4 on PCIe you will feel it for single user and it might even reduce single stream performance but if you batch and have concurrent workloads it will give you a lot more throughput. It will give you a lot more room for context and weights though, which is nice :)
It is basically a gutted ggml/llama.cpp, but I just had the agents build highly optimized version for two specific model quants with zero consideration or compatibility for anything else.
https://github.com/bitexploder/fnfl-v100 -- I am still fixing the odd bug in it and it is really specific to ISTA IQ3_XXS and Unsloth UD4_K_XL in terms of what I have built. Some of the smaller benchmark data is out of date, like I think I got 32GB MoE cache V100 up to about 700-1000pps.
Good to see this re-posted, I posted it two days ago but it sank without a trace so I'm really happy you posted it again. These V100's are a pretty neat GPU and one of very few that is still - somewhat - affordable.
Might be worth depending on the price but GeForce Nvidia GPUs tend to be a better deal than the server class since their EULA doesn’t allow them to be put in datacenters.
v100 caps out at CUDA 12 and the 585 driver. He have like 6 or 8 of them at work (including a few of the 32GB variants) and it's a PITA having to work around that.
Ha, amazing!
I did this manually back in Feb 2024[1], but only got "connector A" mapped out and a partially designed SXM2-to-PCIe interposer designed before getting distracted.
During the process, I discovered Facebook's Big Basin JBOG schematics/specs (from Open Compute), which contained a design derived from NVIDIA's DGX-1 reference implementation. Looks like this clears up some gaps in the current reference. It also explains some of the design choices, like the form factor and similar pinout to OpenPOWER.
https://docs.google.com/spreadsheets/d/1DfH0T9KWWoc2CpKZsRIT...
Nice, I did a similar thing with OCP documents and an SXM2 server schematic set I found, but ultimately lost interest in the project after I rented a server with 8x V100s and realized it was kind of crap for LLM inference.
V100 is a beast if you feed it the right format. A lot of these older cards don’t do well with lower quants. So instead of using Q4, try Q8 or FP16.
If you’re still getting poor performance, point an Astra agent at it and ask it to benchmark and find the pressure points.
Also speculative decoding has changed the game completely. I’ve managed to 2-3x’d the theoretical max for my setup. Apple Silicon is a bit more limited here, and cannot deal with large DFlash block sizes. But AMD and Intel chips are game.
Good luck and godspeed!
I have 4 of these now of the V100S variety. They are aging but solid. If you are not going above 4 cards you probably don’t need SXM2, but the packaging can be easier to deal with. I have PCI variants and network them with 25Gbps and do RDMA.
I have a brain damaged quant of DeepSeek V4.1 (IQ2_XXS) running at about 80-100 t/s and 950 pps. It’s really not that bad, but not as smart as the full model. I have Qwen Flash Next 3.8 on UD4 K XL at about 200 t/s single, 450-550 aggregate t/s and 3-4k pps. Hard to say which is smarter honestly between the two models.
It’s a great model for local LLM. I wrote my own engine and I havent seem any 4 card benchmarks faster than it yet so these cards definitely have more ceiling than most people thought a few months ago. They don’t have a lot of compute but they have just enough to keep their VRAM busy most of the time :)
Those are excellent numbers, especially on QFN. You say the cards are networked, how many cards per node / how many cards per machine?
It's 2xTP 2xPP. Each system runs 2xTP cards, then split and pipelined. This works because each server is doing TP only over its local PCIe and only the split traffic, which is much less, goes over the 25Gbps RDMA link so there is virtually no penalty for doing it this way, but it also does not give any single user speedup. It does free up more room per card to easily host 8x262K KV in VRAM and the vision tower. So it is pretty nice and I am very happy with this setup. I mean, behold: https://stealthgoat.com/fnfl/pelicans6.html the evolution of the pelican :)
Thanks, super information. I am considering a TP4 PP2 system using 2 nodes with 4x 5060ti 16gb on each. Already have 6 such GPUs, with 4 of them running 27B at the moment. Hoping to get similar performance. Ideally with stock vLLM to keep on a well maintained standard software.
You should be fine with 5060 on vLLM. The support for V100 is limited but getting much better. You should be able to get good performance. TP4 will hurt across any sort of network link, but you can see, the 5060 has a lot of compute power so the delays won't hurt as much.
Thanks for the input! It would be TP4 across PCIe and PP2 across network. So I am hoping the network hit will not be too high.
That should be solid. Just splitting / pipeline over the network link is not bad. Tp4 on PCIe you will feel it for single user and it might even reduce single stream performance but if you batch and have concurrent workloads it will give you a lot more throughput. It will give you a lot more room for context and weights though, which is nice :)
What rumtime are you using?
It is basically a gutted ggml/llama.cpp, but I just had the agents build highly optimized version for two specific model quants with zero consideration or compatibility for anything else.
Is your engine public?
https://github.com/bitexploder/fnfl-v100 -- I am still fixing the odd bug in it and it is really specific to ISTA IQ3_XXS and Unsloth UD4_K_XL in terms of what I have built. Some of the smaller benchmark data is out of date, like I think I got 32GB MoE cache V100 up to about 700-1000pps.
Thank you, I will point some people that I know that use this hardware to your engine.
Good to see this re-posted, I posted it two days ago but it sank without a trace so I'm really happy you posted it again. These V100's are a pretty neat GPU and one of very few that is still - somewhat - affordable.
Is v100 still good enough for local LLM. I found some cheap 16GB variant, but my clanker told me it's too old for modern llm.
Might be worth depending on the price but GeForce Nvidia GPUs tend to be a better deal than the server class since their EULA doesn’t allow them to be put in datacenters.
v100 caps out at CUDA 12 and the 585 driver. He have like 6 or 8 of them at work (including a few of the 32GB variants) and it's a PITA having to work around that.