Q: Did you just way over-optimize for a specific CPU and tokenizer? How is it so fast?
No, I way over-optimized for every combination of these! The results are very consistent across CPUs (modern x86 and ARM), and across specific tokenizers.
The major improvements are in optimizing heavily an implementation that usually is outsourced to a Regex engine (pretokenization) using SIMD, minimizing branching and other tricks, as well as heavily optimizing caching of pretoken mappings (if a word has been seen before, look it up its encoded tokens efficiently). Caching is a very hard problem in this domain since the cache grows very quickly, and pretoken distributions are very long-tailed.
Finally, interactions with Python are minimized, and threads have minimal interactions with each other.
Author here: Actually, depending on the nature of the inference you're doing it can be quite significant. Here are some numbers for time-to-first-token (time to process the entire input and produce the first token of output) for an 8B Qwen model on a single B200. Obviously these numbers are more significant with smaller models and on faster GPUs. Credit to fastokens [0] for the benchmark.
I run an AI platform and we need to tokenize fast and early to make a lot of decisions on the subsequent steps (things like routing, rate limiting and such). Its really important to do this efficiently even though its not a large % of total end to end time for the request.
To concur it's "latency critical", not "performance critical", people often confuse those two - optimize it all, but especially the chained critical path latency!
Same here, as we're sitting in the middle between requests and what budget constraints are allowed given a particular token allowance there can be 10 ~ 100 milliseconds improvement in the UX (TTFT) given such massive tokenization speed up.
1/1000 of inference compute is a non-trivial workload at scale. Gartner estimates ~$28B in inference spend for 2026 making this a $28 million dollar per year workload (edit: based on the assumption above)
Totally, edited my comment to specify "based on the assumption above." The main takeaway I was going for was 0.1% is not a small number in this context
Time to first token refers to the time until the model outputs one token, which includes the time to process the entire prompt (doing prefill). The GPU time per token is much lower when doing prefill, so the significance of tokenization is higher.
I initially had a Rust-based word cloud generator that generates word clouds in high resolution in ~100 milliseconds whereas it would take other generators a couple seconds to do the same thing. Does the world need a super-fast word cloud generator? No. Do I want a super-fast word cloud generator? Yes.
I later found enough optimizations to reduce the generation speed all the way down to ~16ms. Do I need a word cloud generator that fast? No. But if I have a word cloud generator it's going to be as fast as possible dammit.
Cool stuff. From my understanding, this is less valuable at inference time and more useful when running offline pre-training data prep.
When tokenizing terabytes of text for your training corpus, the speedup here is probably doing real work in saving you time (and money?). You get a faster iteration cycle when figuring out and adjusting your datasets.
Eh, linear algebra changes are still easy to measure correctness, it's just that you're competing with 50 years of research for most of them, less low hanging fruit.
Practically I would need to wait for hugging face models to adopt this? My harness tokenizer is just an estimate since the model tokenizes on my api calls?
Author here! In my case it's mostly pretraining experiments, where you might want to change your data mixture/filtering/processing of training data, and splits are usually done at a token-level instead of a text level. In this case we usually run for days on a huge number of CPUs to finish tokenizing something like DCLM.
From what I can tell it's also useful for inference when considering time-to-first-token (TTFT) as reported by fastokens.[0]
I'm not sure about the proprietary inference engines, but in the open source ones tokenization is done before looking up if a text sequence is present in the KV-cache. If you have a long prefix that's been seen before (say a system prompt), the time for tokenizing that will be a large part of your TTFT. The tokenizer cache should be warmed up in this case, so the throughput for Gigatoken would be significantly higher than reported in the repo.
> I'm not sure about the proprietary inference engines, but in the open source ones tokenization is done before looking up if a text sequence is present in the KV-cache
Is this necessary? Tokenisation is deterministic, so for a hit/miss check you can lookup on (a hash of) the source text instead of the tokens. You only need the tokens once you're seeking for the exact token index having determined there is a hit. That means tokenisation can proceed in parallel with your cache query, and since these caches are distributed in production systems I imagine the query itself could be slow.
I'm not trying to undermine the utility, and this is obviously excellent work. Being able to tokenise faster on the client also seems useful (precise token counts for context pruning heuristics, instead of `chars / 4`), and on a phone your work translates directly to energy savings. I'm just curious about the cache lookup point.
It's usually not as binary as "hit" or "miss" with a prefix cache, and you need to know the token boundaries to know where the cache hit ends.
The current structures used for KV-caching in vLLM and SGLang work by chunking the KV-cache tokens into prefix trees, and you need to hash chunks of tokens in order to look up in these, meaning you need to be able to slice up your tokens by token count.
Again I have no idea what proprietary engines are doing, but this is why open source stuff needs to tokenize before cache lookup at least.
> and you need to hash chunks of tokens in order to look up in these, meaning you need to be able to slice up your tokens by token count.
The hash just has to uniquely identify the contents. I still don't see what stops you from walking the chunk tree by chunks of characters instead of chunks of tokens, then lazily finding the token boundary once you've found the longest common chunk prefix and also (in parallel) tokenized the input.
You can, but this usually results in sequences with padding/truncation, since you won't know how many tokens your inputs map to before you actually tokenize them. This also makes shuffling difficult.
In practice every training project I've worked on does tokenization in a separate data processing phase.
Wait, since when does it matter whether something being hyper-optimized is useful? The computer going brrrr on an interesting problem is in itself the goal!
It didn’t come off as dismissive to me. I was curious as well as to where such optimizing helps and knew that the answers to your question would help me discover use cases I didn’t think of
I worked on a system a couple years ago with a BERT-based model (64M parameters) used for classification. The rest of the system could process data at gigabytes per second, and so here tokenization at a measly few megabytes per second really slowed things down. The model inference was more expensive than tokenization, but tokenization was still >10% of total runtime.
If you are training an LLM, you need to tokenize the text before it’s trained on. A lot of time this can be done in parallel with the GPU though.
I have spent way too much time waiting 10-15 minutes tokenizing my training dataset only for the run to crash over some minor bug after that. (If I was smarter, I’d test on a smaller batch first.)
It can be useful for checking input token usage before sending it to the model, e.g. preventing calls above a given token bound or grouping requests into batches.
It can also be used by the LLMs to provide the input and output token counts on the different APIs, though I'm not sure if this is how llama.cpp or other OpenAI-like APIs calculate the input/output tokens of a request.
I've data where i cannot store metadata that i need to search semantically so i embed it on the fly at every search with static embedding and tokenizing was more than 99% of the cpu time. Granted that was due the naive implementation of the default tokenizer which was o^2 with document length and just switching to a proper scanner solved most of it without going to simd and whatnot, but still.
Numbers are for the Gigatoken API, but compatibility mode just means eating a bunch of Python overhead (creating lists, reading strings to bytes). You can expect a modest ~200-300x speedup with compatibility mode depending on how you use it.
I can add some benchmarks for compatibility mode in the future. I have a little more juice to squeeze out of the Python interop though, so not quite ready for it yet.
Both the example libraries compared (tokenizers and tiktoken) are Rust-based with Python bindings. There's just a few levers in Rust that can speed it up even more particularly with LLM assistance as the AI Use Discloure here notes:
> Final profiling stages and the last ~4x worth of performance from eliminating branching and improving the pretoken cache hierarchy
1 year ago everyone would have called you insane for suggesting this. Now we all shrug and say yeah maybe we can do this and it’s actually a good idea?
Interesting :
Q: Did you just way over-optimize for a specific CPU and tokenizer? How is it so fast? No, I way over-optimized for every combination of these! The results are very consistent across CPUs (modern x86 and ARM), and across specific tokenizers.
The major improvements are in optimizing heavily an implementation that usually is outsourced to a Regex engine (pretokenization) using SIMD, minimizing branching and other tricks, as well as heavily optimizing caching of pretoken mappings (if a word has been seen before, look it up its encoded tokens efficiently). Caching is a very hard problem in this domain since the cache grows very quickly, and pretoken distributions are very long-tailed.
Finally, interactions with Python are minimized, and threads have minimal interactions with each other.
This is awesome, but tokenization is typically <0.1% of total inference time.
Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!
Author here: Actually, depending on the nature of the inference you're doing it can be quite significant. Here are some numbers for time-to-first-token (time to process the entire input and produce the first token of output) for an 8B Qwen model on a single B200. Obviously these numbers are more significant with smaller models and on faster GPUs. Credit to fastokens [0] for the benchmark.
sglang_speed [baseline]: mean=10.31ms median=6.48ms p99=45.98ms rps=96.8 sglang_speed [gigatoken]: mean=10.13ms median=6.54ms p99=45.16ms rps=98.4
input_len= 2048: TTFT mean 30.74 -> 29.05 ms (+5.5% reduction) | median 31.00 -> 28.80 (+7.1%) | p99 33.02 -> 32.02 (+3.0%) input_len= 8192: TTFT mean 105.20 -> 96.36 ms (+8.4% reduction) | median 103.87 -> 95.49 (+8.1%) | p99 126.88 -> 113.84 (+10.3%) input_len= 32768: TTFT mean 687.05 -> 633.66 ms (+7.8% reduction) | median 708.14 -> 657.35 (+7.2%) | p99 728.95 -> 678.79 (+6.9%)
These are preliminary numbers, so I will need to do some more testing before including this in the README.
[0] https://github.com/crusoecloud/fastokens
I run an AI platform and we need to tokenize fast and early to make a lot of decisions on the subsequent steps (things like routing, rate limiting and such). Its really important to do this efficiently even though its not a large % of total end to end time for the request.
To concur it's "latency critical", not "performance critical", people often confuse those two - optimize it all, but especially the chained critical path latency!
Same here, as we're sitting in the middle between requests and what budget constraints are allowed given a particular token allowance there can be 10 ~ 100 milliseconds improvement in the UX (TTFT) given such massive tokenization speed up.
‘I run an AI platform’ I have so many genuine questions I don’t even know where to start.
Pick one. I would like to hear it.
1/1000 of inference compute is a non-trivial workload at scale. Gartner estimates ~$28B in inference spend for 2026 making this a $28 million dollar per year workload (edit: based on the assumption above)
Source: https://www.gartner.com/en/newsroom/press-releases/2026-07-2...
The issue is it’s cpu compute which is underutilized in gpu clusters anyway, so practically it’s not really 1/1000.
Totally, edited my comment to specify "based on the assumption above." The main takeaway I was going for was 0.1% is not a small number in this context
Always good to make it 0.001%
Time to first token, especially for smaller models, can be sharply reduced.
Latency can be just as important as overall throughput, especially for inference providers like Groq and Cerebras.
Tokenization is <0.1% of the inference time for the first token in the same way it is <0.1% for the last.
Time to first token refers to the time until the model outputs one token, which includes the time to process the entire prompt (doing prefill). The GPU time per token is much lower when doing prefill, so the significance of tokenization is higher.
Have you done preliminary numbers on replacing tokenizer on, say, llama-server?
Running the numbers now
engineering effort to make something 1000x faster that accounts for 0.1% of total runtime is the most software developer thing imaginable
I initially had a Rust-based word cloud generator that generates word clouds in high resolution in ~100 milliseconds whereas it would take other generators a couple seconds to do the same thing. Does the world need a super-fast word cloud generator? No. Do I want a super-fast word cloud generator? Yes.
I later found enough optimizations to reduce the generation speed all the way down to ~16ms. Do I need a word cloud generator that fast? No. But if I have a word cloud generator it's going to be as fast as possible dammit.
"The pursuit of excellence does not need justification."
https://x.com/mitchellh/status/2074225453217505494
This depends on what your workflow is. There are use cases for tokenization that don't always involve immediately feeding the text into a model.
If you're tokenizing to run a tiny SLM for routing purposes, it can be way more than 0.1%.
This is the "GPU driver optimizations don't matter because PC's sit idle at the desktop most of the time" mindset.
It's pretty funny but then again, why not if it's as trivial to simplify as it appears
1000x faster on 0.1% of runtime = 0.1% saved. Amdahl remains undefeated.
Globally that’s ~50 GWh/yr, or ~5.7 MW continuous:
• 4,700 American homes
• a week of British tea
Cool stuff. From my understanding, this is less valuable at inference time and more useful when running offline pre-training data prep.
When tokenizing terabytes of text for your training corpus, the speedup here is probably doing real work in saving you time (and money?). You get a faster iteration cycle when figuring out and adjusting your datasets.
I had to stare at that chart for a minute just to let the numbers sink in. It's genuinely mind-bending, incredible ship OP
So the question becomes, how many other parts of the inference pipeline have left 1000x optimization opportunities lying on the table?
The problem with the rest of inference is that changes are not trivially correct or incorrect, as they are with the tokenization layer.
Eh, linear algebra changes are still easy to measure correctness, it's just that you're competing with 50 years of research for most of them, less low hanging fruit.
the answer is many! This would take hours to write. Full teams and research on nearly every part. So many 'unlocks' coming.
I'm sure there's been a lot more effort put into the other, more consequential, portions of inference time.
Practically I would need to wait for hugging face models to adopt this? My harness tokenizer is just an estimate since the model tokenizes on my api calls?
What sort of setups do people have that are bounded by the speed of the tokenizer?
Author here! In my case it's mostly pretraining experiments, where you might want to change your data mixture/filtering/processing of training data, and splits are usually done at a token-level instead of a text level. In this case we usually run for days on a huge number of CPUs to finish tokenizing something like DCLM.
From what I can tell it's also useful for inference when considering time-to-first-token (TTFT) as reported by fastokens.[0]
I'm not sure about the proprietary inference engines, but in the open source ones tokenization is done before looking up if a text sequence is present in the KV-cache. If you have a long prefix that's been seen before (say a system prompt), the time for tokenizing that will be a large part of your TTFT. The tokenizer cache should be warmed up in this case, so the throughput for Gigatoken would be significantly higher than reported in the repo.
[0] https://github.com/crusoecloud/fastokens
> I'm not sure about the proprietary inference engines, but in the open source ones tokenization is done before looking up if a text sequence is present in the KV-cache
Is this necessary? Tokenisation is deterministic, so for a hit/miss check you can lookup on (a hash of) the source text instead of the tokens. You only need the tokens once you're seeking for the exact token index having determined there is a hit. That means tokenisation can proceed in parallel with your cache query, and since these caches are distributed in production systems I imagine the query itself could be slow.
I'm not trying to undermine the utility, and this is obviously excellent work. Being able to tokenise faster on the client also seems useful (precise token counts for context pruning heuristics, instead of `chars / 4`), and on a phone your work translates directly to energy savings. I'm just curious about the cache lookup point.
It's usually not as binary as "hit" or "miss" with a prefix cache, and you need to know the token boundaries to know where the cache hit ends.
The current structures used for KV-caching in vLLM and SGLang work by chunking the KV-cache tokens into prefix trees, and you need to hash chunks of tokens in order to look up in these, meaning you need to be able to slice up your tokens by token count.
Again I have no idea what proprietary engines are doing, but this is why open source stuff needs to tokenize before cache lookup at least.
> and you need to hash chunks of tokens in order to look up in these, meaning you need to be able to slice up your tokens by token count.
The hash just has to uniquely identify the contents. I still don't see what stops you from walking the chunk tree by chunks of characters instead of chunks of tokens, then lazily finding the token boundary once you've found the longest common chunk prefix and also (in parallel) tokenized the input.
Very cool, thanks.
Can't you tokenize in preloading on demand?
You can, but this usually results in sequences with padding/truncation, since you won't know how many tokens your inputs map to before you actually tokenize them. This also makes shuffling difficult.
In practice every training project I've worked on does tokenization in a separate data processing phase.
Wait, since when does it matter whether something being hyper-optimized is useful? The computer going brrrr on an interesting problem is in itself the goal!
That's fair, I just figure there are useful scenarios as well. Apologies if I came off as dismissive!
It didn’t come off as dismissive to me. I was curious as well as to where such optimizing helps and knew that the answers to your question would help me discover use cases I didn’t think of
I worked on a system a couple years ago with a BERT-based model (64M parameters) used for classification. The rest of the system could process data at gigabytes per second, and so here tokenization at a measly few megabytes per second really slowed things down. The model inference was more expensive than tokenization, but tokenization was still >10% of total runtime.
If you are training an LLM, you need to tokenize the text before it’s trained on. A lot of time this can be done in parallel with the GPU though.
I have spent way too much time waiting 10-15 minutes tokenizing my training dataset only for the run to crash over some minor bug after that. (If I was smarter, I’d test on a smaller batch first.)
It can be useful for checking input token usage before sending it to the model, e.g. preventing calls above a given token bound or grouping requests into batches.
It can also be used by the LLMs to provide the input and output token counts on the different APIs, though I'm not sure if this is how llama.cpp or other OpenAI-like APIs calculate the input/output tokens of a request.
But are those bounded on the speed of tokenization?
I've data where i cannot store metadata that i need to search semantically so i embed it on the fly at every search with static embedding and tokenizing was more than 99% of the cpu time. Granted that was due the naive implementation of the default tokenizer which was o^2 with document length and just switching to a proper scanner solved most of it without going to simd and whatnot, but still.
Pre-training data is pre-tokenized ahead of time before being used to not waste any GPU compute.
A massive speedup like this is a nice efficiency savings on some of these data pipelines for sure.
This is really cool, great work!
Quality software here.
Very interesting project! Are there benchmarks for the "compatibility mode" or are all the numbers for the Gigatoken API?
Numbers are for the Gigatoken API, but compatibility mode just means eating a bunch of Python overhead (creating lists, reading strings to bytes). You can expect a modest ~200-300x speedup with compatibility mode depending on how you use it.
> a modest ~200-300x speedup with compatibility mode
marcelroed is modest, this speedup is not. Good work.
I can add some benchmarks for compatibility mode in the future. I have a little more juice to squeeze out of the Python interop though, so not quite ready for it yet.
wow, best release all week.
quite excellent software
We should just rewrite everything in Rust, especially bloated Python code, and the world would be a better place. ;) Disclosure: I'm a Rust advocate!
Both the example libraries compared (tokenizers and tiktoken) are Rust-based with Python bindings. There's just a few levers in Rust that can speed it up even more particularly with LLM assistance as the AI Use Discloure here notes:
> Final profiling stages and the last ~4x worth of performance from eliminating branching and improving the pretoken cache hierarchy
1 year ago everyone would have called you insane for suggesting this. Now we all shrug and say yeah maybe we can do this and it’s actually a good idea?
We should rewrite all Rust code in Python. Not for any technical reason. I'm just sick of the Rust cult at this point.