I'd be interested in seeing actual agent benchmarks (eg CC or Copilot CLI with grep removed and this tool instead).
For example, I have explored RTK and various LSP implementations and find that the models are so heavily RL'd with grep that they do not trust results in other forms and will continually retry or reread, and all token savings are lost because the model does not trust the results of the other tools.
Codex CLI is quite happy running RTK. Well with GPT 5.5 xhigh anyway
One thing that irks me is that when it doesn't support eg. a cli flag of find, it gives an error message rather than sending the full output of the command instead. Then the agent wastes tokens retrying, or worse, doesn't even try because the prompting may make them afraid to not run commands without rtk
Yeah we're also interested in doing this, it's on the roadmap together with optimization of the prompt and descriptions so that models have an easier time using it.
Perhaps anecdotally: we do use this tool ourselves of course, and it's been working pretty well so far. Anthropic models call it and seem to trust the results.
I forced Claude to have a global memory for RTK and my own AI memory system (GuardRails) which it happily uses both, the only times it doesnt use GuardRails is if I dont mention it at all, otherwise it always uses RTK unless RTK falls apart running a tool it does not support.
I did some evals with pi and GPT 5.5. I tested RTK on / headroom on / both on / both off (all with the standard pi system instructions and no AGENTS.md).
I forget the exact tests I used (a couple of the standard agent evals that people use, one python and one typescript because those are what I use).
I don't claim it was an exhaustive test, or even a good one. It's possible I could have spent a day or so tuning my AGENTS.md and the pi system prompt/tool instructions and gotten better results, because if there's one thing running evals taught me it's that subtle differences there can change the results a lot.
However, I got clearly better results with both off, enough to convince me to stop the tests immediately after 3 rounds.
The problem was that while context use did go down (sometimes), the number of turns to complete went up so the overall cost of the conversation was higher.
It's made me very aware of one thing: so many people are sharing these kind of tools, but either with zero evals (or suspiciously hard to reproduce), or in the case of this one, extensive benchmarks testing the wrong thing.
I'm sure this tool does use fewer tokens than grep, and the benchmarks prove it, but that's not what matters here. What matters is, does an agent using it get the same quality of work done more quickly and for lower cost?
I also like the index feature form https://maki.sh
Source code has a lot of structure, using a real parser instead of grepping and reading files can potentially save a lot of tokens
This looks great! I built a tool in the same space- and I found that the biggest challenge was often to get the agent to prefer to use the tool over bash tools. What’s your experience with that?
Hey, this is something we're actively investigating. We recently added a flag, `--include-text-files`, which, when set, also makes Semble index regular documents (i.e., markdown, text, json). This should also work relatively well.
Perhaps Python is their main language (they seem to be ML peeps, which would make that most likely), which means it's easier for them to do manual reviews even if they're using AI for implementing, etc.
I'd be interested in seeing actual agent benchmarks (eg CC or Copilot CLI with grep removed and this tool instead).
For example, I have explored RTK and various LSP implementations and find that the models are so heavily RL'd with grep that they do not trust results in other forms and will continually retry or reread, and all token savings are lost because the model does not trust the results of the other tools.
I just put something in my global CLAUDE.md (under ~/.Claude) asking it to use the LSP instead of grep and have never had this issue since.
can you share that prompt?
My q would have been this. Lsp solved this no?
Codex CLI is quite happy running RTK. Well with GPT 5.5 xhigh anyway
One thing that irks me is that when it doesn't support eg. a cli flag of find, it gives an error message rather than sending the full output of the command instead. Then the agent wastes tokens retrying, or worse, doesn't even try because the prompting may make them afraid to not run commands without rtk
how effective is RTK for you? worth using?
Wondering too
Yeah we're also interested in doing this, it's on the roadmap together with optimization of the prompt and descriptions so that models have an easier time using it.
Perhaps anecdotally: we do use this tool ourselves of course, and it's been working pretty well so far. Anthropic models call it and seem to trust the results.
I forced Claude to have a global memory for RTK and my own AI memory system (GuardRails) which it happily uses both, the only times it doesnt use GuardRails is if I dont mention it at all, otherwise it always uses RTK unless RTK falls apart running a tool it does not support.
I did some evals with pi and GPT 5.5. I tested RTK on / headroom on / both on / both off (all with the standard pi system instructions and no AGENTS.md).
I forget the exact tests I used (a couple of the standard agent evals that people use, one python and one typescript because those are what I use).
I don't claim it was an exhaustive test, or even a good one. It's possible I could have spent a day or so tuning my AGENTS.md and the pi system prompt/tool instructions and gotten better results, because if there's one thing running evals taught me it's that subtle differences there can change the results a lot.
However, I got clearly better results with both off, enough to convince me to stop the tests immediately after 3 rounds.
The problem was that while context use did go down (sometimes), the number of turns to complete went up so the overall cost of the conversation was higher.
It's made me very aware of one thing: so many people are sharing these kind of tools, but either with zero evals (or suspiciously hard to reproduce), or in the case of this one, extensive benchmarks testing the wrong thing.
I'm sure this tool does use fewer tokens than grep, and the benchmarks prove it, but that's not what matters here. What matters is, does an agent using it get the same quality of work done more quickly and for lower cost?
with AI the "they could so they never wondered if they should" will be a very frequent thing.
Better than grep obviously, but how does this compare to existing LSPs?
Or tools like `ck`: https://beaconbay.github.io/ck/
How does this compare with colgrep?
https://github.com/lightonai/next-plaid/tree/main/colgrep
Semantic code search seems like a useful tool for a human too. Not just for agents.
fantastic token savings and performance... but unlike grep it's probabilistic search on search terms.
is that an issue? the tiny model might not surface something important
I also like the index feature form https://maki.sh Source code has a lot of structure, using a real parser instead of grepping and reading files can potentially save a lot of tokens
Shouldn’t it be a part of the harness at least for local codebase? I wonder how many harnesses are doing that already.
I'm playing with PI as a custom harness ( for Claude code because that what is provided to me )
I will try that ! It make sense and I'm curious to see results, for this or any similar projects mentioned in the thread
allegedly this one is good for PI https://pi.dev/packages/@ff-labs/pi-fff
Would this replace something like codebase-memory-mcp[1] or improve when both is being used?
[1] - https://github.com/DeusData/codebase-memory-mcp
thanks for sharing!
How does it compare to context-mode or serina that are both well established now?
This looks great! I built a tool in the same space- and I found that the biggest challenge was often to get the agent to prefer to use the tool over bash tools. What’s your experience with that?
Setup hooks. Hooks are how your harness forces compliance with your own rules.
Is the benchmark measuring one-shot retrieval accuracy, or Coding agent response accuracy?
Hey! Co-author here. The benchmark currently only measures retrieval accuracy.
We’re interested in measuring it end to end and also optimizing, e.g. the prompt and tools, for this, but we just haven’t gotten around to it.
Two follow-ups:
1) How do you compare accuracy? by checking if the answer is in any of the returned grep/bm25/semble snippets?
2) How do you measure token use without the agent, prompt, and tools?
1) yes! It’s not accuracy, but ndcg 2) we assume that if the agent gets the correct answer in the returned snippets it does not need to read further
Wouldn't NDCG/token results vary wildly depending on the agent's query and the number of returned items?
e.g. agents often run `grep -m 5 "QUERY"` with different queries, instead of one big grep for all items.
The same holds for semble: the agent can fire off many different semble queries with different k/parameters.
I guess the point we’re trying to make is that you need fewer semble queries to achieve the same outcome, compared to grep+readfile calls.
Congratulations on the release!
Could you add fff to the benchmarks?
Does this work well for non-coding documents as well? Say api docs or AI memory files?
Hey, this is something we're actively investigating. We recently added a flag, `--include-text-files`, which, when set, also makes Semble index regular documents (i.e., markdown, text, json). This should also work relatively well.
Very useful thanks for sharing!
grep doesn't need tokens, so what is 98% fewer than zero?
You need readfile to do something with those tokens. Grep only gives you the matching lines, not the context.
`grep -C $NUM` ? ;)
Even so. Take a look at the NDCG numbers for grep. It's not pretty
very curious to give it a spin but why write a cli in python? would surely be faster and more portable with go or rust?
Perhaps Python is their main language (they seem to be ML peeps, which would make that most likely), which means it's easier for them to do manual reviews even if they're using AI for implementing, etc.