All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md.
What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly. Of course, it's always dependent on statistical noise + host system load, and running sufficiently large benchmarks is simply too expensive, so take em with a grain of salt.
Why does it work you may ask? Well, LLMs basically brute force words/phrases and pipe that into find/grep/pgrep/whatever (or as recently discussed here write a python script for it - https://news.ycombinator.com/item?id=49654229). Semantic search looks for similarities so you have to do less brute forcing. Comes of course at the cost of indexing everything first.
I should not trust their "vibe-coded productivity/token cost saving hacks" but I should trust yours?
Save 30% token costs when using Claude Code, Codex, OpenCode for free - with open source, local semantic search. Works for small and large codebases and monorepos! Enterprise-ready and fully compliant via Ollama and SQLite-vec.
Releases v0.0.42 Latest last month
Why should I trust that what you're peddling isn't snakeoil?
I literally say you should take benchmarks with a grain of salt :)
> Of course, it's always dependent on statistical noise + host system load, and running sufficiently large benchmarks is simply too expensive, so take em with a grain of salt.
And the savings listed are coming from a benchmark harness that implements different OSS bugs one time with and one without lumen - in those cases the % saved are reproducible (caveat: it was on older models, Opus 4.6 I believe).
Also I explain WHY it saves tokens - because the model doesn’t have to brute force different terms until it finds the match it needs, but uses semantic „distance“ so the embedding does it for the model.
I just went through a lot of benchmarking and the only thing that seemed better than rg was chunkhound, which sounds similar to this project. Actually a small Jina embedding model actually did better than voyage AI, but took a long time to index. Also chunkhound doesn’t work well with worktrees. In the end, I decided to stick with rg.
I'm in the process of evals for these tools after my org adopted them. My RTK findings are the same. It worsens task performance and overall you don't save money. I wanted to give the same treatment to other tools like ponytail and caveman (especially caveman, I mean there's no way that telling a computer to talk like a caveman is a valid engineering technique right?). To my horror, caveman is looking to be the only tool that actually doesn't regress on reasoning while taking costs down. But I still have a lot more evals to write, so this isn't conclusive or anything. (Also I haven't tried Lumen yet)
I am actually rather fond of caveman. I haven't evaluated it for token cost, in part because frankly I think that part of the pitch is a load of malarkey. Output that's shown to the user is such a small percentage of overall tokens these days.
But anecdotally I do think it saves me quite a lot of time on reading LLM outputs. And that, if nothing else, is good for my sanity.
The caveman gimmick makes sense to me as a clever hack. Caveman talk is a longstanding meme that's presumably well-represented in the models' training data. So just asking it to do that is just an ultra-concise way to tell the LLM to be ultra-concise. Which, in turn, is theoretically good for accuracy because putting too many instructions in the prompt is bad for task performance.
Similar for ponytail, I don’t know if it saves tokens, but there is less output to read (and usually less over engineering). Occasionally I have to push for more complex code, but that is much nicer than constantly asking for simpler code.
That's also a good point. When I'm using caveman (and especially cavekit), I don't have to spend quite so much energy on dealing with it building features I didn't ask for and don't want.
I spent way too long trying to reproduce the results in Pi and failing before I decided that I shouldn't trust author benchmarks for any of these tools. Then I found that I couldn't even close to reproduce their benchmark results using the exact model and their harness.
If any person other than the author has time to verify these Lumen benchmark results I'd be curious to hear it. I don't have the time to do it myself at the moment.
A bit different but in the same domain GitNexus has been really good for me. Symbol based search and impact analysis. Once I added git hooks to auto index and added some explicit instructions it saved quite a bit for me. I have no affiliation just wanted to add that as some thing to try out.
You use caveman to save on tokens, I use it to make the agent shut up. We are not the same =)
When caveman was released, token efficiency was a lot more relevant. But in the time I've actually come to like its matter-of-fact and short way of responding. There are other similar ones, but caveman is familiar.
skills have a pretty straightforward token win - you remove the decision making from the agent so it does specific things in a row. at some point youre better off inverting control to a script that only occasionally calls the agent for a small decision
Jetbrains IDEs are a perfect solution for this. They expose IDE actions (e.g, search, see occurrences, go to implementation) in their MCP server, which the harnesses can then call directly instead of figuring out the code themselves.
I also found LSP like skills to usually have no advantage over rg. The agent needs to read the code to understand it; navigation is a small portion of that.
It's pretty damn obvious to anyone who ever bothered to look at rtk gain output, no benchmark needed at all. Agent runs
rtk command-that-prints-100k-tokens | tail -5
costs 5 lines, maybe 100 tokens without rtk, but rtk will report 100k savings. Of course it doesn't know about that tail -5.
Worse, since rtk defaults to persisting that savings stat, it breaks sandboxing. Prefixing with rtk leads to random auto-mode denials from time to time too (this is independent of disabling savings stat persistence).
Honestly have no idea why anyone who knows the first thing about CLIs would take rtk gain seriously. I guess clueless vibe coders who has hardly ever worked in a terminal before will look at the stat and feel good about it?
That said, rtk is still mildly useful for compressing repeated test run outputs and stuff, but you should only ever use it on whitelisted commands; wrapping everything like they suggest you to do is just stupid.
It seems like most of these tools are mostly vaporware. Benchmarks done on Headroom and RTK show that neither result in real savings. If it were possible to have such a simple pre-process step why wouldn’t the AI Labs upstream the optimizations themselves?
My guess is they mostly don’t work or make the behavior much more confusing for the model. I really think there needs to be some kind of independent benchmark.
Here are other cases demonstrating the exact same issues with these kinds of tools:
Naively, I think some optimizations would require access to the whole codebase and that would make people nervous (plus incur more cost).
Eg absurd idea, but you could write something that minifies a codebase (by token, rather than byte) and then translates edits back into the expanded code. Probably an insane use of fuse lol. Partially minifying on each tool call sounds like a huge pain with a lot of state to track.
There’s also a lot of common situations where humans prefer solutions that take more tokens because it’s easier for us to read (eg for loop vs map vs list comprehension), which may have some gains.
I strongly suspect there is some form of token compression that works, but I don’t think it will be as simple as “pipe arbitrary text with no context into this tool”.
Jetbrains feels like a place this might come from. “Take this code, parse it to an AST, find the fewest token representation of it” feels like something they’d do, or maybe Astral (specifically in Python land, type checkers feel sort of adjacent as well).
I fully agree. Some of these tautologies, RAGs or RAG adjacent ideas feel like there is real potential. I know in my own testing adding a knowledge graph provided cost saving benefits.
Ultimately, the problem is people aren't actually benchmarking any of these ideas or if they are its via an extremely misleading methodology. For example, headroom violates cache frequently, nullifying any potential savings it could have; but does their benchmarks show that? Nope. [1]
As an aside, I wonder how many days are we away from Codex or Claude taking all the "read" tool calls and re-assembling a full map of your codebase and then offering it as a feature.
I think people don’t do genuine benchmarks because the market forces them to pretend their solution works for anything you can throw AI at. Companies whose valuation is based on them being the RAG/compression/routing/etc company. They can’t admit it only works well in a specific domain because then they’re immediately $300M in the hole.
I have more faith in companies with a more targeted approach. Eg gzip does fine, but video codecs beat compressing raw video by a ton.
> As an aside, I wonder how many days are we away from Codex or Claude
That sounds like SourceGraph but twice as expensive, although it does have “AI” so probably lol
> If it were possible to have such a simple pre-process step why wouldn’t the AI Labs upstream the optimizations themselves?
Not defending these tools, but one reason these might not be upstreamed is because it would negatively impact vendor margins, and they have no incentive to save their users money
Judging by the leaks, OpenAI and Anthropic already train reasoning traces to use fewer tokens (they deliberately omit articles and prepositions, use very short sentences, etc.), even though you pay per token. So it wouldn't make sense to do that if the only incentive was "make them pay for as many tokens as possible per task."
It's more subtle than that. If a user has to wait longer for a solution/pay more, they'll be less satisfied and may switch to a competitor. More unnecessary tokens also means more unnecessary compute. Longer sessions are increasingly more expensive to serve than shorter sessions.
And there's always the Jevons effect: as a resource becomes cheaper, demand often increases, and so does net resource consumption.
So, imho, frontier labs have every incentive to reduce token usage per task (while also making you use AI for more and more tasks in your daily life)
Average cost per attempt, without → with RTK:
Claude/Fable: $1.72 → $1.64 (~5% cheaper)
DeepSeek: $0.115 → $0.121 (~5% more expensive)
Almost all Claude savings came from a single task.
Excluding it, savings were under 1%.
It took me a few rereads to parse out the top-line.
This article really buries the lede.
Have not benchmarked it, the intent is mostly to save time, not necessarily tokens. I noticed that the models need a lot of toolcalls to ground themselves, and often have trouble with getting an overview.
Glad to see that more and more people realize these are just snake oils. Without objective metrics like benchmarks, none of the claims mean anything.
That's also how I feel about skills/plugins. While some provide important context for specific projects/environments, I am very skeptical about (over)generalized skills like "writing JS tests" or "creating a spec". There are dozens of these skills internally at my company, but I haven't seen a single benchmark that shows any of those are better than just plain, single sentence prompts in a meaningful way (aka statistically significant).
I see the same thing and have effectively the same philosophy. If I'm using something like figma or glean or playwright/chrome dev tools, plugin/skill/mcp - likely very useful.
But so many of the weird collections of skills that people on YouTube get viral followings for - I just don't get it.
People excitedly ask me what skills I use and I feel bad just saying only things we've directly authored for some express purpose. None of the "hot" ones.
I've written a large handful of skills, but they aren't like vim plugins. I don't just leave them "on".
This has been my experience at least- curious if I'm just behind the times.
I also effectively didn't leave the IDE+ChatGPT copy/paste workflow until the first release of Claude code. So maybe I'm slow to adopt.
The technique looked dubious from the start, because LLMs were trained to expect certain outputs from common bash tools. If the output is not what it expects, an LLM may issue more tool calls than before, because it will assume the tool is broken, the arguments passed to it were wrong, or it's a newer/older version of the tool etc => more tokens. Sounds like just adding to the prompt to use `grep` and `tail` extensively will do the trick without any special tooling.
This was the problem I saw. I installed rtk when it came out and liked the idea of it. But over time with newer model generations I kept seeing the model get confused in the reasoning text and retry a command bypassing rtk. I didn't even need a benchmark to see it was regularly an impediment to the final outcome.
This makes sense. “Don’t try to penny-pinch your employees” is a lesson most managers learn eventually, and I guess agent-orchestrators will have to learn it too.
I just spawn a subagent in the cheapest range (for example flash-lite) to summarise a tool use. It's the only way that has worked based on my benchmarks and generalises well.
Surprised this is so far down. A subagent with a cheap model like haiku or similar is the way to go instead of dumping tool output directly into the main agent context
Is there an easy way to do this with eg codex? It seems like eg sol agents can’t spawn Luna subagents, so eg a “code research” subagent can save the main agent’s context, but can’t save tokens necessarily. (I suppose a tool to call codex CLI would work, but a bit unsatisfying.)
Some months ago I was evaluating command output compressors to integrate into Dirac[1] as that seemed like an easy win that would compliment and compound with Dirac's other mechanisms.
I tested rtk among these and it was actually a net negative in both CPU time and accuracy, the latter would throw LLMs way off and make it hard to recover. If you are building a coding agent, I'd hard pass on rtk.
~ $ time grep Return * 2> /dev/null | wc -l
966
grep Return * 2> /dev/null 0.36s user 0.02s system 98% cpu 0.382 total
wc -l 0.00s user 0.00s system 1% cpu 0.380 total
~ $ time rtk grep Return * 2> /dev/null | wc -l
260
rtk grep Return * 2> /dev/null 4.10s user 17.10s system 92% cpu 23.008 total
wc -l 0.00s user 0.00s system 0% cpu 23.007 total
Much worse CPU consumption, and more importantly, plain wrong result. These kind of results compromise the entire agent performance because the model trusts wrong output. Without the correct results, any hypothetical savings are penny wise pound foolish
So yeah I am still on the lookout for a credible CLI wrapper, do let me know if you have any in mind.
Any magic tool that declares a savings of over 10% can be immediately classified as snake oil. You can check yourself, load any of those projects up in GitHub and notice the math is always extremely misleading. It will be something like theoretical input bytes, or amount of command stripped off, or some other lie.
If the tool won’t be upfront about those things, they are not worth looking into any further. It’s used car salesman strategy.
We have been working in this space for the past year. Based on our experience, I no longer trust any claims unless they are backed by benchmark results (yes, benchmarks are painful to run reliably and expensive).
It is possible to reduce token usage. It’s just much harder than the basic approach.
One quick win is to just avoid wasteful tokens, for example run all the QA tools like the unit tests in --quiet mode, which only prints warnings/failures.
I don't understand how this or all these magic skill bundles and methodologies get traction and why they are so popular. It's either plain worse or has serious trade offs.
Much earlier, I tried to set up some static analysis tools so that the coding agent would have access to dataflow analysis etc. tools instead of just grep for typed python. If there were benefits, they weren't easily apparent :(
I don't think they're comparable. RTK just modifies the output of CLI tools to reduce the number of tokens, a Roslyn MCP gives the agent a fundamentally superior way of interacting with a C# codebase.
Just another instance of the bitter lesson. The model itself knows how to be clever and conserve tokens in command output by using shell primitives and as the models get smarter they get better at anticipating large output and defensively adapting the input commands.
What if the next-gen models are trained on RTK output as well? Then you will actually have less tokens in the context window, and the model won't become confused (which would require more turns, wasting tokens)
doesn't change the fact that it doesn't do what it claims to now. I just don't care about vague promises and "trust us bro" vibes that tech is sold for nowadays. It claims x, it doesn't deliver x. Maybe it could in the future, or maybe not.
That's indeed not something that RTK can promise. However, if RTK becomes popular enough, it's unavoidable that its output will start seeping into the training data of future models, which will make RTK perform better
This is my first time hearing about RTK, and yikes! The benchmarks mean nothing; this thing is actively dangerous to use.
If my agent runs a shell command, show my agent the output of that shell command. I don't have a problem with automatically pruning or paginating large outputs, as long as the agent still has some form of access to the original output (e.g., by searching).
But removing verbosity from an 'ls' command? That's ridiculous! If my agent runs `ls -la`, do not drop the owner and date from the output, because THAT'S AN EXPECTED PART OF THE OUTPUT. Nothing should be trying to predict the agent's intent and interfering, unless you want your agent to get confused and fail.
These LLMs were trained on predictable shell behavior, and RTK deliberately subverts the model's expectations. There's no way that isn't degrading the model's capability.
All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md.
What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly. Of course, it's always dependent on statistical noise + host system load, and running sufficiently large benchmarks is simply too expensive, so take em with a grain of salt.
Why does it work you may ask? Well, LLMs basically brute force words/phrases and pipe that into find/grep/pgrep/whatever (or as recently discussed here write a python script for it - https://news.ycombinator.com/item?id=49654229). Semantic search looks for similarities so you have to do less brute forcing. Comes of course at the cost of indexing everything first.
You can find the project here: https://github.com/ory/lumen
I should not trust their "vibe-coded productivity/token cost saving hacks" but I should trust yours?
Why should I trust that what you're peddling isn't snakeoil?I literally say you should take benchmarks with a grain of salt :)
> Of course, it's always dependent on statistical noise + host system load, and running sufficiently large benchmarks is simply too expensive, so take em with a grain of salt.
And the savings listed are coming from a benchmark harness that implements different OSS bugs one time with and one without lumen - in those cases the % saved are reproducible (caveat: it was on older models, Opus 4.6 I believe).
Also I explain WHY it saves tokens - because the model doesn’t have to brute force different terms until it finds the match it needs, but uses semantic „distance“ so the embedding does it for the model.
Only way to find out is to do some testing yourself i think.
I’m using less tokens with Lumen but I also use a bunch of other tokens hacks/skills; it’s hard to measure the impact exactly but it feels significant
I just went through a lot of benchmarking and the only thing that seemed better than rg was chunkhound, which sounds similar to this project. Actually a small Jina embedding model actually did better than voyage AI, but took a long time to index. Also chunkhound doesn’t work well with worktrees. In the end, I decided to stick with rg.
Half of what we see or consume is hype, so we should view everything with nuance. Sometimes truth lies in the middle.
I'm in the process of evals for these tools after my org adopted them. My RTK findings are the same. It worsens task performance and overall you don't save money. I wanted to give the same treatment to other tools like ponytail and caveman (especially caveman, I mean there's no way that telling a computer to talk like a caveman is a valid engineering technique right?). To my horror, caveman is looking to be the only tool that actually doesn't regress on reasoning while taking costs down. But I still have a lot more evals to write, so this isn't conclusive or anything. (Also I haven't tried Lumen yet)
I am actually rather fond of caveman. I haven't evaluated it for token cost, in part because frankly I think that part of the pitch is a load of malarkey. Output that's shown to the user is such a small percentage of overall tokens these days.
But anecdotally I do think it saves me quite a lot of time on reading LLM outputs. And that, if nothing else, is good for my sanity.
The caveman gimmick makes sense to me as a clever hack. Caveman talk is a longstanding meme that's presumably well-represented in the models' training data. So just asking it to do that is just an ultra-concise way to tell the LLM to be ultra-concise. Which, in turn, is theoretically good for accuracy because putting too many instructions in the prompt is bad for task performance.
Similar for ponytail, I don’t know if it saves tokens, but there is less output to read (and usually less over engineering). Occasionally I have to push for more complex code, but that is much nicer than constantly asking for simpler code.
That's also a good point. When I'm using caveman (and especially cavekit), I don't have to spend quite so much energy on dealing with it building features I didn't ask for and don't want.
This sounds quite similar to dirac which made a stir a few months ago:
https://github.com/dirac-run/dirac
I spent way too long trying to reproduce the results in Pi and failing before I decided that I shouldn't trust author benchmarks for any of these tools. Then I found that I couldn't even close to reproduce their benchmark results using the exact model and their harness.
If any person other than the author has time to verify these Lumen benchmark results I'd be curious to hear it. I don't have the time to do it myself at the moment.
I did the same with another project that does the same thing, called ck, and wasn't able to wring out any improved performance over just plain grep.
Yes rg + python seems to beat far fancier tools.
A bit different but in the same domain GitNexus has been really good for me. Symbol based search and impact analysis. Once I added git hooks to auto index and added some explicit instructions it saved quite a bit for me. I have no affiliation just wanted to add that as some thing to try out.
You use caveman to save on tokens, I use it to make the agent shut up. We are not the same =)
When caveman was released, token efficiency was a lot more relevant. But in the time I've actually come to like its matter-of-fact and short way of responding. There are other similar ones, but caveman is familiar.
Isn't that what everyone was doing in the beginning of harnesses(cursor) and learned not to do for various reasons? https://news.ycombinator.com/item?id=43164253
That’s a good example, I believe it could be similar case like with RAG. I didn’t hear for a while about any recommendations to use RAG.
skills have a pretty straightforward token win - you remove the decision making from the agent so it does specific things in a row. at some point youre better off inverting control to a script that only occasionally calls the agent for a small decision
How does lumen compare to semble?
https://github.com/MinishLab/semble
Jetbrains IDEs are a perfect solution for this. They expose IDE actions (e.g, search, see occurrences, go to implementation) in their MCP server, which the harnesses can then call directly instead of figuring out the code themselves.
You would think, except in their own testing they found no benefit. Some measures it was even worse on.
https://blog.jetbrains.com/ai/2026/05/what-happens-when-you-...
I also found LSP like skills to usually have no advantage over rg. The agent needs to read the code to understand it; navigation is a small portion of that.
> One of: Claude Code, Cursor, Codex, or OpenCode
What makes it incompatible with Pi, Zed or any other harness?
Only the amount of free time I have to work on it - nothing fundamentally prevents it. PRs welcomed!
TBF, what difference does it make when the very harnesses you're using these "hacks" in are themselves vibe-coded?
Lumen is pretty cool, it's just local RAG, but it's a realistic approach at RAG. I would still keep tool-calling in some places though.
How does this compare to ripgrep and fzf?
> Whether it's caveman
the kinda guy who honestly thinks caveman.md reduces costs, actually adds it to his system prompt and is painstakingly reading the terse output
So we replace all those "hacks" and snake-oil with even more pseudoscientific hacks and snake-oil?
See my other comment on the person that commented the same thing you did!
It's pretty damn obvious to anyone who ever bothered to look at rtk gain output, no benchmark needed at all. Agent runs
costs 5 lines, maybe 100 tokens without rtk, but rtk will report 100k savings. Of course it doesn't know about that tail -5.Worse, since rtk defaults to persisting that savings stat, it breaks sandboxing. Prefixing with rtk leads to random auto-mode denials from time to time too (this is independent of disabling savings stat persistence).
Honestly have no idea why anyone who knows the first thing about CLIs would take rtk gain seriously. I guess clueless vibe coders who has hardly ever worked in a terminal before will look at the stat and feel good about it?
That said, rtk is still mildly useful for compressing repeated test run outputs and stuff, but you should only ever use it on whitelisted commands; wrapping everything like they suggest you to do is just stupid.
It seems like most of these tools are mostly vaporware. Benchmarks done on Headroom and RTK show that neither result in real savings. If it were possible to have such a simple pre-process step why wouldn’t the AI Labs upstream the optimizations themselves? My guess is they mostly don’t work or make the behavior much more confusing for the model. I really think there needs to be some kind of independent benchmark.
Here are other cases demonstrating the exact same issues with these kinds of tools:
https://blog.jetbrains.com/ai/2026/07/rtk-claude-code-token-... https://brandonbarker.me/writing/headroom-fewer-tokens-bigge...
Naively, I think some optimizations would require access to the whole codebase and that would make people nervous (plus incur more cost).
Eg absurd idea, but you could write something that minifies a codebase (by token, rather than byte) and then translates edits back into the expanded code. Probably an insane use of fuse lol. Partially minifying on each tool call sounds like a huge pain with a lot of state to track.
There’s also a lot of common situations where humans prefer solutions that take more tokens because it’s easier for us to read (eg for loop vs map vs list comprehension), which may have some gains.
I strongly suspect there is some form of token compression that works, but I don’t think it will be as simple as “pipe arbitrary text with no context into this tool”.
Jetbrains feels like a place this might come from. “Take this code, parse it to an AST, find the fewest token representation of it” feels like something they’d do, or maybe Astral (specifically in Python land, type checkers feel sort of adjacent as well).
I fully agree. Some of these tautologies, RAGs or RAG adjacent ideas feel like there is real potential. I know in my own testing adding a knowledge graph provided cost saving benefits.
Ultimately, the problem is people aren't actually benchmarking any of these ideas or if they are its via an extremely misleading methodology. For example, headroom violates cache frequently, nullifying any potential savings it could have; but does their benchmarks show that? Nope. [1]
As an aside, I wonder how many days are we away from Codex or Claude taking all the "read" tool calls and re-assembling a full map of your codebase and then offering it as a feature.
[1] https://docs.headroomlabs.ai/docs/benchmarks
I think people don’t do genuine benchmarks because the market forces them to pretend their solution works for anything you can throw AI at. Companies whose valuation is based on them being the RAG/compression/routing/etc company. They can’t admit it only works well in a specific domain because then they’re immediately $300M in the hole.
I have more faith in companies with a more targeted approach. Eg gzip does fine, but video codecs beat compressing raw video by a ton.
> As an aside, I wonder how many days are we away from Codex or Claude
That sounds like SourceGraph but twice as expensive, although it does have “AI” so probably lol
> If it were possible to have such a simple pre-process step why wouldn’t the AI Labs upstream the optimizations themselves?
Not defending these tools, but one reason these might not be upstreamed is because it would negatively impact vendor margins, and they have no incentive to save their users money
Judging by the leaks, OpenAI and Anthropic already train reasoning traces to use fewer tokens (they deliberately omit articles and prepositions, use very short sentences, etc.), even though you pay per token. So it wouldn't make sense to do that if the only incentive was "make them pay for as many tokens as possible per task."
It's more subtle than that. If a user has to wait longer for a solution/pay more, they'll be less satisfied and may switch to a competitor. More unnecessary tokens also means more unnecessary compute. Longer sessions are increasingly more expensive to serve than shorter sessions.
And there's always the Jevons effect: as a resource becomes cheaper, demand often increases, and so does net resource consumption.
So, imho, frontier labs have every incentive to reduce token usage per task (while also making you use AI for more and more tasks in your daily life)
Even JetBrains is now AI blog-slop, how disappointing.
Not all, but definitely on some. I also find it very off-putting.
Main takeaway:
It took me a few rereads to parse out the top-line. This article really buries the lede.I wrote article about it couple months ago that I didn't believe it works. Nice to see numbers now
https://mroczek.dev/articles/the-token-compression-illusion-...
Yes, your article was noticed in my circle, but numbers was what was lacking
I use treesitter to build outlines of files and directories: https://github.com/resolveworks/trace
Have not benchmarked it, the intent is mostly to save time, not necessarily tokens. I noticed that the models need a lot of toolcalls to ground themselves, and often have trouble with getting an overview.
Glad to see that more and more people realize these are just snake oils. Without objective metrics like benchmarks, none of the claims mean anything.
That's also how I feel about skills/plugins. While some provide important context for specific projects/environments, I am very skeptical about (over)generalized skills like "writing JS tests" or "creating a spec". There are dozens of these skills internally at my company, but I haven't seen a single benchmark that shows any of those are better than just plain, single sentence prompts in a meaningful way (aka statistically significant).
I see the same thing and have effectively the same philosophy. If I'm using something like figma or glean or playwright/chrome dev tools, plugin/skill/mcp - likely very useful.
But so many of the weird collections of skills that people on YouTube get viral followings for - I just don't get it.
People excitedly ask me what skills I use and I feel bad just saying only things we've directly authored for some express purpose. None of the "hot" ones.
I've written a large handful of skills, but they aren't like vim plugins. I don't just leave them "on".
This has been my experience at least- curious if I'm just behind the times.
I also effectively didn't leave the IDE+ChatGPT copy/paste workflow until the first release of Claude code. So maybe I'm slow to adopt.
The technique looked dubious from the start, because LLMs were trained to expect certain outputs from common bash tools. If the output is not what it expects, an LLM may issue more tool calls than before, because it will assume the tool is broken, the arguments passed to it were wrong, or it's a newer/older version of the tool etc => more tokens. Sounds like just adding to the prompt to use `grep` and `tail` extensively will do the trick without any special tooling.
This was the problem I saw. I installed rtk when it came out and liked the idea of it. But over time with newer model generations I kept seeing the model get confused in the reasoning text and retry a command bypassing rtk. I didn't even need a benchmark to see it was regularly an impediment to the final outcome.
This makes sense. “Don’t try to penny-pinch your employees” is a lesson most managers learn eventually, and I guess agent-orchestrators will have to learn it too.
I just spawn a subagent in the cheapest range (for example flash-lite) to summarise a tool use. It's the only way that has worked based on my benchmarks and generalises well.
Surprised this is so far down. A subagent with a cheap model like haiku or similar is the way to go instead of dumping tool output directly into the main agent context
Is there an easy way to do this with eg codex? It seems like eg sol agents can’t spawn Luna subagents, so eg a “code research” subagent can save the main agent’s context, but can’t save tokens necessarily. (I suppose a tool to call codex CLI would work, but a bit unsatisfying.)
Some months ago I was evaluating command output compressors to integrate into Dirac[1] as that seemed like an easy win that would compliment and compound with Dirac's other mechanisms.
I tested rtk among these and it was actually a net negative in both CPU time and accuracy, the latter would throw LLMs way off and make it hard to recover. If you are building a coding agent, I'd hard pass on rtk.
Much worse CPU consumption, and more importantly, plain wrong result. These kind of results compromise the entire agent performance because the model trusts wrong output. Without the correct results, any hypothetical savings are penny wise pound foolishSo yeah I am still on the lookout for a credible CLI wrapper, do let me know if you have any in mind.
[1] https://dirac.run/
Any magic tool that declares a savings of over 10% can be immediately classified as snake oil. You can check yourself, load any of those projects up in GitHub and notice the math is always extremely misleading. It will be something like theoretical input bytes, or amount of command stripped off, or some other lie.
If the tool won’t be upfront about those things, they are not worth looking into any further. It’s used car salesman strategy.
Open question, how does this instruction in agent-rules.md look?
"Cap large/unknown command output: `COMMAND 2>&1 | head -c 4000`. Never stream full logs, tests, or large files."
I use that instead of RTK. Empirically, I found RTK makes my agents run longer to complete similar tasks.
Ponytail and Caveman seem to help somewhat.
I think anyone who is even a little bit realistic knows that most technologies overclaim, or evaluate under very favorable conditions.
This is not a good thing of course, but I also feel that acting surprised that this is going on is a little unnecessary.
Having said that: most tools are not helpful
If you are using maven you should tell your agent to use its quiet mode or rtk, because mvn love to write a lot of useless output.
I believe RTK would work well in that use case.
Sometimes creating less verbose variants yourself (a simple script, build.sh, with pointers to logs) can be a quick win.
We have been working in this space for the past year. Based on our experience, I no longer trust any claims unless they are backed by benchmark results (yes, benchmarks are painful to run reliably and expensive).
It is possible to reduce token usage. It’s just much harder than the basic approach.
One quick win is to just avoid wasteful tokens, for example run all the QA tools like the unit tests in --quiet mode, which only prints warnings/failures.
I don't understand how this or all these magic skill bundles and methodologies get traction and why they are so popular. It's either plain worse or has serious trade offs.
It is just "putting a wet phone in rice" of AI
i don't know if such hacks works, but in C# if you use roslyn mcp, you save a lot.
Much earlier, I tried to set up some static analysis tools so that the coding agent would have access to dataflow analysis etc. tools instead of just grep for typed python. If there were benefits, they weren't easily apparent :(
I haven't tried it since it was first released but it didn't seem to work at all for me back then.
It was so slow that the roslyn results would be lagged well behind any edits it was making, which would just leave it confused.
I don't think they're comparable. RTK just modifies the output of CLI tools to reduce the number of tokens, a Roslyn MCP gives the agent a fundamentally superior way of interacting with a C# codebase.
Yes. and i find model makes less errors and reasoning the codebase well, especially when you do a large refactor.
My main issue with rtk is that rtk randomly messes modification and agent start polling same tool continuously.
Which one specifically? The ones I found looked like unmaintained throw-away experiments.
I’m using dpupek/roslyn-code-navigator (codenav-mcp) with Codex.
rtk gain mechanism is oversimplified. 1 token != 4 bytes for the standard prompt / context window at coding agent.
Just another instance of the bitter lesson. The model itself knows how to be clever and conserve tokens in command output by using shell primitives and as the models get smarter they get better at anticipating large output and defensively adapting the input commands.
well yeah.... now you're giving it output it wasn't trained on.
What if the next-gen models are trained on RTK output as well? Then you will actually have less tokens in the context window, and the model won't become confused (which would require more turns, wasting tokens)
doesn't change the fact that it doesn't do what it claims to now. I just don't care about vague promises and "trust us bro" vibes that tech is sold for nowadays. It claims x, it doesn't deliver x. Maybe it could in the future, or maybe not.
That's indeed not something that RTK can promise. However, if RTK becomes popular enough, it's unavoidable that its output will start seeping into the training data of future models, which will make RTK perform better
So it's a kind of self-fulfilling prophecy
This is my first time hearing about RTK, and yikes! The benchmarks mean nothing; this thing is actively dangerous to use.
If my agent runs a shell command, show my agent the output of that shell command. I don't have a problem with automatically pruning or paginating large outputs, as long as the agent still has some form of access to the original output (e.g., by searching).
But removing verbosity from an 'ls' command? That's ridiculous! If my agent runs `ls -la`, do not drop the owner and date from the output, because THAT'S AN EXPECTED PART OF THE OUTPUT. Nothing should be trying to predict the agent's intent and interfering, unless you want your agent to get confused and fail.
These LLMs were trained on predictable shell behavior, and RTK deliberately subverts the model's expectations. There's no way that isn't degrading the model's capability.