Coincidentally I've been playing with small (~30B) self-hostable models for coding tasks today -- specifically plugging them into Cloudflare OS (which I work on) and asking each to build a collaborative whiteboard.
I'm finding that the Mixture-of-Experts (MoE) models (Qwen 3.6-35B, and Nemotron 3.5 Lightning) are, well, terrible at this. They just couldn't get the job done at all. Went way off the rails. They are really fast though!
Whereas ~30B dense models (not MoE) are pretty decent. I tried Muse Glimmer, Gemma 4-31B, Qwen 3.6-27B, and Laguna XS[0]. They were all able to build a working collaborative whiteboard app, without any guidance (other than feeding back error logs to the model). I also asked each to then draw a monkey by calling the API of the whiteboard it has just built. Laguna drew random scribbles but the rest all managed to produce something monkey-like.
(Frontier models in comparison will write the app in one shot with no errors at all.)
Note that both Qwen 3.6 and Gemma 4 each have both MoE and dense variants. I find this very confusing, because e.g. ollama's model index typically only distinguishes variants by their size, but MoE vs. dense makes a huge difference in how they actually perform. IMO they should use a suffix, like Qwen 3.6-moe vs. Qwen 3.6-dense, or maybe Qwen 3.6-fast vs. Qwen 3.6-smart...
[0] EDIT: Turns out Laguna XS is MoE, I misunderstood. It performed similarly to the dense models. But maybe this explains why it couldn't write code and think about monkey shapes at the same time!
How they have any credibility after they told everyone they could run Deepseek R1 on their laptop (by giving the Qwen 7B distill the `deepseek` moniker)...
There is an old rule of thumb that says the quality of an MoE is equivalent to a dense model with the geometric mean of its total and active parameters. So, for example, the Qwen3.6 would be equivalent to a dense model with approximately sqrt(35×3) ≈ 10.25B parameters.
Both MoEs and dense models are always getting better, so I don't think this comparison is meaningful across generations. But still for a first approximation, this tends to hold (you wouldn't expect a lot from a 10B model in coding yet).
I've been pretty impressed with Laguna. I downloaded their coding agent and have used it for a task here and there (the larger variant). Obviously it's nothing like a frontier LLM, but it surprised me with how good it was. And I think the model personality and way it talks is pretty pleasant
Only if you know what it means. To many people these suffixes are gibberish. Also ollama doesn't actually show that suffix in its model index. Don't say "don't use ollama", you sound like a Linux user telling people not to use Mac. (I say this as a Linux user who despises Apple products myself.)
If you haven't yet, you might want to try gemma-4-E4B-it-qat or gemma-4-12B-it-qat with Structured Outputs. My main use case is tagging photos and generating headlines.
As an aside, the bigger S 2.1 runs faster than the dense Qwen 3.6 and Gemma 4 models on the same hardware (assuming the same hardware is big enough to run it) and definitely feels smarter, and more capable of long tasks, but it doesn't feel as heavily tuned for code as Qwen 3.6.
One major consequence of the ramapocalypse, I think, is an even higher focus on small efficient models. I personally believe that the multi-trillion parameter models are fundamentally missing things and the push to smaller, more efficient will drive evolutionary structural changes that will lead to future gains
I'd assume the closed weight models are all working on shrinking their parameter counts anyways. They too benefit from smaller models. It'd be foolish for these SOTA labs to not be working at reducing parameter counts.
The incentive is there, but their money making niche is to solve problems you can't solve locally with a 30b, so they're unlikely to stray into territory owned by ultra cheap to run open weights models.
I would say even without rampocalypse there would still be the strong incentive to innovate at the edge and under more extreme constraints. The incentives are just even stronger now.
I'm looking forward to seeing what types of new things people create over the coming years once there is less obsession with massive unwieldy LLMs. I think the incentives are just too strong to ignore.
With the benefit of LLMs already being proven, in a couple of years we will have vastly better hardware for inference I guess.
I feel like now hardware is stagnating a bit, because the software side has moved too fast for the hardware to catch up. Once we settle on some good, optimal software architecture for the models, dedicated hardware will easily increase throughout by 10x or 100x, for a fraction of the cost.
LLMs seems quite simple, maybe we'll be able to print/assemble at home our own chips with the desired models/weights.
Maybe we'll have model weights being shared like game cartridges.
Well, everything—even human life on Earth—is just temporary, but RAM supply lagging centralized-AI-driven demand increases continuing to squeeze the consumer market may not be a short term phenomenon.
> LLMs seems quite simple, maybe we'll be able to print/assemble at home our own chips with the desired models/weights.
So, the solution to the RAM crunch is “everyone has their own home chip fab and deals with the raw material supply and hazardous waste disposal”?
I personally think of this like sorting algorithms. Quick sort does the same thing bubble sort does so why do we need quick sort? Pushing for efficiency drives innovation. It does this for many reasons but a big one is that putting a cap on a resource forces you to consider the others available and often you find that all it took was a little effort and suddenly the alternate path that looked a little worse is actually better than you realized.
This has a lot to do with how MCTS works BTW. The current best path is often only the current best path because a lot of investment has been sunk into it. If you were to put equal resources into a different path you may find that it was actually far better. It is just that the early rollouts favored the other 'best path' so you sunk a lot of resources into that one. We are very early in our exploration of LLM architecture. I highly doubt we are anywhere near the best path right now.
The diffusion models are interesting, but those also seem hacky.
I think the next form of AIs will be simpler and more abstract.
The building blocks of our brain don't have the notion of a "token" embed into them, it's lower level that that.
I think first step is to find a better way to represent information.
LLMs shouldn't "compute" stuff using language tokens, but some other, more efficient logical mechanisms. LLMs should first "feel" the solution, reason internally in that optimised space, then, only when interacting with a human should it convert all that into actual tokens/language.
The problem with that is we don't have any kind of training data in that abstract sense, maybe we could use RL to figure that out but current RL techniques are too slow and prone to breakage that anybody trying to use them to train a big enough general model (LLM, diffusion, world model, etc) will either fail or have to make a very very big investment.
The other option is maybe hook up humans to EEG or the likes and map their brains while they solve different kinds of problem, or just see and feel the world around them
I fear the future of local models will be controlled by governments. I feel like some time soon there's going to be a crackdown on what is available to download, what is hostable, and what is "acceptable". I partially suspect it has something to do with why 128 GB seems to be the most you can currently purchase for a single machine, despite the price.
I'm having an extremely hard time thinking of companies that have prospered due to software optimization. Most of them were swept away by hardware advances, instead.
The 1980's US car industry comes to mind. Nearly wiped out because they refused to make efficient vehicles. SpaceX is arguably showing how a rethink towards efficient can take over an entire industry. I am sure there are strong examples in software as well but they aren't coming to mind.
I think when successful, optimization really just means 'finally built right' and people forget the ridiculously inefficient ways before.
I would argue about 30% of Apple's success was just from not being slow annoying bloatware. I could never stomach it, but I know a lot of people who basically rage-quit Windows for the Apple ecosystem. I did the same, but for Linux.
Many of the current internet titans started by making things more efficient and accessible. Google for search, Facebook for connecting to people online, Microsoft for working with PCs at a reasonable price, Amazon for buying online as well as AWS. There are examples in other industries as well, Toyota is famous for it for example. There are probably counter examples but efficiency gains can be a huge deciding factor making companies successful.
I think finding significant efficiency gains with LLMs and the like may lead to qualitatively better products. Looking at people's experiences to DSV4F makes me believe that even more than before too.
I don't think people are realizing that speed can allow for categorically different user experiences that are more than just "worse than frontier capabilities but faster".
The headroom for hardware advances is a lot lower now than it has been for most of the industry's existence, when Moore's law held strong. Now we find ourselves limited by cost, physics, fab capacity, and complexity of spinning up more fab capacity.
I think the path of least resistance will end up being the cheapest and that is scaling up the parameters a ridiculous amount until you get highly capable models that can develop/distill/design the RAM efficient models. Going straight for low param is foolish and just a cope by smaller labs because they don't have the compute/talent to train the large ones.
This is 100% true for pretrains, likely true for RL as well although maybe there is some benefit to smaller activated params there. There is of course 0 benefit to small dense models relative to large sparse ones that are equally as memory efficient if you have enough computers.
Let's not forget the Bitter Lesson. Small models sound really nice but at some point you're just fighting the laws of information theory. Efficiency gains on the small model side are nice, but efficiency gains + giant model tends to be even better...
More compute/larger datasets during training != larger models. The Bitter Lesson was that just scaling things up beats custom hand-crafted optimizations. Up until 2024, we thought that meant scaling up the parameter count, but then that started to plateau. After o1 was released, we thought it was about scaling up test-time compute. Now, seeing how 30b models can easily outperform 200b models from a few years ago, it seems like what we need to scale is RL, at least for agentic capabilities. It looks like 30b is already enough for good agentic capabilities. Larger models aren't considerably "smarter" (especially since they're mostly MoE anyway, with something like the same 30-50b active parameter range), they just know more (better world knowledge), which lets them make more informed decisions. Maybe we just need to scale up the retrieval layer.
I'm just saying that any larger model with all of our optimizations of today will always beat a smaller model with the same optimizations. Until the smaller models + optimizations are at AGI levels I don't think anyone will really care about what smaller models can do other than for routing / toolcalling
I feel like the industry has quietly moved past the Bitter Lesson. In 2023 the story was naive parameter/data scaling and “emergent” intelligence properties. But there wasn’t enough data or compute to keep pushing in that direction, and the gains from it have been sublinear anyway.
Now, the labs spend enormous effort curating data pipelines to fit the models to a large assortment of very specific tools, tasks, harnesses, domains, etc. They also kind of fit to benchmarks by creating loads of synthetic training data that resembles benchmark tasks. None of this feels like the “scale up primitive methods and turn off your brain” message Sutton originally delivered.
You're totally right that we're far away from brain-level efficiency, but I'm just saying any efficiency gains we make towards small models will likely be felt on large ones as well, and we'll all move the goalposts to what the frontier can do. Hypothetically, getting GPT 5.6 performance in a 30B model would be amazing, but just imagine what you could do with a GPT 5.6-sized model at that point.
I honestly hope to see this across all applications, games, services, operating systems, etc. We've been in a period of wasteful RAM usage for over a decade. Constraints, whatever their origin, can be a good thing.
If China makes half decent RAM I would bet more on things like 128GM of RAM being the default on low spec laptops 10 years from now.
While I do love optimized software, the hardware side, especially for PCs, has been stagnating for way too long. At least now we have a valid use case for doubling available RAM every 2-3 years again.
I had a reasonably beefy Lenovo consumer line laptop that I bought in 2011, 8GBs of RAM. Its screen hinge broke and I couldn't repair it but I'm fairly sure it was otherwise still usable in 2023-24, once the HDD was replaced with an SSD. I think even now entry level laptops are sold with 8GB of RAM.
By comparison a PC from 2000 was utterly unusable in 2012-13.
I count on a 128GB baseline in 10 years. Beyond the current atmosphere of despair, I really want to see what Apple especially is cooking. Local AI is right up their alley and the current scarcity is unacceptable for them in so many ways.
I got the feeling laptops gonna feel very different in 2036.
Given local AI requirements, I expect minimum configurations for higher end machines to accelerate quickly. Yes, an entry level MacBook Air might be at 32 GB, but the entry MacBook Pro should quickly have 128 GB as a baseline. All this assumes we can actually make all the RAM we need. Until then, we’re going to be artificially capped.
Really? I feel like because nobody has RAM they're being pushed to the cloud frontier models. If we could all have our own 64GB+ VRAM GPUs, I feel like the open weight model scene would be even stronger.
> NeMo Switchyard, an open source library for smart routing
> When deployed, NeMo Switchyard can intelligently direct each request to the most capable and suitable model for the job
How do routers like this handle prompt caching when you send the second request?
Sticky models per session? but then the second message of that session won't be sent to a suitable model, and will only be sent to the same model as previous one.
I've seen ones that are configurable to pick a trade off point between lower cost (cache stickiness) and routing performance (best model for that turn).
But yeah I'm skeptical all this overhead is worth it.
Looks like your great question doesn’t have an answer, but looking at the routing strategies things get even more confused, since the proposed ones tend to rely on extra llm calls to determine which model to pick.
The nice thing is that it makes sense for specific setups, less conversation oriented.
As an example, you need to classify batches of data, and have many fine tuned models. Or you need to do speed to text and need to pick which whisper to use.
You can write your own strategy, in that case an harness with subagents would be able to leverage this, picking the right model and then keeping its session sticky, but overall the lack of concern for caching points towards use cases where you do not gain much from it.
The repo is probably a better entrypoint to it, bit more concise description than the press releases: https://github.com/NVIDIA-NeMo/Switchyard (Notably: "Experimental software. Not for production use."). Unclear if they actually want you to deploy it or not, press release says yes, README says no, do with that what you will.
Doesn't seem to mention "cache" in the README nor the docs, but the code has mentions of it (https://github.com/search?q=repo%3ANVIDIA-NeMo%2FSwitchyard+...), I'm not sure what their thinking is there. "Good luck" essentially? Seems to be per-provider at best, but weird position for a routing library to take.
To keep it simple, forget about routers and imagine you're in Cursor using GPT for a while, reaching a cache of says 200k.
You decide to switch to DeepSeek in the same session via the model picker, and continue as usual. What happens is that the cache for DeepSeek is created with the 200k + the incremental message. After this, cache can be kept warm for both models; two instances of the cache exists, one for GPT and one for DS.
You switch back to GPT. The whole session is sent to the model with the 200k original from GPT and the incremental messages you sent to DS. The 200k is read from cache and the incrementals are new, and then added to the cache.
Let's say every second message you switch between GPT and DS; cache was 200k and each incremental message is 1k. If you kept going with only GPT, cache hit rate would be 200k/(200k+1k) = 99.5%. When you switch between two models with warm cache, hit rate instead becomes 200k/(200k+2k) = 99%.
Model routers work the same way. Keep the cache warm, replicate it in two places. For this reason, when you set up your model pool for routing, you want to keep the model pool small and differentiated.
To elaborate, because I don't think some of the people reading this understand the reason, typically a lot or most of the cost in "agentic" API usage is cached read + generation. Cached read costs scale with turn count, which multi-model switching doesn't increase, and of course generation gets cheaper if you do some of it with a cheaper model. When you switch models the "catching up" batch of messages is just a single prefill and then that goes into cache. You don't even need to have the same chat history across models so long as the view from each model's perspective looks like a series of appends.
The main problem with model routing in my experience is that to work well the router needs to be pretty strong, maybe even moreso than any of the actual models in service. There are probably clever solutions to this but I haven't seen any that look better than just using sub-agents.
Given model A with cache C(a) and model B with C(b)
Isn't this not true because the moment you switch models from A to B, you need to provide C(b) the latest conversation diff since C(b) last updated, say many turns ago?
If you take 10 turns with a model A, it has to read the cache 10 times and write a lot of tokens (the expensive part). Switching to model B is just prefilling the diff + your new message, which is still just one turn. So total number of turns does not increase for a long session even with many switches.
I didn't understand this before reading the sibling comments so I'm not sure I got it fully right but I think the total cost becomes like this:
* Input tokens: Pay for both models
* Output tokens: Pay for the model that generates
* Cached tokens: Pay per turn, so in total a weighted average over both models?
Since output tokens are the most expensive, I can see how this is an overall win for many use cases as benchmarks also show. The hard part is routing correctly.
See sibling answer but essentially the effectiveness of cache is not diminished by having a separate one per model (relative to the win of doing more turns and generation with a cheaper model).
Caching should be possible as long as all the models use the same shared cache. The models don't even need to be running on the same server if the shared cache is distributed.
I have a feeling people reading this are thinking that a model router would be used to route between different providers. And in that case, a shared cache would be impossible, although some caching would still be effective. I think, ideally, a router like this is in front of a set of models hosted in one place.
Prompt caching isn’t about caching the literal text of the prompt. It’s about caching the result of running prefill on the prompt (or, equivalently, the result of generating the prompt one token at a time by autoregressive inference, or some combination of the above in the case of speculative decoding). This is often called the “KV” cache, and it is very model-specific.
They conveniently decided not to include the Qwen range of models in the Artificial Analysis graph, except the out-of-league Max variant. At least be brave and honest.
There's a famous Pascal quote, "I have only made this letter longer because I have not had the time to make it shorter."
Communicating an idea concisely is difficult. Most people struggle to get ideas across at all, asking them to do it well with fewer words is often out of reach.
I love the wave of new small model releases. Pleasantly surprising that an NVIDIA model runs so well on Apple Silicon using MLX! I was using nemotron-3.5-lightning:30b-mlx with OpenCode on my old (cheap) Mac this morning and no bad experiences except for running slowly.
big week for open models... seems like companies are noticing the 26-35b sweet spot... though I think a 12b-a1b-MoE model would be helpful for the 16gb folks
Not by the looks of it, but it got me thinking, currently in the middle of Level1Techs coverage on the model and switchyard and he mentions "how easy it is to customize it". Fully admitting that I haven't yet read the docs, my issue with that is "we can train LORAs for 35B as well, why use this (according to benchmarks) worse model for customization instead of a slightly bigger better one?"
Assuming I eat my words after going through the docs and this is actually a more efficient model / loras adapt better, I don't see as much value in it as is, as a REAP of it (remove least-important experts, domain-locked tests show ~98% retained accuracy) to something like 20B-A3B (rouhgly matching gpt oss, which while a good model, is outdated knowledge-wise and not as good with tool in my xp).
Having a 20B-A3B model at q4 that has a lora to be your local orchestrator (delegating coding to server/cloud models) and ci/cd runner does start sounding like an appealing proposition to me, as that would fit in 16gb vram easily (fitting many consumer gpus and 24gb macs).
The suite is across many categories, not only coding, and most of the tasks are low-horizon (or what the opposite of long-horizon is), where the max thinking time is around 10 minutes.
Gemini models are really smart, unfortunately they don't play well with any harness, so hard to use in practice.
But try them out for one-shot tasks, they are really good. Don't use them for coding in a harness, but you can ask them to generate code/planning (still, for coding only other models are indeed recommended).
True, but the ad is to my own product, there are no advertisers, maybe there will be at some point, but if they were, that would barely cover the costs of testing the models, and likely never get a ROI on the hundreds of hours I've spent building it.
Would I want it to grow and make money at some point? Sure, why not, then I can test even more models at higher reasoning levels. Meanwhile it's just me testing models when they come out and publishing the results for anyone who finds them useful.
I don't see why posting some info and a link with my own findings, in a relevant discussion is considered spam. Should it be?
loading the model would be similar vram footprint, correct, however the size of KV is based on 'active' params, not total params. So while at 1k ctx both will be in the same ballpark vram footprint-wise, at 100k the story will be very different. 27B at q4 kv for 256k takes ~8gb, while 35B at q4 kv around ~3.5gb, so at full precision kv those would be ~32gb and ~14gb (all ballparks, if you want exact numbers, its not hard to test).
As for the "cost", here i think the interesting arguments are around speed vs accuracy/"getting the job done", not literal $ cost per token.
Speed is (for the most part) active-parameter based, so a 30B-A3B model is roughly 10x the speed of a dense 30B (realistically closer to 8x) in the case when both fit. That's the proposition of MoE and why everyone is trying to make massive models with very few active params, so that they are still fast while having access to a lot of knowledge (at the cost of reasoning, as reasoning ability 'for the most part' comes from active params).
You can test this by running this nemo or 35B on the 3090. I have and its very fun (but sadly a worse model than 27B, so I usually keep 27B on my 3090)
Ran a quick test so that we both have accurate numbers, without MTP* at 10k ctx 27B hovers around 42 ts in llama.cpp, 35B around 135 ts. So not the 8x I assumed, just over 3x, but thats still a big difference.
For the sake of testing I turned MTP off, as that heavily depends on what the generated text is (structured text like code is very often a lot more predictable, therefore bigger boosts) and the quality of the quantization, as drafters learn how to "mimic" the full precision generated tokens, so when you layer the fact that MTP is a 'guesser' of the main model's next token, and quantization affecting what exact token is generated, it'd make comparisons like this needlessly noisy.
Yeah, that was my experience too (2x or 3x is indeed considerably faster), but not workflow-changing faster at 45tps baseline, especially for asynchronous tasks (which is my goal with a local 3090, to just let it do things non-stop, without my intervention).
What was the result with MTP?
Isn't MTP "losless"? The result should still be relevant when averaged across a fee queriers across different domaine I guess.
MTP is lossless in the sense that running a model with and without (at temp=0, meaning no randomness) will produce identical results.
It's true that with enough samples across domains and runs with MTP it should even out around concrete numbers, but I don't have time currently for long tests. On a quick test (before I remembered MTP is on), 27B was around 60-70 ts and 35B around 180-200 ts, both going up and down but mostly in those ballparks, which is inline with the ~3x from not using MTP.
One somewhat related thing is that, without drafters (the models doing just generation) ts tends to slowly go down, while with drafters, as the model is "checking drafts" instead of pure generation, even when the avg ts has fallen to say 50 ts (27B, ~128k ctc for example), there still tend to be random spikes to up to 70 (mostly during coding), which is always funny to me but also makes models a bit more "resilient" to the slow degradation of ts.
So, on a crude calculation, for a 3090 with 936.2 GB/s, a model that has 20 GB of active params would run at 45tps and one with 3GB active params at 300tps?
In practice, I don't think I saw over 100tps on a 3090, for a local 20-30B model, be it MoE or not.
Coincidentally I've been playing with small (~30B) self-hostable models for coding tasks today -- specifically plugging them into Cloudflare OS (which I work on) and asking each to build a collaborative whiteboard.
I'm finding that the Mixture-of-Experts (MoE) models (Qwen 3.6-35B, and Nemotron 3.5 Lightning) are, well, terrible at this. They just couldn't get the job done at all. Went way off the rails. They are really fast though!
Whereas ~30B dense models (not MoE) are pretty decent. I tried Muse Glimmer, Gemma 4-31B, Qwen 3.6-27B, and Laguna XS[0]. They were all able to build a working collaborative whiteboard app, without any guidance (other than feeding back error logs to the model). I also asked each to then draw a monkey by calling the API of the whiteboard it has just built. Laguna drew random scribbles but the rest all managed to produce something monkey-like.
(Frontier models in comparison will write the app in one shot with no errors at all.)
Note that both Qwen 3.6 and Gemma 4 each have both MoE and dense variants. I find this very confusing, because e.g. ollama's model index typically only distinguishes variants by their size, but MoE vs. dense makes a huge difference in how they actually perform. IMO they should use a suffix, like Qwen 3.6-moe vs. Qwen 3.6-dense, or maybe Qwen 3.6-fast vs. Qwen 3.6-smart...
[0] EDIT: Turns out Laguna XS is MoE, I misunderstood. It performed similarly to the dense models. But maybe this explains why it couldn't write code and think about monkey shapes at the same time!
> ollama's model index typically only distinguishes variants by their size
Don't use ollama. The entire project is just a series of stupid decisions like this.
What's the sota? The classic vLLM/llama.cpp? LM Studio? Unsloth studio any good?
How they have any credibility after they told everyone they could run Deepseek R1 on their laptop (by giving the Qwen 7B distill the `deepseek` moniker)...
And last time I have checked, you still can't run rerankers with it, yet you can download the models. See issue 3368, 2years old now.
There is an old rule of thumb that says the quality of an MoE is equivalent to a dense model with the geometric mean of its total and active parameters. So, for example, the Qwen3.6 would be equivalent to a dense model with approximately sqrt(35×3) ≈ 10.25B parameters.
Both MoEs and dense models are always getting better, so I don't think this comparison is meaningful across generations. But still for a first approximation, this tends to hold (you wouldn't expect a lot from a 10B model in coding yet).
I've been pretty impressed with Laguna. I downloaded their coding agent and have used it for a task here and there (the larger variant). Obviously it's nothing like a frontier LLM, but it surprised me with how good it was. And I think the model personality and way it talks is pretty pleasant
I’m confused about the naming suggestion. Seems like the AxB bit differentiates pretty clearly, no?
Only if you know what it means. To many people these suffixes are gibberish. Also ollama doesn't actually show that suffix in its model index. Don't say "don't use ollama", you sound like a Linux user telling people not to use Mac. (I say this as a Linux user who despises Apple products myself.)
Ollama uses "consumer UI guidelines", it's the second worst AI tool to use for anything more than writing fanfic.
Now I have to ask – which is the worst?
Good luck writing fanfics with primitive top-p and top-k samplers.
What’s good for data extraction? I have a hard time getting models to just pull names and titles from a blob of text.
Build/vibe-code your own benchmark and find out, the work pays off and lets you try many models.
Also try fine-tuning small models and see how they perform.
If you haven't yet, you might want to try gemma-4-E4B-it-qat or gemma-4-12B-it-qat with Structured Outputs. My main use case is tagging photos and generating headlines.
Laguna XS is MoE, however.
Oh!
I think I missed that and assumed it wasn't because it performed similarly to the dense models. Interesting!
As an aside, the bigger S 2.1 runs faster than the dense Qwen 3.6 and Gemma 4 models on the same hardware (assuming the same hardware is big enough to run it) and definitely feels smarter, and more capable of long tasks, but it doesn't feel as heavily tuned for code as Qwen 3.6.
One major consequence of the ramapocalypse, I think, is an even higher focus on small efficient models. I personally believe that the multi-trillion parameter models are fundamentally missing things and the push to smaller, more efficient will drive evolutionary structural changes that will lead to future gains
I'd assume the closed weight models are all working on shrinking their parameter counts anyways. They too benefit from smaller models. It'd be foolish for these SOTA labs to not be working at reducing parameter counts.
The incentive is there, but their money making niche is to solve problems you can't solve locally with a 30b, so they're unlikely to stray into territory owned by ultra cheap to run open weights models.
I would say even without rampocalypse there would still be the strong incentive to innovate at the edge and under more extreme constraints. The incentives are just even stronger now.
I'm looking forward to seeing what types of new things people create over the coming years once there is less obsession with massive unwieldy LLMs. I think the incentives are just too strong to ignore.
This is just temporary though, right?
With the benefit of LLMs already being proven, in a couple of years we will have vastly better hardware for inference I guess.
I feel like now hardware is stagnating a bit, because the software side has moved too fast for the hardware to catch up. Once we settle on some good, optimal software architecture for the models, dedicated hardware will easily increase throughout by 10x or 100x, for a fraction of the cost.
LLMs seems quite simple, maybe we'll be able to print/assemble at home our own chips with the desired models/weights.
Maybe we'll have model weights being shared like game cartridges.
> This is just temporary though, right?
Well, everything—even human life on Earth—is just temporary, but RAM supply lagging centralized-AI-driven demand increases continuing to squeeze the consumer market may not be a short term phenomenon.
> LLMs seems quite simple, maybe we'll be able to print/assemble at home our own chips with the desired models/weights.
So, the solution to the RAM crunch is “everyone has their own home chip fab and deals with the raw material supply and hazardous waste disposal”?
I...don't imagine so.
Hazardous waste?
My idea was more like you get some pre-made chips, that you can maybe assemble together configure at home with your desired models.
Maybe each one of this chip is a layer, so you can stack as many layers as you want.
I personally think of this like sorting algorithms. Quick sort does the same thing bubble sort does so why do we need quick sort? Pushing for efficiency drives innovation. It does this for many reasons but a big one is that putting a cap on a resource forces you to consider the others available and often you find that all it took was a little effort and suddenly the alternate path that looked a little worse is actually better than you realized.
This has a lot to do with how MCTS works BTW. The current best path is often only the current best path because a lot of investment has been sunk into it. If you were to put equal resources into a different path you may find that it was actually far better. It is just that the early rollouts favored the other 'best path' so you sunk a lot of resources into that one. We are very early in our exploration of LLM architecture. I highly doubt we are anywhere near the best path right now.
Definitely, LLMs are highly ineficient now.
The diffusion models are interesting, but those also seem hacky.
I think the next form of AIs will be simpler and more abstract.
The building blocks of our brain don't have the notion of a "token" embed into them, it's lower level that that.
I think first step is to find a better way to represent information.
LLMs shouldn't "compute" stuff using language tokens, but some other, more efficient logical mechanisms. LLMs should first "feel" the solution, reason internally in that optimised space, then, only when interacting with a human should it convert all that into actual tokens/language.
The problem with that is we don't have any kind of training data in that abstract sense, maybe we could use RL to figure that out but current RL techniques are too slow and prone to breakage that anybody trying to use them to train a big enough general model (LLM, diffusion, world model, etc) will either fail or have to make a very very big investment.
The other option is maybe hook up humans to EEG or the likes and map their brains while they solve different kinds of problem, or just see and feel the world around them
I fear the future of local models will be controlled by governments. I feel like some time soon there's going to be a crackdown on what is available to download, what is hostable, and what is "acceptable". I partially suspect it has something to do with why 128 GB seems to be the most you can currently purchase for a single machine, despite the price.
And we push ever closer to the edge of cyberpunk. Psst, interested in some black market heretical Mythos weights?
I am literally betting my company on this being true.
Me too. I think there are a few waves we can ride here. Let's collaborate?
What company? I am 100% focused on this as a concept in my own internal research.
It's a bad bet, historically.
I'm having an extremely hard time thinking of companies that have prospered due to software optimization. Most of them were swept away by hardware advances, instead.
The 1980's US car industry comes to mind. Nearly wiped out because they refused to make efficient vehicles. SpaceX is arguably showing how a rethink towards efficient can take over an entire industry. I am sure there are strong examples in software as well but they aren't coming to mind.
I think when successful, optimization really just means 'finally built right' and people forget the ridiculously inefficient ways before.
I would argue about 30% of Apple's success was just from not being slow annoying bloatware. I could never stomach it, but I know a lot of people who basically rage-quit Windows for the Apple ecosystem. I did the same, but for Linux.
Many of the current internet titans started by making things more efficient and accessible. Google for search, Facebook for connecting to people online, Microsoft for working with PCs at a reasonable price, Amazon for buying online as well as AWS. There are examples in other industries as well, Toyota is famous for it for example. There are probably counter examples but efficiency gains can be a huge deciding factor making companies successful.
You just listed a bunch of 0 to 1 companies, not 1 to 10 companies.
They weren’t quantitatively better than previous companies. They were qualitatively better.
I think finding significant efficiency gains with LLMs and the like may lead to qualitatively better products. Looking at people's experiences to DSV4F makes me believe that even more than before too.
I don't think people are realizing that speed can allow for categorically different user experiences that are more than just "worse than frontier capabilities but faster".
The headroom for hardware advances is a lot lower now than it has been for most of the industry's existence, when Moore's law held strong. Now we find ourselves limited by cost, physics, fab capacity, and complexity of spinning up more fab capacity.
Betting on innovation continuing to figure out ways to squeeze more out of less has historically been the right move. Look at Apple.
And I'd argue "hardware advances" are more proof of optimization.
IBM wants a word
On what? OS/2 didn't fail due to lacking software optimization.
I hope ssd streaming gets more popular, maybe more breakthroughs like that will help change things
I think the path of least resistance will end up being the cheapest and that is scaling up the parameters a ridiculous amount until you get highly capable models that can develop/distill/design the RAM efficient models. Going straight for low param is foolish and just a cope by smaller labs because they don't have the compute/talent to train the large ones.
This is 100% true for pretrains, likely true for RL as well although maybe there is some benefit to smaller activated params there. There is of course 0 benefit to small dense models relative to large sparse ones that are equally as memory efficient if you have enough computers.
Many on HN are in deep denial about this imo.
Right now there is a stark difference between what smaller models can do and bigger models can do.
Smaller models are suitable for simple tasks like classification / summarisation while larger models are better in agentic capabilities.
Depends on the type of agentic task though. For simple operations, a small model can be quite beneficial.
Let's not forget the Bitter Lesson. Small models sound really nice but at some point you're just fighting the laws of information theory. Efficiency gains on the small model side are nice, but efficiency gains + giant model tends to be even better...
More compute/larger datasets during training != larger models. The Bitter Lesson was that just scaling things up beats custom hand-crafted optimizations. Up until 2024, we thought that meant scaling up the parameter count, but then that started to plateau. After o1 was released, we thought it was about scaling up test-time compute. Now, seeing how 30b models can easily outperform 200b models from a few years ago, it seems like what we need to scale is RL, at least for agentic capabilities. It looks like 30b is already enough for good agentic capabilities. Larger models aren't considerably "smarter" (especially since they're mostly MoE anyway, with something like the same 30-50b active parameter range), they just know more (better world knowledge), which lets them make more informed decisions. Maybe we just need to scale up the retrieval layer.
I'm just saying that any larger model with all of our optimizations of today will always beat a smaller model with the same optimizations. Until the smaller models + optimizations are at AGI levels I don't think anyone will really care about what smaller models can do other than for routing / toolcalling
I feel like the industry has quietly moved past the Bitter Lesson. In 2023 the story was naive parameter/data scaling and “emergent” intelligence properties. But there wasn’t enough data or compute to keep pushing in that direction, and the gains from it have been sublinear anyway.
Now, the labs spend enormous effort curating data pipelines to fit the models to a large assortment of very specific tools, tasks, harnesses, domains, etc. They also kind of fit to benchmarks by creating loads of synthetic training data that resembles benchmark tasks. None of this feels like the “scale up primitive methods and turn off your brain” message Sutton originally delivered.
How many bits of information are in a real brain?
Don't know but a single neuron as a computation unit is far more complex than an ANN [1].
[1] https://christofkoch.com/biophysics-book/
You're totally right that we're far away from brain-level efficiency, but I'm just saying any efficiency gains we make towards small models will likely be felt on large ones as well, and we'll all move the goalposts to what the frontier can do. Hypothetically, getting GPT 5.6 performance in a 30B model would be amazing, but just imagine what you could do with a GPT 5.6-sized model at that point.
I honestly hope to see this across all applications, games, services, operating systems, etc. We've been in a period of wasteful RAM usage for over a decade. Constraints, whatever their origin, can be a good thing.
Same here, back to when algorithms and data structures mattered.
If China makes half decent RAM I would bet more on things like 128GM of RAM being the default on low spec laptops 10 years from now.
While I do love optimized software, the hardware side, especially for PCs, has been stagnating for way too long. At least now we have a valid use case for doubling available RAM every 2-3 years again.
I had a reasonably beefy Lenovo consumer line laptop that I bought in 2011, 8GBs of RAM. Its screen hinge broke and I couldn't repair it but I'm fairly sure it was otherwise still usable in 2023-24, once the HDD was replaced with an SSD. I think even now entry level laptops are sold with 8GB of RAM.
By comparison a PC from 2000 was utterly unusable in 2012-13.
I count on a 128GB baseline in 10 years. Beyond the current atmosphere of despair, I really want to see what Apple especially is cooking. Local AI is right up their alley and the current scarcity is unacceptable for them in so many ways.
I got the feeling laptops gonna feel very different in 2036.
Given local AI requirements, I expect minimum configurations for higher end machines to accelerate quickly. Yes, an entry level MacBook Air might be at 32 GB, but the entry MacBook Pro should quickly have 128 GB as a baseline. All this assumes we can actually make all the RAM we need. Until then, we’re going to be artificially capped.
Really? I feel like because nobody has RAM they're being pushed to the cloud frontier models. If we could all have our own 64GB+ VRAM GPUs, I feel like the open weight model scene would be even stronger.
> NeMo Switchyard, an open source library for smart routing
> When deployed, NeMo Switchyard can intelligently direct each request to the most capable and suitable model for the job
How do routers like this handle prompt caching when you send the second request?
Sticky models per session? but then the second message of that session won't be sent to a suitable model, and will only be sent to the same model as previous one.
I've seen ones that are configurable to pick a trade off point between lower cost (cache stickiness) and routing performance (best model for that turn).
But yeah I'm skeptical all this overhead is worth it.
https://github.com/NVIDIA-NeMo/Switchyard#routing-strategies
Looks like your great question doesn’t have an answer, but looking at the routing strategies things get even more confused, since the proposed ones tend to rely on extra llm calls to determine which model to pick.
The nice thing is that it makes sense for specific setups, less conversation oriented.
As an example, you need to classify batches of data, and have many fine tuned models. Or you need to do speed to text and need to pick which whisper to use.
You can write your own strategy, in that case an harness with subagents would be able to leverage this, picking the right model and then keeping its session sticky, but overall the lack of concern for caching points towards use cases where you do not gain much from it.
The repo is probably a better entrypoint to it, bit more concise description than the press releases: https://github.com/NVIDIA-NeMo/Switchyard (Notably: "Experimental software. Not for production use."). Unclear if they actually want you to deploy it or not, press release says yes, README says no, do with that what you will.
Doesn't seem to mention "cache" in the README nor the docs, but the code has mentions of it (https://github.com/search?q=repo%3ANVIDIA-NeMo%2FSwitchyard+...), I'm not sure what their thinking is there. "Good luck" essentially? Seems to be per-provider at best, but weird position for a routing library to take.
I personally think it's snake-oil marketing with all these smart-model-routing products/projects
prompt-cache won't work with these
To keep it simple, forget about routers and imagine you're in Cursor using GPT for a while, reaching a cache of says 200k.
You decide to switch to DeepSeek in the same session via the model picker, and continue as usual. What happens is that the cache for DeepSeek is created with the 200k + the incremental message. After this, cache can be kept warm for both models; two instances of the cache exists, one for GPT and one for DS.
You switch back to GPT. The whole session is sent to the model with the 200k original from GPT and the incremental messages you sent to DS. The 200k is read from cache and the incrementals are new, and then added to the cache.
Let's say every second message you switch between GPT and DS; cache was 200k and each incremental message is 1k. If you kept going with only GPT, cache hit rate would be 200k/(200k+1k) = 99.5%. When you switch between two models with warm cache, hit rate instead becomes 200k/(200k+2k) = 99%.
Model routers work the same way. Keep the cache warm, replicate it in two places. For this reason, when you set up your model pool for routing, you want to keep the model pool small and differentiated.
First principles of model routing: https://try.works/first-principles-of-model-routing
role-model router and protocol: https://github.com/try-works/role-model
note: edited to keep the answer to the below message clearer
To elaborate, because I don't think some of the people reading this understand the reason, typically a lot or most of the cost in "agentic" API usage is cached read + generation. Cached read costs scale with turn count, which multi-model switching doesn't increase, and of course generation gets cheaper if you do some of it with a cheaper model. When you switch models the "catching up" batch of messages is just a single prefill and then that goes into cache. You don't even need to have the same chat history across models so long as the view from each model's perspective looks like a series of appends.
The main problem with model routing in my experience is that to work well the router needs to be pretty strong, maybe even moreso than any of the actual models in service. There are probably clever solutions to this but I haven't seen any that look better than just using sub-agents.
> which multi-model switching doesn't increase
Given model A with cache C(a) and model B with C(b)
Isn't this not true because the moment you switch models from A to B, you need to provide C(b) the latest conversation diff since C(b) last updated, say many turns ago?
If you take 10 turns with a model A, it has to read the cache 10 times and write a lot of tokens (the expensive part). Switching to model B is just prefilling the diff + your new message, which is still just one turn. So total number of turns does not increase for a long session even with many switches.
I didn't understand this before reading the sibling comments so I'm not sure I got it fully right but I think the total cost becomes like this:
* Input tokens: Pay for both models * Output tokens: Pay for the model that generates * Cached tokens: Pay per turn, so in total a weighted average over both models?
Since output tokens are the most expensive, I can see how this is an overall win for many use cases as benchmarks also show. The hard part is routing correctly.
Can you explain how does it work? like how is the previous K/V cache used when you switch to another model?
Source?
See sibling answer but essentially the effectiveness of cache is not diminished by having a separate one per model (relative to the win of doing more turns and generation with a cheaper model).
edit: updated the answer above to be more qualitative instead
Caching should be possible as long as all the models use the same shared cache. The models don't even need to be running on the same server if the shared cache is distributed.
I have a feeling people reading this are thinking that a model router would be used to route between different providers. And in that case, a shared cache would be impossible, although some caching would still be effective. I think, ideally, a router like this is in front of a set of models hosted in one place.
How do caches work across models? I would have thought that was very model specific - if not I’ve really misunderstood what’s getting cached.
I am not sure the author of the comment you are replying to understands that LLM systems have prompt caches
Huh?
Prompt caching isn’t about caching the literal text of the prompt. It’s about caching the result of running prefill on the prompt (or, equivalently, the result of generating the prompt one token at a time by autoregressive inference, or some combination of the above in the case of speculative decoding). This is often called the “KV” cache, and it is very model-specific.
They conveniently decided not to include the Qwen range of models in the Artificial Analysis graph, except the out-of-league Max variant. At least be brave and honest.
I would like to propose something:
- problem: massive deluge of information because of AI
- solution: human beings should adopt a minimalist style of communicating in writing.
- e.g. this entire website page can be ten bullet points.
There's a famous Pascal quote, "I have only made this letter longer because I have not had the time to make it shorter."
Communicating an idea concisely is difficult. Most people struggle to get ideas across at all, asking them to do it well with fewer words is often out of reach.
This resonates well with engineers: making something complex is easy, making them simpler is harder
Why use more words when few do trick?
nods
k
While I agree with the spirit, I don't think the solution to bad prose is slightly less bad prose. We can write good prose instead.
In my opinion: the only way forward is zero-knowledge-proof authenticated social media.
We can’t have legitimate debate if we have to assume a few bad actors are cloning their voice by the thousands, poisoning debate.
If we can pin one account to a real person, we won’t get rid of LLM-content and misinformation, but at least we can hold them accountable.
Is the network based on trust and peer to peer confirmation of private keys from who you know in real life that you validated isn’t a robot ?
Or do you have something else in mind ?
> We can’t have legitimate debate
I'm not sure people really want that
it takes more effort to write with conciseness
Judging by how my nieces and nephews text this has already happened!
hell yeah
24 comments so far about Nemotron on this earlier submission: https://news.ycombinator.com/item?id=49257947
I love the wave of new small model releases. Pleasantly surprising that an NVIDIA model runs so well on Apple Silicon using MLX! I was using nemotron-3.5-lightning:30b-mlx with OpenCode on my old (cheap) Mac this morning and no bad experiences except for running slowly.
big week for open models... seems like companies are noticing the 26-35b sweet spot... though I think a 12b-a1b-MoE model would be helpful for the 16gb folks
An open source model from Nvidia is a free drug to later buy their chips.
Nemotron 3.5 Lightning runs on how little GPU vram? Can q4 run on 16gb?
Not by the looks of it, but it got me thinking, currently in the middle of Level1Techs coverage on the model and switchyard and he mentions "how easy it is to customize it". Fully admitting that I haven't yet read the docs, my issue with that is "we can train LORAs for 35B as well, why use this (according to benchmarks) worse model for customization instead of a slightly bigger better one?"
Assuming I eat my words after going through the docs and this is actually a more efficient model / loras adapt better, I don't see as much value in it as is, as a REAP of it (remove least-important experts, domain-locked tests show ~98% retained accuracy) to something like 20B-A3B (rouhgly matching gpt oss, which while a good model, is outdated knowledge-wise and not as good with tool in my xp).
Having a 20B-A3B model at q4 that has a lora to be your local orchestrator (delegating coding to server/cloud models) and ci/cd runner does start sounding like an appealing proposition to me, as that would fit in 16gb vram easily (fitting many consumer gpus and 24gb macs).
Good luck getting it to run with NVFP4 on a DGX Spark, the very architecture Nvidia created that format for.
Can you elaborate on the problem? The article claims support for dgx spark
The new Meta 30B models seems A LOT better:
https://aibenchy.com/compare/meta-muse-glimmer-30b-xhigh/nvi...
Muse Glimmer 30B seems to be on par with Qwen 3.6 27B (4 months old)
but
Qwen 3.8 27B is dropping this week...
Yes, I was surprised to see doing it as well as Qwen 3.7 27b.
Even though that model is already "old", qwen was way ahead everyone else in that size category before this Meta model.
Also, probably for non-Chinese usage, using a non-Chinese model might lead to better results.
The top 4 models on that site are all variants of Gemini Flash? That does not match my experience at all.
I should add a F.a.q. for this question.
The suite is across many categories, not only coding, and most of the tasks are low-horizon (or what the opposite of long-horizon is), where the max thinking time is around 10 minutes.
Gemini models are really smart, unfortunately they don't play well with any harness, so hard to use in practice.
But try them out for one-shot tasks, they are really good. Don't use them for coding in a harness, but you can ask them to generate code/planning (still, for coding only other models are indeed recommended).
yep, the person you're responding to created the benchmark and is using HN comments as advertisement.
I did, avertisment is a big word, as I gain nothing from the traffic, I run the website for myself, and some other people find it useful too.
Happy to hear what would make the website more useful.
You are displaying ads on your website, there's a clear financial incentive.
True, but the ad is to my own product, there are no advertisers, maybe there will be at some point, but if they were, that would barely cover the costs of testing the models, and likely never get a ROI on the hundreds of hours I've spent building it.
Would I want it to grow and make money at some point? Sure, why not, then I can test even more models at higher reasoning levels. Meanwhile it's just me testing models when they come out and publishing the results for anyone who finds them useful.
I don't see why posting some info and a link with my own findings, in a relevant discussion is considered spam. Should it be?
I believe it is pretty reasonable to show ads about your own product on your website, and shouldn't be considered spam.
At what cost difference?
I don't think it matters, if it's for local/on-device usage.
The cost is similar vram footprint I guess (?)
loading the model would be similar vram footprint, correct, however the size of KV is based on 'active' params, not total params. So while at 1k ctx both will be in the same ballpark vram footprint-wise, at 100k the story will be very different. 27B at q4 kv for 256k takes ~8gb, while 35B at q4 kv around ~3.5gb, so at full precision kv those would be ~32gb and ~14gb (all ballparks, if you want exact numbers, its not hard to test).
As for the "cost", here i think the interesting arguments are around speed vs accuracy/"getting the job done", not literal $ cost per token.
I am asking mostly for running on a 3090.
I think the tps difference between them (both fitting in vram) won't be more than 2x in practice.
I would happily take 20tps over 40tps, if the model gets 3x more correct answers.
Speed is (for the most part) active-parameter based, so a 30B-A3B model is roughly 10x the speed of a dense 30B (realistically closer to 8x) in the case when both fit. That's the proposition of MoE and why everyone is trying to make massive models with very few active params, so that they are still fast while having access to a lot of knowledge (at the cost of reasoning, as reasoning ability 'for the most part' comes from active params).
You can test this by running this nemo or 35B on the 3090. I have and its very fun (but sadly a worse model than 27B, so I usually keep 27B on my 3090)
I remember running both qwen 30b-a3b and 27b on my 3090, and on the initial test, the 27b was only like 2x slower.
Ran a quick test so that we both have accurate numbers, without MTP* at 10k ctx 27B hovers around 42 ts in llama.cpp, 35B around 135 ts. So not the 8x I assumed, just over 3x, but thats still a big difference.
For the sake of testing I turned MTP off, as that heavily depends on what the generated text is (structured text like code is very often a lot more predictable, therefore bigger boosts) and the quality of the quantization, as drafters learn how to "mimic" the full precision generated tokens, so when you layer the fact that MTP is a 'guesser' of the main model's next token, and quantization affecting what exact token is generated, it'd make comparisons like this needlessly noisy.
Thanks for sharing.
Yeah, that was my experience too (2x or 3x is indeed considerably faster), but not workflow-changing faster at 45tps baseline, especially for asynchronous tasks (which is my goal with a local 3090, to just let it do things non-stop, without my intervention).
What was the result with MTP?
Isn't MTP "losless"? The result should still be relevant when averaged across a fee queriers across different domaine I guess.
MTP is lossless in the sense that running a model with and without (at temp=0, meaning no randomness) will produce identical results. It's true that with enough samples across domains and runs with MTP it should even out around concrete numbers, but I don't have time currently for long tests. On a quick test (before I remembered MTP is on), 27B was around 60-70 ts and 35B around 180-200 ts, both going up and down but mostly in those ballparks, which is inline with the ~3x from not using MTP.
One somewhat related thing is that, without drafters (the models doing just generation) ts tends to slowly go down, while with drafters, as the model is "checking drafts" instead of pure generation, even when the avg ts has fallen to say 50 ts (27B, ~128k ctc for example), there still tend to be random spikes to up to 70 (mostly during coding), which is always funny to me but also makes models a bit more "resilient" to the slow degradation of ts.
lightning is sparse, glimmer is dense
Oh, good to know, I just quickly tested and published the results.
I will add model sizes (total/active params) for each model, good point.
and Glimmer has 10x active params of Lightning. Meaning ~ 10 slower on same HW
Is that the case?
If the entire model fits in vram, won't the tps be comparable?
autoregressive generation is memory bandwidth bounded
So, on a crude calculation, for a 3090 with 936.2 GB/s, a model that has 20 GB of active params would run at 45tps and one with 3GB active params at 300tps?
In practice, I don't think I saw over 100tps on a 3090, for a local 20-30B model, be it MoE or not.