I wish that "small" LLMs would stop being confidently very incorrect. Admittedly this is a bit of an intentionally esoteric test, but the confident way in which it presents a totally incorrect answer is a bit concerning.
"please write 250 words on the etymology and history of the word schlong"
The actual origin of the word is from middle high German and Yiddish-speaking Ashkenazi Jewish communities.
For comparison qwen 3.6 35B A3B does perfect on this and will give a solid description of the word's real origins and how it has made it into casual profanity/vulgarity as used in US English, and even mentions specific stand-up comedians and famous public figures of specific ethnic/religious origin in the US NE who introduced it into wider use.
Ask it for something that's not a narrow niche scientific or technical field, but something that would be less common to make it into a 20B size model, and see just how it does.
I think for smaller models, they need to be more defensive on unknown information and frontier model level tool calling capabilities.
LLMs are kind of a compact knowledge box of its training data and it's understandable it would not have information about every topic and in that case just do a web search or a proper tool invocation to get the data and then synthesize.
It's far more likely basic consumer devices will advance such that much larger models efficiently run than finding novel ways to compress all of human experience to fit on today's mobile hardware.
We’re already seeing incredible knowledge compression out of the models the GP mentioned like Qwen 35B-A3B, which feels well within the realm of “runs on a phone” in the next handful of years.
And by then we’ll probably have been further surprised by just how much information and capacity for reasoning can be crammed into a few gigs of weights. Models just keep getting better for a given size, it’ll be interesting to see where the limit of that is.
Would it not be better to ask models to search the topic on the Internet and then answer? I do not understand why we expect small LLMs to answer from own knowledge.
> Ask models to search on the Internet and answer?
To me, the benefit of running small models is that they fit on your device exactly that you don't need any internet connection. It's all local and offline, so you can still consult for information in any scenario.
The real target of small models is surely the opposite: knowing with clarity how to conversationally or otherwise reason about your requirements in natural language, sound and imagery, consult APIs that interact with apps, and then have a strong, generic capability to reason about what it finds.
If you want it to know more things locally, you could hopefully distribute wikipedia or other appropriate large sources on a flash storage, properly pre-indexed, with a local tool API, and let it work its magic on that. Not least because that is as I understand it, considerably easier to incrementally update.
This is, without doubt, a very difficult target, not least because it is rather at odds with the way LLMs work on a crude level.
I thought the benefit of small models is that they are a natural language UI to whatever they are connected to.
It seems strange to me to expect a small model to answer everything correctly when every device that they are being used on support networking and knowledge is constantly evolving.
I wonder if someone already made a somewhat turnkey setup like this? Small models should ideally not waste a lot of parameters on trying to memorize too much knowledge and then hallucinate / interpolate a lot of the gaps.
I don't, really, but 20B is also not that small... It's an intentionally weird question to see how confidently incorrect something will be. It certainly writes a plausible sounding explanation that could fool someone for whom English is their 2nd or 3rd language, or is not familiar with specific North American slang.
It's also something I've seen has great results with esoteric individual pieces of knowledge that works fine in a Q6 or Q8 quantized LLM but breaks down in a bad way at worse quantization.
I've also been asking LLMs to draw SVGs of literal pelican cases and the results can be more amusing than pelicans on bicycles. You can get pelican cases with cameras, firearms, long cases, square cases, cases that look nothing like pelicans (but more like Zero Halliburton aluminum briefcases), etc. You also get cases that are open or closed depending on the whims of the LLM.
Seems like less of a problem in smaller models where bullshit tends to become very obvious to anyone with half a clue about the given subject than it is in larger models where the illusion is complete enough that the confidently stated answers are very incorrect in more subtle ways.
Small or overly quantized LLMs are a genre of humor. Same goes for small image generators. Janky generative AI is like the Geocities web pages of today.
They are not incorrect. They don't even hold a position in the first place.
An LLM continues the prompt it is given. What is more likely to come after a question? An answer, not an "oh sorry I'm not sure". Sure, you could make the latter more likely, but then the model would be unusable. Larger models simply contain more answers, more ways to stumble into them, and a granular enough geography to stay on the trail.
The most interesting thing is the "dreaming" idea for on-device adaptation. I guess regularly adjusting weights becomes possible when the model is so small and the math is simplified.
I'm assuming that doesn't actually exist yet, though, as I don't see anything about an implementation in the code that's been released.
But, it's a really interesting idea for a personal model. There's a risk of more AI psychosis if these things actually start "learning", but the value of it is also probably pretty big. I'm not sure I buy it will actually be able to self-improve, though. The best models are helping improve themselves, but the best models are considerably smarter and more capable than this one. I've asked models like Gemma 4 31B to help figure out training and synthesizing data, and it mostly fails on anything more than categorization and summarization. This little model is much dumber than that.
So, I'm skeptical, but maybe there's deterministic tooling that can assist and maybe it will be scoped tightly enough to just learn and update facts and not so much try to retrain the whole thing.
For small models like this, it’s super important that it works well at tool calling etc. imho because it can’t memorize facts and isn’t big enough to tell when it doesn’t know. I could use it for high quality tool routing or a backup fast model for smaller task set. E.g. I use GPT-5.6 for voice channels at home. I’d prefer to be able to have this do basic tool calls and stuff because of the local speed.
Will give it a crack as a quick model in my clawlike.
This matches what I've seen shipping Apple's on-device model in a Mac app.
The model is reliable at the semantic half. Give it the OCR text of a receipt and it correctly identifies the vendor and the date. What it does not do reliably is follow mechanical instructions. A user asked for dates formatted as TT-MM-JJJJ and got files literally named TT-MM-JJJJ, because it reproduced the format string instead of filling it in. Another asked for uppercase, and the model acknowledged the request in its reasoning and returned lowercase.
The failures were not consistent, which is worse than failing every time. You cannot tell users "this doesn't work", only "this works most of the time", and nobody accepts that from something touching their files.
What fixed it was moving the mechanical part out of the model entirely. The model decides what the document is about; ordinary deterministic code decides how the name is written. Every time I moved that line back toward the model, quality dropped.
Which is a version of your point: with a small model the win isn't making it smarter, it's shrinking what you make it responsible for.
Beats how? In my experience 1-bit bonsai 27B is quite "dumb" when asked a question about a lot of things, as a canned repository of static knowledge from its training dataset. I mean, I literally asked it for a 250 word description of Seattle and it hallucinated a tallest building in the city with an observation deck that doesn't exist, and didn't mention the Space Needle.
Really basic stuff. But then again, the entire thing was running in <6GB of RAM.
But before anyone says 1-bit bonsai 27B beats anything, please actually run it and ask it some questions about topics you already know the answer to.
While Qwen 3.6 35B A3B in Q8 with full context capability (llama-server in no-mmap mode with 262k context will eat 47GB, so not comparable in size either) knows a great deal. The 35B-A3B can even translate multiple pages of English into Farsi and its Farsi output is not far off the quality of what Google Translate does.
Sure, then you should allow that they are also tools for transforming sequences of questions into answers. Language models are based on compression of information, using them as a knowledge base is entirely within capability.
> Language models are based on compression of information, using them as a knowledge base is entirely within capability
It's also the aspect of LLMs that degrades fastest with quantisation. You can't reasonably expect accurate knowledge of everything in the world in a few gigabytes.
Which means that small models need to be conditioned to rely more heavily on tools to fetch accurate information, and ideally not try and generate facts purely based on their (extremely lossy) internal knowledge
It's kinda like lossily compressing the entire content of a CD-DA disc to fit onto a 3.5" floppy disk and then complaining that it doesn't sound very nice.
The 27B model beats 35B A3B. The quantized 1-bit version obviously doesn't, but it would likely beat a 1-bit quantized A3B variant. Comparing 3.5 35B or 3.6 35B to this model (Maple) is not meaningful.
“Current approaches to low precision primarily focus on converting models trained in full precision to lower bitwidths. We view this as fundamentally the wrong approach…”
Very excited to see how it performs, I’ve been a bit skeptical of the efficacy of converting existing models - really cool to see one trained from scratch in the ternary format.
matrix multiplication can be effectively replaced with additions
, lowering the total arithmetic workload needed to infer through a model."
Interesting! I never knew that before... I will have to do some more research on this, but yes, it definitely sounds possible!
(Also (and this is just a wild guess/hunch here!)
in theory, matrix multiplication could be replaced with pre-computed cached lookups from a lookup table in memory
if, if and only if the resultant pre-computed lookup table of result matrices could fit into memory available!
In other words, think of a Matrix not as a 2D array, but as a 1D string. Think of the Matrix multiply operation as first appending the 1D string of the first matrix to the 1D string of the second matrix, then using that string as a Key to look-up a precomputed result in Memory...
Of course, the limitation to this approach is the availability of memory relative to the permutations of input values... It could work -- but only for smaller matrices... which leads to another strange idea... in that set of pre-computed matrix permutations, there might be duplicates... figure out a way to put duplicates in memory only once (could use pointers or some other scheme to do this) and you could theoretically store more matrices in the same memory and/or slightly larger matrices!
Anyway, just rambling out loud! :-)
Remember, "almost all programming can be viewed as an exercise in caching" to quote the legendary programmer Terje Mathisen! :-) )
Looks promising though much like the bonsai tenary one it hallucinates knowledge quite aggressively. The online chat having search tools covers this up somewhat, but it's still there.
...from very unscientific casual vibes it does seem pretty good though considering the speed
Would also be curious what their search tool backend looks like - that too is very fast for very rapid multiple searches
Makes me wonder how much we can 'get away with' in terms of raw model capability, when you have tool access for anything specialised or specific.
I probably could make up a bit of an incorrect carrot cake recipe if asked on the spot, but with a Google search I can give you something far more robust.
Maybe we don't need a 'country of geniuses' in our pocket, but more a helpful assistant that can reasonably reason!
I haven't compared it for perf, but at first glance this appears to be an LLM (text-only) while both binary and ternary bonsai 27B models are VLMs (where the vision tower and the adapter MLP weights are unquantized and kept in float16).
I wish that "small" LLMs would stop being confidently very incorrect. Admittedly this is a bit of an intentionally esoteric test, but the confident way in which it presents a totally incorrect answer is a bit concerning.
"please write 250 words on the etymology and history of the word schlong"
https://pastes.io/uhshFgn4
The actual origin of the word is from middle high German and Yiddish-speaking Ashkenazi Jewish communities.
For comparison qwen 3.6 35B A3B does perfect on this and will give a solid description of the word's real origins and how it has made it into casual profanity/vulgarity as used in US English, and even mentions specific stand-up comedians and famous public figures of specific ethnic/religious origin in the US NE who introduced it into wider use.
Ask it for something that's not a narrow niche scientific or technical field, but something that would be less common to make it into a 20B size model, and see just how it does.
chat test link: https://chat.deepgrove.ai/
I think for smaller models, they need to be more defensive on unknown information and frontier model level tool calling capabilities.
LLMs are kind of a compact knowledge box of its training data and it's understandable it would not have information about every topic and in that case just do a web search or a proper tool invocation to get the data and then synthesize.
You need a small LLM that can reason very well and use tools like web search very well.
No one is going to compress human knowledge into few bits.
Not at the moment, but who knows what types of models and storage types we would have in 100 years.
It's far more likely basic consumer devices will advance such that much larger models efficiently run than finding novel ways to compress all of human experience to fit on today's mobile hardware.
Information can only be compressed so much
We’re already seeing incredible knowledge compression out of the models the GP mentioned like Qwen 35B-A3B, which feels well within the realm of “runs on a phone” in the next handful of years.
And by then we’ll probably have been further surprised by just how much information and capacity for reasoning can be crammed into a few gigs of weights. Models just keep getting better for a given size, it’ll be interesting to see where the limit of that is.
Would it not be better to ask models to search the topic on the Internet and then answer? I do not understand why we expect small LLMs to answer from own knowledge.
> Ask models to search on the Internet and answer?
To me, the benefit of running small models is that they fit on your device exactly that you don't need any internet connection. It's all local and offline, so you can still consult for information in any scenario.
The real target of small models is surely the opposite: knowing with clarity how to conversationally or otherwise reason about your requirements in natural language, sound and imagery, consult APIs that interact with apps, and then have a strong, generic capability to reason about what it finds.
If you want it to know more things locally, you could hopefully distribute wikipedia or other appropriate large sources on a flash storage, properly pre-indexed, with a local tool API, and let it work its magic on that. Not least because that is as I understand it, considerably easier to incrementally update.
This is, without doubt, a very difficult target, not least because it is rather at odds with the way LLMs work on a crude level.
I thought the benefit of small models is that they are a natural language UI to whatever they are connected to.
It seems strange to me to expect a small model to answer everything correctly when every device that they are being used on support networking and knowledge is constantly evolving.
This is a very bad way of thinking of it. Small LLMs have clues about real knowledge but only surface level answers will be accurate.
True but they could query a local 50GB Wikipedia dump for factual info’s via embedding search. All still local.
I wonder if someone already made a somewhat turnkey setup like this? Small models should ideally not waste a lot of parameters on trying to memorize too much knowledge and then hallucinate / interpolate a lot of the gaps.
I don't, really, but 20B is also not that small... It's an intentionally weird question to see how confidently incorrect something will be. It certainly writes a plausible sounding explanation that could fool someone for whom English is their 2nd or 3rd language, or is not familiar with specific North American slang.
It's also something I've seen has great results with esoteric individual pieces of knowledge that works fine in a Q6 or Q8 quantized LLM but breaks down in a bad way at worse quantization.
Parameter count is not everything.
20b parameters * 1.5 bits per parameter is just 30 billion bits, about 3.75gb
a full 20b fp16 is about 40GB.
I find it weird how a smaller model still produces decent text, except it bullshits all the way.
Maybe we don’t necessarily expect them to answer from their own knowledge, but to either do that or say “I don’t know”.
English Wikipedia: "As of 16 October 2024, the size of the current version including all articles compressed is about 24.7 GB without media."
Models in this size range should aim for correct tool calling and avoiding hallucinations, not universal knowledge. (apparently they don't.)
The schlong test is nearly as funny as drawing shit on bicycles test
I've also been asking LLMs to draw SVGs of literal pelican cases and the results can be more amusing than pelicans on bicycles. You can get pelican cases with cameras, firearms, long cases, square cases, cases that look nothing like pelicans (but more like Zero Halliburton aluminum briefcases), etc. You also get cases that are open or closed depending on the whims of the LLM.
Seems like less of a problem in smaller models where bullshit tends to become very obvious to anyone with half a clue about the given subject than it is in larger models where the illusion is complete enough that the confidently stated answers are very incorrect in more subtle ways.
Small or overly quantized LLMs are a genre of humor. Same goes for small image generators. Janky generative AI is like the Geocities web pages of today.
lol I like the first one though. Reads like a great sarcasm response.
I wonder if kids will do this to their parents.
They are not incorrect. They don't even hold a position in the first place.
An LLM continues the prompt it is given. What is more likely to come after a question? An answer, not an "oh sorry I'm not sure". Sure, you could make the latter more likely, but then the model would be unusable. Larger models simply contain more answers, more ways to stumble into them, and a granular enough geography to stay on the trail.
The most interesting thing is the "dreaming" idea for on-device adaptation. I guess regularly adjusting weights becomes possible when the model is so small and the math is simplified.
I'm assuming that doesn't actually exist yet, though, as I don't see anything about an implementation in the code that's been released.
But, it's a really interesting idea for a personal model. There's a risk of more AI psychosis if these things actually start "learning", but the value of it is also probably pretty big. I'm not sure I buy it will actually be able to self-improve, though. The best models are helping improve themselves, but the best models are considerably smarter and more capable than this one. I've asked models like Gemma 4 31B to help figure out training and synthesizing data, and it mostly fails on anything more than categorization and summarization. This little model is much dumber than that.
So, I'm skeptical, but maybe there's deterministic tooling that can assist and maybe it will be scoped tightly enough to just learn and update facts and not so much try to retrain the whole thing.
For small models like this, it’s super important that it works well at tool calling etc. imho because it can’t memorize facts and isn’t big enough to tell when it doesn’t know. I could use it for high quality tool routing or a backup fast model for smaller task set. E.g. I use GPT-5.6 for voice channels at home. I’d prefer to be able to have this do basic tool calls and stuff because of the local speed.
Will give it a crack as a quick model in my clawlike.
This matches what I've seen shipping Apple's on-device model in a Mac app.
The model is reliable at the semantic half. Give it the OCR text of a receipt and it correctly identifies the vendor and the date. What it does not do reliably is follow mechanical instructions. A user asked for dates formatted as TT-MM-JJJJ and got files literally named TT-MM-JJJJ, because it reproduced the format string instead of filling it in. Another asked for uppercase, and the model acknowledged the request in its reasoning and returned lowercase.
The failures were not consistent, which is worse than failing every time. You cannot tell users "this doesn't work", only "this works most of the time", and nobody accepts that from something touching their files.
What fixed it was moving the mechanical part out of the model entirely. The model decides what the document is about; ordinary deterministic code decides how the name is written. Every time I moved that line back toward the model, quality dropped.
Which is a version of your point: with a small model the win isn't making it smarter, it's shrinking what you make it responsible for.
A benchmark table comparing to Qwen 3.5 35B-A3B seems strange when Qwen 3.6 35B-A3B has been out for some time and is significantly better.
I didn't notice the version difference when first reading the article! So this is a heads up to people like me.
Their main comparison is 1-bit Bonsai 27B (Qwen3.6 27B) which beats A3B anyway.
Beats how? In my experience 1-bit bonsai 27B is quite "dumb" when asked a question about a lot of things, as a canned repository of static knowledge from its training dataset. I mean, I literally asked it for a 250 word description of Seattle and it hallucinated a tallest building in the city with an observation deck that doesn't exist, and didn't mention the Space Needle.
Really basic stuff. But then again, the entire thing was running in <6GB of RAM.
But before anyone says 1-bit bonsai 27B beats anything, please actually run it and ask it some questions about topics you already know the answer to.
While Qwen 3.6 35B A3B in Q8 with full context capability (llama-server in no-mmap mode with 262k context will eat 47GB, so not comparable in size either) knows a great deal. The 35B-A3B can even translate multiple pages of English into Farsi and its Farsi output is not far off the quality of what Google Translate does.
I haven't tested something as badly quantized as 35B A3B Q2 which is somewhere around 12GB on disk. https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF
these tools are for manipulating and retrieving and transforming sequences of context. using it as a knowledgebase is just expecting the wrong thing.
Sure, then you should allow that they are also tools for transforming sequences of questions into answers. Language models are based on compression of information, using them as a knowledge base is entirely within capability.
> Language models are based on compression of information, using them as a knowledge base is entirely within capability
It's also the aspect of LLMs that degrades fastest with quantisation. You can't reasonably expect accurate knowledge of everything in the world in a few gigabytes.
Which means that small models need to be conditioned to rely more heavily on tools to fetch accurate information, and ideally not try and generate facts purely based on their (extremely lossy) internal knowledge
It's kinda like lossily compressing the entire content of a CD-DA disc to fit onto a 3.5" floppy disk and then complaining that it doesn't sound very nice.
I also find that the 35B A3B in Q4 outdoes Ternary Bonsai 27B.
Mind you, because I am not that interested in long-context, fire-and-forget tasks, I think the faster MoE model has more value generally.
I'm not sure what you mean.
Looking at the chart on this website, Bonsai Qwen 3.6 27B has a lower average benchmark score than Qwen 3.5 35B-A3B (77.1 vs 82.9)
The 27B model beats 35B A3B. The quantized 1-bit version obviously doesn't, but it would likely beat a 1-bit quantized A3B variant. Comparing 3.5 35B or 3.6 35B to this model (Maple) is not meaningful.
“Current approaches to low precision primarily focus on converting models trained in full precision to lower bitwidths. We view this as fundamentally the wrong approach…”
Very excited to see how it performs, I’ve been a bit skeptical of the efficacy of converting existing models - really cool to see one trained from scratch in the ternary format.
At this point I think Apple just needs to simply not do anything stupid and these small model makers are going to hand them models
In mice. I mean Mac Mini M4 not an iPhone.
Also I love AI sites. Fancy font, plain serious style, we "introduce" rather than "release". It's an AI not an animal after all.
My question is what LLM the author used for dev the web page? Maybe Claude? or ChatGPT?
>"At ultra-low bitwidths,
matrix multiplication can be effectively replaced with additions
, lowering the total arithmetic workload needed to infer through a model."
Interesting! I never knew that before... I will have to do some more research on this, but yes, it definitely sounds possible!
(Also (and this is just a wild guess/hunch here!)
in theory, matrix multiplication could be replaced with pre-computed cached lookups from a lookup table in memory
if, if and only if the resultant pre-computed lookup table of result matrices could fit into memory available!
In other words, think of a Matrix not as a 2D array, but as a 1D string. Think of the Matrix multiply operation as first appending the 1D string of the first matrix to the 1D string of the second matrix, then using that string as a Key to look-up a precomputed result in Memory...
Of course, the limitation to this approach is the availability of memory relative to the permutations of input values... It could work -- but only for smaller matrices... which leads to another strange idea... in that set of pre-computed matrix permutations, there might be duplicates... figure out a way to put duplicates in memory only once (could use pointers or some other scheme to do this) and you could theoretically store more matrices in the same memory and/or slightly larger matrices!
Anyway, just rambling out loud! :-)
Remember, "almost all programming can be viewed as an exercise in caching" to quote the legendary programmer Terje Mathisen! :-) )
Anyway, great article!
As of now, 3 of the 5 comments on this page are just 0-1 karma accounts high-fiving the article. Suspicious.
You can play with it online. I was pretty impressed with the speed/quality given it's size.
It's definitely not going to replace a larger frontier model, but it's worth keeping an eye on.
amazing work! impressed with https://deepgrove.ai/chat
the dreaming demo peaks at 5.9 GB, is that mac-only? ios would jetsam a third-party app well before that on most iphones
Looks promising though much like the bonsai tenary one it hallucinates knowledge quite aggressively. The online chat having search tools covers this up somewhat, but it's still there.
...from very unscientific casual vibes it does seem pretty good though considering the speed
Would also be curious what their search tool backend looks like - that too is very fast for very rapid multiple searches
Makes me wonder how much we can 'get away with' in terms of raw model capability, when you have tool access for anything specialised or specific.
I probably could make up a bit of an incorrect carrot cake recipe if asked on the spot, but with a Google search I can give you something far more robust.
Maybe we don't need a 'country of geniuses' in our pocket, but more a helpful assistant that can reasonably reason!
Anyone compare this to ternary bonsai vs the 1 bit
I haven't compared it for perf, but at first glance this appears to be an LLM (text-only) while both binary and ternary bonsai 27B models are VLMs (where the vision tower and the adapter MLP weights are unquantized and kept in float16).
Edge is edging closer!
Super cool and a taste of what's to come with local AI becoming more accessible to low-end hardware.
This is cool!