Recently tried the pelican test on GPT-OSS which was probably one of the best local models of 2025. So cool to see how models have improved in the SVG pelican!
Doesn't have to be specific training, just has to consume simonw's blog. It's got lots of SVG pelicans, with helpful commentary on how good they are. I think there might be some kind of hill climbing going on here.
I have been overly critical and arguing in bad faith about your writing in the past. As well as negative towards you, which in turn was breeding a bad environment.
While I dont really enjoy LLMs, you did help me realize my unreasonable feelings as well as realize the occupation (and the joys I got from it) is essentially dead from it’s previous iteration and that I should let go and just join in the “I’m doing it for the money and attention” crowd. I will still just hand code my own projects and not use LLMs when I can.
I think it’s cool you started the pelican meme however useful it really is even if only aesthetically.
As of today, it has fallen to 8/9th on the rankings. I don't see a reason where you would use this model over competitors. However, price economics are bit confusing, as currently the effective input price of Hy3 via OpenRouter is now the same as DeepSeek-hosted DeepSeek Flash V4.
Do people really use 100B+ models for writing? I am no writer but to me it seems like writing is one of the easiest tasks with barely any logic or reasoning and as long as its not longer than a handful of pages I expect even 8B models to perform great.
It's pretty clear you've never experimented with it. Creative writing demands everything the model can do and more, and most problems are still unsolved. It's extremely heavy reasoning-wise, more so than coding (check e.g. Engram paper for some insights), but also needs good scattered retrieval, careful subjective training for prose quality, character, and human likeness, a ton of facts baked in, and much much more. Mode collapse is not solved. No LLM does creative writing well but historically only the absolute largest models were able to do write anything complex more or less convincingly and were creative enough.
> I am no writer but to me it seems like writing is one of the easiest tasks with barely any logic or reasoning
Virtually all logic or reasoning is, in one way or another, part of the support for writing. It’s what separates actual writing from generating nonsense that happens to fit grammar rules.
The specific details depend on the domain, of course, but I can’t see how anyone familiar with the output of writing can think that there is little logic or reasoning in doing it well.
If you can't make yourself sound adequately smart, it could lead to people ignoring you, and/or acting in spite of your opinions/logic, and/or spending extra effort trying to decipher you. That is not an optimal situation, especially in cases where you would be right[1].
1: I don't think you're right in this instance, but that's beside the point.
The wording wasnt very good I ment compared to programming or math the amount of logic and reasoning is small (Research level math hardly compares to writing a book in raw reasoning and logic). And I thing the smaller models have enough "intelligence" to write coherent with logical world building, but only the big models can truly do hard math and programming work
The largest model I've post-trained in the last 2 years of working on this problem was Kimi 2.5 at 1T parameters.
The simplest way I'd put it is, teaching a model to write coherently (follow rules, patterns, etc.) is easy enough: just use teacher forcing. Teaching a model to write creatively is easy enough: just use RL and punish it for not being creative.
Teaching a model to write well and creatively takes learning two partially opposing objectives that spike the learning requirements in ways that smaller models really struggle with.
Once creativity is being measured in isolation, getting multiple responses from the model is enough to measure creativity a ton of different ways: wordfreq to identify overused phrases, getting multiple responses for the same prompt and promoting the least similar as preferred for policy optimization, etc.
But that's of limited use for stuff like getting diverse names and such. You want creativity and coherency, and if you just punish the model for using an overused phrase, the first thing it does is strongly learn a new overused phrase (or gibberish).
(Also I don't think you mean unsupervised. You probably mean without humans [since LLMs struggle to judge creativity], but that's not what unsupervised means.)
I did read the the full comment and I did in fact mean exactly what I wrote when I used the term "unsupervised". I think the condescension does nothing but get in the way. Try extending the benefit of the doubt.
> enough to measure creativity a ton of different ways ...
The things you listed seem more like temperature than creativity to me. At this point it occurs to me that this is likely yet another case of highly misleading technical jargon. Suffice to say that truly creative writing requires something entirely different than unusual sentence structure - in fact it doesn't require unusual phrasing at all.
Re unsupervised, it seems the misunderstanding here follows naturally from the previous difference in word meaning. Hopefully you see the difficulty of scoring long form answers for the creativity of the underlying ideas, as well as the impossibility of using a labeled dataset to train on such a criteria.
With such a low baseline for what's unusual, you do need to get the LLM writing unusual phrases relative to its baseline. Otherwise you get things like repeated n-grams and overused constructs ("it's not X it's Y"), and suddenly the output is predictably not perceived as creative by humans even if you were to insert some otherwise creative or novel premise.
Getting the model to break out of that baseline without disrupting the model's ability to follow technical rules, maintain logic and reasoning, etc. is the difficult part.
-
Also you're again saying unsupervised then following up with descriptions that sure sound like you're referring to RL and supervised learning respectively this time. (supervised learning can improve creativity by the way
> Getting the model to break out of that baseline without disrupting the model's ability to follow technical rules, maintain logic and reasoning, etc. is the difficult part.
Sure, that is also somewhat challenging and is necessary to get human sounding prose. However doing so is not sufficient to produce "creative" literature by any reasonable metric.
> you're again saying unsupervised then following up with descriptions that sure sound like you're referring to RL and supervised learning respectively this time.
Are you sure it isn't you who is confused about the usage of those terms? I merely suggested that both preparing and making use of labeled data (ie supervised learning) seemed like it would prove quite difficult here. Quoting from wikipedia (https://en.wikipedia.org/wiki/Unsupervised_learning):
> Unsupervised learning is a framework in machine learning where, in contrast to supervised learning, algorithms learn patterns exclusively from unlabeled data.
This is admittedly an aside from the content of the post itself, but... why do so many mobile sites insist on preventing zooming in and seem to share the same incredibly buggy image zooming? It's quite frustrating.
Curious how people feel about this compared to DS4 Flash, given they are pretty close in size. Also curious how well it holds up to heavy quantization.
DS4 Flash can currently run reasonably well on systems with ~96gb+ RAM, I wonder if Hy3 can compete there.
One thing that might not be obvious about about DSV4 is how much innovation the Deepseek team implemented in its architecture. When llama.cpp fully supports its lightning indexer, the full 1M context will only require about 6G of RAM. So even though they are similar in size, I believe Deepseek will be much more efficient in that regard.
> I wonder if Hy3 can compete there
Highly depends on how well Hy3 is resilient to quantization. DSV4 is useful even at 2-bit quants.
Having heavily evaluated both antirez’s ds4 flash and Qwen 3.6 27B at FP8 and Q8: it depends. The quantised Flash is better in a number of tasks despite running much slower on my DGX Spark-alike.
27B is amazing for its size but has some surprising limits when used for longer agentic coding sessions, especially if you’re using tools that are outside the stock standard web tech stuff: it really isn’t good at Relay, for example.
I think its good advice to test both on your own evals for sure, but the MoE parameters are already natively FP4 in ds4. Dropping to 2bpw isn't as big of a loss as it seems (and as corroborated by antirez's work).
Its also only 13B active, so your decode speed would be nearly 2x that of Qwen3.6-27B. So there are other latent benefits as well.
I'm running the qwen3.6-27B + dflash on a spark and tgen is way up, but keep the draft count low, acceptance rate is terrible beyond half a dozen and it requires more memory
Wow thanks for pointing this out! This is actually what I was hoping would happen when the deepspec stuff dropped! And having zlab create these confirms my bias that I think open models are the way.
Isn't Q8 way overkill these days? I see many graphs showing Q4 or Q5 having less than %1 deviation. Nvidia's NVFP4 Qwen quantization should be even better due to its better training methods.
Q8 isn't overkill if you have sufficient RAM to fit the whole model, and you care about quality. There's a number of people who have enough hardware to fit exactly one 27B to 35B size Q8 model and not more than that, so if you can fit the whole thing in Q8, no reason to use Q4 or Q6.
When orgs/bencmarks claim 1% deviation, in most cases that means measuring perplexity loss on datasets like wikitext or c4. Even if the loss is calculated via KLD or similar, its not a good proxy for whats actually degradaing at the task level across an entire rollout.
And for MoEs, very small amounts of loss can mean you're flipped to entirely different experts (this is also a problem more broadly with numerical stability issues too).
Qwen3.6 below Q8 often can't exit a reasoning loop (until it hits max output token count), forgets to insert a tool call, often mistakenly inserts them inside the thinking block... It's still usable though.
Careful with those graphs, they're usually evaluating the model on KLD on relatively short transcripts. When you're running with 100k token contexts and the model running close loop a difference that looks small in terms of KLD may be quite substantial.
I'm not aware of any great benchmarks that work by giving it a live agentic harness and a number of realistic tasks that take most of the context window to accomplish and evaluate success rate and tokens to completion... but that's what you'd really want to use to judge different quantization levels.
I was playing with Hy3 via openrouter yesterday (and I've also been using DS4 Flash/Pro as a daily driver since I cancelled my Anthropic sub a week ago).
I've found DS4 Flash to be very temperental (via Claude Code). The speed is great, but it often builds a completely wrong mental model and charges off down the wrong path. I find myself needing to rein it in regularly (and also compact the history, which undercuts the whole cache price advantage).
Hy3 isn't as fast, but so far it seems to stay on track much more reliably than DS4 Flash. It also doesn't seem to degrade as much with longer context. I'm not sure what the real pricing is, but I feel like it's a very competitive model.
As an aside, I also nabbed a 50m token pack for LongCat 2.0 to give it a whirl. Not free, but it's so cheap they're basically giving it away. Very impressed too - seems roughly on par with Hy3. Not frontier-level intelligence, but a dependable workhorse that can navigate a codebase well and can reliably execute what you tell it to do.
Hy3 lacks the DSv4 architecture's KV Cache efficiency.
Whereas I can run DSv4 Flash on a pair of DGX Sparks and have enough memory left over for 3M tokens of KV cache, with Hy3 (quantized to FP4), there is only room for ~130K tokens of KV cache.
Lower context window notwithstanding, Hy3's coding benchmarks hold their own against DeepSeek v4 Pro & MiMo v2.5 Pro. That's quite something for a model priced like DeepSeek v4 Flash & MiMo v2.5 (for non-cached tokens), which are 3x cheaper than their respective Pro variants.
It's impressive indeed. I would also expect the next checkpoint of DSv4 Flash to come in somewhere at this level (DeepSeek has had over 2 months to continue training since it released).
It's exciting that the open models continue to get better and more efficient across the board!
This model is shockingly small for how capable it is. its a little bit bigger than deepseekV4 flash but around as capable if not more on some benchmarks than V4 pro, i wouldnt be surprised if this becomes a popular local model.
I've been wondering about that. GLM-5.2 is also half the size of DeepSeek V4 Pro. (But costs roughly twice as much.)
I looked into DeepSeek's architecture a little bit and the main focus was how can we save as much money as possible. They did a lot of cost cutting with the attention mechanisms. This allowed them to offer an insanely cheap price even on massive contexts, but seems to have come at the cost of performance?
At least, that's my guess, when I see smaller models costing more and outperforming, I think, "they must have denser attention?"
The current Deepseek V4 Pro is still just their initial preview AFAIK, with the "real" model release rumored to come later this month. GLM-5.2 might be outperforming simply because it's had more post-training on top of the GLM-5 base.
Yeah i shouldve been more clear, a model of this size could run on 2 dgx sparks so out of the range of a lot of the typical consumer sure, but I think there is definitely a market for that size
I feel like I'm taking crazy pills with hy3, it's either benchmaxxed to hell and back or skill issue on my part but I'd rather use dense gemma. I don't think there's a single model that's wasted more of my time in recent memory.
The Hy3 preview has been a mediocre performer in my benchmarks of security auditing with models, and yes, it is outperformed by Gemma 4 (31b soundly beats it, the MoE does slightly better, even at 4-bit quantization when using the QAT version). Qwen 3.6 27b also beats it.
I'll try it again now that it's out of preview and has been updated with more post-training. It presumably can't be worse, so maybe it's better enough to compete with a 31b model.
There are extreme diminishing returns in real world performance as models get bigger. 10x bigger might mean 5-10% better on benchmarks, a margin that can easily mean it's functionally equivalent in real world use or even a worse performer depending on the context it's being used in, and how good you are at providing meaningful context.
Of course the bigger model embeds more knowledge, but when neither model has the knowledge necessary to perform the task, hy3 makes idiotic decisions all the time whereas gemma 31b has a decent hit rate.
hy3 feels like someone who's read a lot of books and says the right words but has nothing of substance between their ears, gemma feels like a reasonably intelligent person who doesn't understand the domain, the latter is muuuch easier to work with than the former.
Gemma 4 is the first really small model that feels smart, to me. I mean, Qwen 3.6 is arguably better at some coding tasks. But, Gemma 4 has shockingly good reasoning for a small model. Even the tiny 12B, at 7GB on disk in the 4-bit QAT quant, feels like a really big model of a couple years ago. It's a good tool user, can search the web (when given the appropriate skill or MCP), has good vision capabilities, and pretty good prose.
I've only used the Hy3 preview, so I don't want to judge too harshly, yet. But, I wasn't very impressed with it a couple of months ago.
What we really need is a breakthrough in inference or LLM architecture to allow running GLM-5.2-level models at the size of Qwen 3.6 27b or smaller on consumer devices like a 48GB Macbook Pro, and at least at 100 tokens/second. My hypothesis is that a smaller, less capable but faster model paired with a good harness can run for longer and brute force its way out to solve problems that the bigger models can one-shot.
used to heat my dorm room with a Pentium D that I overclocked and pointed a small fan at. Could open a window in winter, turn off the room radiator, and keep it cozy.
perhaps at-home LLMs will bring me back to that. fun days of hacking and thermodynamics.
Why not! And free hot showers during the summer. Just looking at the progress made since 2023, I don't think the LLM architecture we have today is the most efficient. We need creative game developers to start making LLMs.
I tried out the model it's pretty great, better than ~~gpt5.4~~ gpt-5.4-mini perhaps, atleast close enough to sonnet 5 in performance that I didn't notice much of a gap.
Not really at gpt 5.5 tier though, and probably below glm 5.2...
But most of all it just works for me for most things I tried and it's exceedingly cheap so there is no reason not to use it, if you need a foss model.
A lot of contaminated benchmarks in the blog post about Hy3, needs real testing though I have a distinct feeling it's benchmaxxed like a lot of Chinese models.
The economics is on the Fable tier people are willing to spend a lot on it and on the Open tier you have to give it away to drive usage. The bottom tiers are also getting more and more competitive.
Been using this and GLM 5.2 back and forth. I like the speed of Hy3. Also seems very happy to follow instructions. Still haven’t found any open models that follow instructions as good as Mimo v2 pro though
MiMo v2.5 Pro is very spiky, in my experience. Sometimes excellent, sometimes mediocre. Weirdly high non-deterministic behavior. Run the same task three times, get three different results. I mean, they're all rolling dice for the next word, but MiMo seems to run hot on the randomness dimension in my benchmarks.
But, it performs very well for its size. I just looked it up, and it's much smaller than I thought it was when I was testing it. 310B A15B is tiny for how well it performs. I guess that explains why it's so cheap.
Quite interesting to see them and Meta and others release before OpenAI supposedly is to release GPT 5.6 today, would it be better to release it before or after? Calm before the storm type of thing?
It's a very good model for this size and price. I tried it with a couple of small tasks - just an year ago this would be the level of the leading models.
I'm sorry but what on earth is going on with that bar chart, the bars are not consistent. E.g., in the frontierscience-olympiad chart Hy3 preview scores the same as DeepSeek (70.0) but Hy3 preview's bar is visibly lower.
If they can't tell me what it is or how to use it then they can't explain to me how to install it or anything else properly so forget it. I'm not wasting my time trying to figure it out.
Got really excited for a minute that the long-standing [Hy](https://hylang.org) project had had a release, but it's just some confusingly-named LLM. Shame.
The strange names are mainly initials from Chinese pinyin. The first generation of Tencent Hunyuan was released in 23Q3, so it is already quite a veteran.
Pelican from a few days ago: https://simonwillison.net/2026/Jul/6/hy3/ - I was using the free tier on OpenRouter, which expires on July 21st.
I tried the preview model 41 days ago and got a pelican with a "change pelican color" button: https://static.simonwillison.net/static/2026/hy3-preview-pel...
I quite enjoy that the "animate wheels" button animates the sun instead.
Recently tried the pelican test on GPT-OSS which was probably one of the best local models of 2025. So cool to see how models have improved in the SVG pelican!
I'm skeptical, as tests go, I think that's burned out now. They could easily be training specifically to get a better pelican...
Doesn't have to be specific training, just has to consume simonw's blog. It's got lots of SVG pelicans, with helpful commentary on how good they are. I think there might be some kind of hill climbing going on here.
If they were training specifically for it the result would be much better
I have been overly critical and arguing in bad faith about your writing in the past. As well as negative towards you, which in turn was breeding a bad environment. While I dont really enjoy LLMs, you did help me realize my unreasonable feelings as well as realize the occupation (and the joys I got from it) is essentially dead from it’s previous iteration and that I should let go and just join in the “I’m doing it for the money and attention” crowd. I will still just hand code my own projects and not use LLMs when I can. I think it’s cool you started the pelican meme however useful it really is even if only aesthetically.
Curious why TFA calls out "Tencent in China".
Is there a Tencent AI lab elsewhere (MiniMax have some association with Tencent, for example)?I think it's just their version of "Designed by Apple in California"
Tencent seems to have subsidiaries: https://en.wikipedia.org/wiki/Tencent#Subsidiaries
Also they have large European / South African shareholders.
Tony the Tiger on a bicycle selling addictive items to children is not new.
A month ago I wrote a blog post about how Hy3 was topping the OpenRouter rankings despite no one talking about it: https://news.ycombinator.com/item?id=48317294
As of today, it has fallen to 8/9th on the rankings. I don't see a reason where you would use this model over competitors. However, price economics are bit confusing, as currently the effective input price of Hy3 via OpenRouter is now the same as DeepSeek-hosted DeepSeek Flash V4.
https://openrouter.ai/tencent/hy3-preview
https://openrouter.ai/deepseek/deepseek-v4-flash
I had to stop using it because I was getting rate limited like crazy. Probably why it has dropped. Seemed like they couldn't keep up with demand.
Because it was free with generous limits and high availability, until it wasn't.
That was the preview model right? This one appears to be significantly better.
I mean it's still a small model, but at least the benchmark scores (incl. on DeepSWE) went up significantly.
It costs as much as Flash, but the benchmarks are on par with Pro (or above in some cases).
Of course, benchmarks are mostly meaningless -- the only real benchmark is the actual work you give it :)
It is really slow on openrouter and I ran into many http errors.
Writes pretty engaging prose, finetunes well, now MIT licensed... what's not to like?
Oh and very good world knowledge for the size: better than than DS4 Flash
Do people really use 100B+ models for writing? I am no writer but to me it seems like writing is one of the easiest tasks with barely any logic or reasoning and as long as its not longer than a handful of pages I expect even 8B models to perform great.
It's pretty clear you've never experimented with it. Creative writing demands everything the model can do and more, and most problems are still unsolved. It's extremely heavy reasoning-wise, more so than coding (check e.g. Engram paper for some insights), but also needs good scattered retrieval, careful subjective training for prose quality, character, and human likeness, a ton of facts baked in, and much much more. Mode collapse is not solved. No LLM does creative writing well but historically only the absolute largest models were able to do write anything complex more or less convincingly and were creative enough.
> I am no writer but to me it seems like writing is one of the easiest tasks with barely any logic or reasoning
Virtually all logic or reasoning is, in one way or another, part of the support for writing. It’s what separates actual writing from generating nonsense that happens to fit grammar rules.
The specific details depend on the domain, of course, but I can’t see how anyone familiar with the output of writing can think that there is little logic or reasoning in doing it well.
Of course there is logic but its nowhere near the complexity of math or programming
let me just say, you're not going to sound smart saying that
Useless comment
If you can't make yourself sound adequately smart, it could lead to people ignoring you, and/or acting in spite of your opinions/logic, and/or spending extra effort trying to decipher you. That is not an optimal situation, especially in cases where you would be right[1].
1: I don't think you're right in this instance, but that's beside the point.
> with barely any logic or reasoning
I take it you enjoy works of literature with inconsistent world building?
Or do you mean professional as opposed to creative writing? Because the bar is even higher for that.
The wording wasnt very good I ment compared to programming or math the amount of logic and reasoning is small (Research level math hardly compares to writing a book in raw reasoning and logic). And I thing the smaller models have enough "intelligence" to write coherent with logical world building, but only the big models can truly do hard math and programming work
> The wording wasnt very good
Writing isn't so easy after all.
The largest model I've post-trained in the last 2 years of working on this problem was Kimi 2.5 at 1T parameters.
The simplest way I'd put it is, teaching a model to write coherently (follow rules, patterns, etc.) is easy enough: just use teacher forcing. Teaching a model to write creatively is easy enough: just use RL and punish it for not being creative.
Teaching a model to write well and creatively takes learning two partially opposing objectives that spike the learning requirements in ways that smaller models really struggle with.
> is easy enough: just use RL and punish it for not being creative.
How are you scoring creativity in an unsupervised manner? That seems anything but easy.
Did you try reading the whole comment?
Once creativity is being measured in isolation, getting multiple responses from the model is enough to measure creativity a ton of different ways: wordfreq to identify overused phrases, getting multiple responses for the same prompt and promoting the least similar as preferred for policy optimization, etc.
But that's of limited use for stuff like getting diverse names and such. You want creativity and coherency, and if you just punish the model for using an overused phrase, the first thing it does is strongly learn a new overused phrase (or gibberish).
(Also I don't think you mean unsupervised. You probably mean without humans [since LLMs struggle to judge creativity], but that's not what unsupervised means.)
I did read the the full comment and I did in fact mean exactly what I wrote when I used the term "unsupervised". I think the condescension does nothing but get in the way. Try extending the benefit of the doubt.
> enough to measure creativity a ton of different ways ...
The things you listed seem more like temperature than creativity to me. At this point it occurs to me that this is likely yet another case of highly misleading technical jargon. Suffice to say that truly creative writing requires something entirely different than unusual sentence structure - in fact it doesn't require unusual phrasing at all.
Re unsupervised, it seems the misunderstanding here follows naturally from the previous difference in word meaning. Hopefully you see the difficulty of scoring long form answers for the creativity of the underlying ideas, as well as the impossibility of using a labeled dataset to train on such a criteria.
The first thing I read from you was a sardonic browbeating in response to the exact comment I gave an earnest response.
And even in domains that lean heavily on "usual phrasing", like technical writing, human writing has notably higher perplexity compared to another LLM's outputs: https://www.sciencedirect.com/science/article/abs/pii/S10766...
With such a low baseline for what's unusual, you do need to get the LLM writing unusual phrases relative to its baseline. Otherwise you get things like repeated n-grams and overused constructs ("it's not X it's Y"), and suddenly the output is predictably not perceived as creative by humans even if you were to insert some otherwise creative or novel premise.
Getting the model to break out of that baseline without disrupting the model's ability to follow technical rules, maintain logic and reasoning, etc. is the difficult part.
-
Also you're again saying unsupervised then following up with descriptions that sure sound like you're referring to RL and supervised learning respectively this time. (supervised learning can improve creativity by the way
> Getting the model to break out of that baseline without disrupting the model's ability to follow technical rules, maintain logic and reasoning, etc. is the difficult part.
Sure, that is also somewhat challenging and is necessary to get human sounding prose. However doing so is not sufficient to produce "creative" literature by any reasonable metric.
> you're again saying unsupervised then following up with descriptions that sure sound like you're referring to RL and supervised learning respectively this time.
Are you sure it isn't you who is confused about the usage of those terms? I merely suggested that both preparing and making use of labeled data (ie supervised learning) seemed like it would prove quite difficult here. Quoting from wikipedia (https://en.wikipedia.org/wiki/Unsupervised_learning):
> Unsupervised learning is a framework in machine learning where, in contrast to supervised learning, algorithms learn patterns exclusively from unlabeled data.
Novita is offering free Hy3 on OpenRouter until July 21st
https://openrouter.ai/tencent/hy3:free
https://x.com/novita_labs/status/2074158304159510819
Nice
This is admittedly an aside from the content of the post itself, but... why do so many mobile sites insist on preventing zooming in and seem to share the same incredibly buggy image zooming? It's quite frustrating.
Curious how people feel about this compared to DS4 Flash, given they are pretty close in size. Also curious how well it holds up to heavy quantization.
DS4 Flash can currently run reasonably well on systems with ~96gb+ RAM, I wonder if Hy3 can compete there.
> given they are pretty close in size
One thing that might not be obvious about about DSV4 is how much innovation the Deepseek team implemented in its architecture. When llama.cpp fully supports its lightning indexer, the full 1M context will only require about 6G of RAM. So even though they are similar in size, I believe Deepseek will be much more efficient in that regard.
> I wonder if Hy3 can compete there
Highly depends on how well Hy3 is resilient to quantization. DSV4 is useful even at 2-bit quants.
I have been telling people exactly this the last few months.
We have not seen the full power of deepseek v4 yet.
That's a 2-bit quant of DS4 flash. You're probably better off running Qwen3.6-27B at Q8.
Having heavily evaluated both antirez’s ds4 flash and Qwen 3.6 27B at FP8 and Q8: it depends. The quantised Flash is better in a number of tasks despite running much slower on my DGX Spark-alike.
27B is amazing for its size but has some surprising limits when used for longer agentic coding sessions, especially if you’re using tools that are outside the stock standard web tech stuff: it really isn’t good at Relay, for example.
I think its good advice to test both on your own evals for sure, but the MoE parameters are already natively FP4 in ds4. Dropping to 2bpw isn't as big of a loss as it seems (and as corroborated by antirez's work).
Its also only 13B active, so your decode speed would be nearly 2x that of Qwen3.6-27B. So there are other latent benefits as well.
z-lab has been dropping dflash addons for a lot of models
https://huggingface.co/collections/z-lab/dflash
I'm running the qwen3.6-27B + dflash on a spark and tgen is way up, but keep the draft count low, acceptance rate is terrible beyond half a dozen and it requires more memory
Wow thanks for pointing this out! This is actually what I was hoping would happen when the deepspec stuff dropped! And having zlab create these confirms my bias that I think open models are the way.
For most coding or agentic tasks, Qwen 3.6 27B likely outperforms, yes.
For 'general intelligence', DS4 Flash seems to be a noticeable step up still.
I suspect it would depend on the task. DS4-flash does, as previously mentioned, handle quantization very well. Even at 2-bit it's still very coherent.
qwen 27b at q8 is slower and worse than ds4 at q2 in my experience.
Isn't Q8 way overkill these days? I see many graphs showing Q4 or Q5 having less than %1 deviation. Nvidia's NVFP4 Qwen quantization should be even better due to its better training methods.
Q8 isn't overkill if you have sufficient RAM to fit the whole model, and you care about quality. There's a number of people who have enough hardware to fit exactly one 27B to 35B size Q8 model and not more than that, so if you can fit the whole thing in Q8, no reason to use Q4 or Q6.
When orgs/bencmarks claim 1% deviation, in most cases that means measuring perplexity loss on datasets like wikitext or c4. Even if the loss is calculated via KLD or similar, its not a good proxy for whats actually degradaing at the task level across an entire rollout.
And for MoEs, very small amounts of loss can mean you're flipped to entirely different experts (this is also a problem more broadly with numerical stability issues too).
Qwen3.6 below Q8 often can't exit a reasoning loop (until it hits max output token count), forgets to insert a tool call, often mistakenly inserts them inside the thinking block... It's still usable though.
1.01 over 30k tokens is over a googol (a large number with 100 zeroes)
It depends on model size I think, but yeah, from my understanding at ~30B and below Q6 or even Q4 will get you 95%+ of the way there
Careful with those graphs, they're usually evaluating the model on KLD on relatively short transcripts. When you're running with 100k token contexts and the model running close loop a difference that looks small in terms of KLD may be quite substantial.
I'm not aware of any great benchmarks that work by giving it a live agentic harness and a number of realistic tasks that take most of the context window to accomplish and evaluate success rate and tokens to completion... but that's what you'd really want to use to judge different quantization levels.
I was playing with Hy3 via openrouter yesterday (and I've also been using DS4 Flash/Pro as a daily driver since I cancelled my Anthropic sub a week ago).
I've found DS4 Flash to be very temperental (via Claude Code). The speed is great, but it often builds a completely wrong mental model and charges off down the wrong path. I find myself needing to rein it in regularly (and also compact the history, which undercuts the whole cache price advantage).
Hy3 isn't as fast, but so far it seems to stay on track much more reliably than DS4 Flash. It also doesn't seem to degrade as much with longer context. I'm not sure what the real pricing is, but I feel like it's a very competitive model.
As an aside, I also nabbed a 50m token pack for LongCat 2.0 to give it a whirl. Not free, but it's so cheap they're basically giving it away. Very impressed too - seems roughly on par with Hy3. Not frontier-level intelligence, but a dependable workhorse that can navigate a codebase well and can reliably execute what you tell it to do.
Hy3 lacks the DSv4 architecture's KV Cache efficiency.
Whereas I can run DSv4 Flash on a pair of DGX Sparks and have enough memory left over for 3M tokens of KV cache, with Hy3 (quantized to FP4), there is only room for ~130K tokens of KV cache.
Lower context window notwithstanding, Hy3's coding benchmarks hold their own against DeepSeek v4 Pro & MiMo v2.5 Pro. That's quite something for a model priced like DeepSeek v4 Flash & MiMo v2.5 (for non-cached tokens), which are 3x cheaper than their respective Pro variants.
It's impressive indeed. I would also expect the next checkpoint of DSv4 Flash to come in somewhere at this level (DeepSeek has had over 2 months to continue training since it released).
It's exciting that the open models continue to get better and more efficient across the board!
DS4-Flash is not only "significantly" smaller, it will also benefit from a lot more speed thanks to DSpark
299B for Hy3 vs 284B* for Flash
Edit: fixed, got bad info
flash is 284b isnt it? https://artificialanalysis.ai/models/deepseek-v4-flash
Oh, it is. I was looking at the Huggingface repo which listed the lower number at the top of the page, looks like that's wrong.
Same, I was relying on that. Weird!
I don’t like DS4 in my experiences with it I still prefer qwen locally and glm on api
This model is shockingly small for how capable it is. its a little bit bigger than deepseekV4 flash but around as capable if not more on some benchmarks than V4 pro, i wouldnt be surprised if this becomes a popular local model.
I've been wondering about that. GLM-5.2 is also half the size of DeepSeek V4 Pro. (But costs roughly twice as much.)
I looked into DeepSeek's architecture a little bit and the main focus was how can we save as much money as possible. They did a lot of cost cutting with the attention mechanisms. This allowed them to offer an insanely cheap price even on massive contexts, but seems to have come at the cost of performance?
At least, that's my guess, when I see smaller models costing more and outperforming, I think, "they must have denser attention?"
The current Deepseek V4 Pro is still just their initial preview AFAIK, with the "real" model release rumored to come later this month. GLM-5.2 might be outperforming simply because it's had more post-training on top of the GLM-5 base.
If the "final" release of Deepseek V4 Pro outperforms GLM-5.2 while maintaining the current Deepseek price, then it's going to be a marvel.
hardly, its still quite big unless by "local" you mean people that spend many thousands on rigs :)
Yeah i shouldve been more clear, a model of this size could run on 2 dgx sparks so out of the range of a lot of the typical consumer sure, but I think there is definitely a market for that size
> Hy3 has 295B parameters in total. To serve it on 8 GPUs, we recommend using H20-3e or other GPUs with larger memory capacity.
I would.
I feel like I'm taking crazy pills with hy3, it's either benchmaxxed to hell and back or skill issue on my part but I'd rather use dense gemma. I don't think there's a single model that's wasted more of my time in recent memory.
The Hy3 preview has been a mediocre performer in my benchmarks of security auditing with models, and yes, it is outperformed by Gemma 4 (31b soundly beats it, the MoE does slightly better, even at 4-bit quantization when using the QAT version). Qwen 3.6 27b also beats it.
I'll try it again now that it's out of preview and has been updated with more post-training. It presumably can't be worse, so maybe it's better enough to compete with a 31b model.
For context Hy is 295B, roughly on par with DeepSeek V4 Flash (284B).
I haven't tested it yet so I cannot comment on the quality (nor the comparison with 10x smaller (!) models)
There are extreme diminishing returns in real world performance as models get bigger. 10x bigger might mean 5-10% better on benchmarks, a margin that can easily mean it's functionally equivalent in real world use or even a worse performer depending on the context it's being used in, and how good you are at providing meaningful context.
Of course the bigger model embeds more knowledge, but when neither model has the knowledge necessary to perform the task, hy3 makes idiotic decisions all the time whereas gemma 31b has a decent hit rate.
hy3 feels like someone who's read a lot of books and says the right words but has nothing of substance between their ears, gemma feels like a reasonably intelligent person who doesn't understand the domain, the latter is muuuch easier to work with than the former.
Gemma 4 is the first really small model that feels smart, to me. I mean, Qwen 3.6 is arguably better at some coding tasks. But, Gemma 4 has shockingly good reasoning for a small model. Even the tiny 12B, at 7GB on disk in the 4-bit QAT quant, feels like a really big model of a couple years ago. It's a good tool user, can search the web (when given the appropriate skill or MCP), has good vision capabilities, and pretty good prose.
I've only used the Hy3 preview, so I don't want to judge too harshly, yet. But, I wasn't very impressed with it a couple of months ago.
Gemma 4 31B is underrated. It surprises me a lot.
What we really need is a breakthrough in inference or LLM architecture to allow running GLM-5.2-level models at the size of Qwen 3.6 27b or smaller on consumer devices like a 48GB Macbook Pro, and at least at 100 tokens/second. My hypothesis is that a smaller, less capable but faster model paired with a good harness can run for longer and brute force its way out to solve problems that the bigger models can one-shot.
That would be great during the winter months
used to heat my dorm room with a Pentium D that I overclocked and pointed a small fan at. Could open a window in winter, turn off the room radiator, and keep it cozy.
perhaps at-home LLMs will bring me back to that. fun days of hacking and thermodynamics.
Why not! And free hot showers during the summer. Just looking at the progress made since 2023, I don't think the LLM architecture we have today is the most efficient. We need creative game developers to start making LLMs.
Unfortunately I don't expect indie game devs can afford high spec hardware at current prices.
im more expecting the harness to be a literal LLM, Like how you put vibration dampeners on all kinds of mechanical structures
I tried out the model it's pretty great, better than ~~gpt5.4~~ gpt-5.4-mini perhaps, atleast close enough to sonnet 5 in performance that I didn't notice much of a gap.
Not really at gpt 5.5 tier though, and probably below glm 5.2...
But most of all it just works for me for most things I tried and it's exceedingly cheap so there is no reason not to use it, if you need a foss model.
Edited: gpt-5.4-mini not the base gpt-5.4
Hy3 DeepSWE - 28%
GPT5.4 xhigh DeepSWE - 52%
A lot of contaminated benchmarks in the blog post about Hy3, needs real testing though I have a distinct feeling it's benchmaxxed like a lot of Chinese models.
I think you’ve got the models wrong…gpt-5.4? I doubt there is any open source mode matching it. Maybe in a year
Yeah I meant gpt-5.4-mini, but GLM 5.2 is pretty close to gpt-5.4 base, and much better than it when it comes to design stuff.
GLM 5.2 already matches GPT-5.4 easily.
Worse than GLM-5.2, much more expensive than deepseek v4. Tough bargain.
There are now 3 tiers of competition:
-Fable + Gpt 5.6 Sol
-Opus + Gpt 5.6 Terra + Grok 4.5 + Muse Spark 1.1
-Open Chinese models: GLM + et family
The economics is on the Fable tier people are willing to spend a lot on it and on the Open tier you have to give it away to drive usage. The bottom tiers are also getting more and more competitive.
This site can’t be reached https://hy.tencent.com/research/hy3 is unreachable.
Interesting way to show off a model last on every benchmark. Not sure any other lab is doing this
More like second-to-last on most of them, to my eye. Which is impressive as it's smaller and cheaper than the others.
Been using this and GLM 5.2 back and forth. I like the speed of Hy3. Also seems very happy to follow instructions. Still haven’t found any open models that follow instructions as good as Mimo v2 pro though
MiMo v2.5 Pro is very spiky, in my experience. Sometimes excellent, sometimes mediocre. Weirdly high non-deterministic behavior. Run the same task three times, get three different results. I mean, they're all rolling dice for the next word, but MiMo seems to run hot on the randomness dimension in my benchmarks.
But, it performs very well for its size. I just looked it up, and it's much smaller than I thought it was when I was testing it. 310B A15B is tiny for how well it performs. I guess that explains why it's so cheap.
Er, actually, Pro is a big 1T model (which makes more sense given how well it does, sometimes). Regular MiMo is small, and I haven't tested it.
Quite interesting to see them and Meta and others release before OpenAI supposedly is to release GPT 5.6 today, would it be better to release it before or after? Calm before the storm type of thing?
It's a very good model for this size and price. I tried it with a couple of small tasks - just an year ago this would be the level of the leading models.
I was expecting a new release of the Hy language (https://hylang.org).
Yes, I was gonna say... Today, I am this old.
Congrats, the trial chat is QR locked. The AI companies are spoiled and get crazier every day (not only this one, US companies as well).
I'm sorry but what on earth is going on with that bar chart, the bars are not consistent. E.g., in the frontierscience-olympiad chart Hy3 preview scores the same as DeepSeek (70.0) but Hy3 preview's bar is visibly lower.
That UI demo page is… really quite janky.
Very impressive model for its size
I would never use any product that can't explain on its own front page what it is and why I should use it.
It's like a computer, but you can talk to it.
If they can't tell me what it is or how to use it then they can't explain to me how to install it or anything else properly so forget it. I'm not wasting my time trying to figure it out.
Got really excited for a minute that the long-standing [Hy](https://hylang.org) project had had a release, but it's just some confusingly-named LLM. Shame.
The strange names are mainly initials from Chinese pinyin. The first generation of Tencent Hunyuan was released in 23Q3, so it is already quite a veteran.