The big lab models are academically interesting but business wise they seem toast long term. There’s no way for these model companies to compete when the models are becoming a pure commodity and others offering options that are orders of magnitude cheaper.
It’s not that the big labs couldn’t theoretically just also put out 100x cheaper options but their business model requires them to generate huge revenues from higher priced tokens or they’ll implode.
castform founder here. despite our bet on fine-tuned smaller open-source models, i'm still quite bullish on the big labs. i think scaled closed models will continue to dominate for more general purpose use-case like codegen, search, etc. but intelligence has lots of long-tail applications and i think for these longer-tail applications, finetuned custom models will rule
There is so much opportunity for purpose built models like this. Ideally a harness should spin up a subagent to offload to targeted models for specific tasks like this. I know this is not a novel idea. Claude code does some of this by handing off the "explore" agent work to haiku. I just love seeing that specialized LLMs are being developed.
> Claude code does some of this by handing off the "explore" agent work to haiku.
That is not handing off to a specialized model, its just handing off to a lighter and interior model (compared to the parent model). That by itself can create issues like the lighter model not capturing all the data that the parent needs.
The idea is that we get specialized models that are better then general purpose models. But its rare for a specialized model to beat a strong general model.
There is a reason why we hear less about this idea of smaller expert models, because large strong models to the tasks just as good.
And if the tasks is repetitive to the point that specialization is useful, you can get into a situation that your better off having a program written for that reputative nature, then delegating to other models. And then have the main strong model, deal with the (semi)cleaned up data.
> There is a reason why we hear less about this idea of smaller expert models, because large strong models to the tasks just as good.
Smaller models are cheaper, sometimes faster. I agree that the “we’re an LLM fine-tuned for X” hasn’t worked out because you can just train Claude to do X (and Anthropic will), but not burning Opus/Fable tokens on dumb-but-token-heavy tasks is good sense.
As we move from “integrate AI into Y” to “optimize the ROI on Y”, we’ll see more of this.
castform founder here. while it's "rare for a specialized model to beat a strong general model" today, i think the tech/knowhow on how to do so is getting there. we see some early signs of this in industry e.g.
at the end of the day, models are only as good as the data they're trained on. and if one has access to proprietary data, it should yield specialized models that do better than larger general models
You can register models with mcp. I think it’s an expensive solution, but it is available in the framework. I use a light weight bus protocol that lets agents interact and pass short messages with pointers. It’s very efficient.
There are! Chroma has Context1, SID has SID-1, and you'd actually be surprised at how easy it is to post-train your own with pretty good pass@ recall@ ndcg@ etc.
There's also Hornet who have shared some interesting talks & blogs lately. I don't know that I'd exclusively use agents for retrieval the way Neon outlines here as well. I think distillation similar to what ZeroEntropy has done for bespoke retrieval & reranking with _some_ agent manipulation on top-k results works better (IME).
> Claude code does some of this by handing off the "explore" agent work to haiku
This is no longer necessarily true. As of 2.1.198 [0] (released July 1st): "The built-in Explore agent now inherits the main session’s model (capped at opus) instead of running on haiku"
I feel like the future is people building applications with tightly integrated LLMs that work hand in hand with the application's own lifecycle and code.
I also didn't realize that people were using agentic harnesses for search, it's an interesting idea. If the context length is short enough it should be fairly cheap compared to running "normal" agentic coding workloads where you have O(100k) context length for doing almost anything.
castform founder here. that's a future we are really excited about too :) ideally, you can post-train the llm within the application itself, as it's being used. both interesting infrastructure & algorithmic challenges here
There has been an over-obsession with frontier models and benchmarks. Most of the work will be done by task specific models. You don't put Phds on the factory floor.
(one of the blog post authors here) -> once you set up a finetuning pipeline, it's often trivial to rerun it on top of a new open weights model. so, it's orthogonal to base model improvements
castform founder here: totally! we also think model routing is also a post-training problem i.e. getting a model to predict the difficulty of a task and match it to the right model -> we're gonna be sharing more on that soon :)
> There is so much opportunity for purpose built models like this.
OpenAI etc could themselves do this, and maybe they already do? Where the public-facing interface delegates to multiple little goblins behinds the scenes
Exactly. There could be a lot of value for inference companies to do this. Could save a lot of money being able to hand off highly repetitive known tasks to far smaller specialized models.
There is a more serious question in here that's not being answered. How effective is the retrieval in finding buried needles in larger and larger haystacks. And there's a correlary question, how effective could you be in finding paired needles in that haystack where you need to hold a needle to unlock finding another needle.
Considering the state of the field ( RAG/retrieval/evaluation) I have 0 trust in it, even more if it's closed source with bullshit claim like that.
Everything is vibe sloped to death, and dead after a few months to a couple of years (and not hard to be 100 cheaper than GPT-5.6 sol ... DS is basically free and I guess already 100 times cheaper or more, and here another slope ).
I use detailed project files. It has data regarding the project and subtasks as well as task status. It doesn’t depend on agent context and it’s managed to keep the agent on track. Feature creep with the new models is a very real issue. Capturing principles and how to reconcile tasks helps too. Even today it brought up a source of truth issue it had detected. There were multiple authorities born out of a patch and it used that principle to highlight and resolve the problem.
This feels like the database equivalent of "use the right data structure." We've spent two years assuming the biggest general-purpose model should do everything. It makes more sense for retrieval, reranking, reasoning, and generation to each have their own optimized model if the routing cost is negligible.
I have done my own testing and found that smaller models can beat their larger siblings on fact retrieval from documents. I haven’t investigated it in depth with a large enough dataset but my guess is that larger models overthink it while smaller ones just do it. I would like if they compared this with 5.6 Luna instead.
we actually have the test benchmark against luna too! it's just not in our title but you can see it in the first diagram below the title. luna does pretty well tbh but sol is just a tad bit better. but luna is way cheaper.
Anecdotally, it feels like Opus, Fable, and Sol "get distracted" when you use them for writing code. Great at reasoning and coordination but they will go off on a tangent and refactor half the code base. I only use them for reasoning (of course) and coordinating subagents.
Have been feeling the same. There's a sweet spot that threads the needle between "too dumb to search the right thing / relay the correct results" and "too smart to just stop overthinking and just report the damn thing"
On what do you guys test the model. Its very dubious that there is no common retrieval benchmark such as browsecomp plus or similar tested. And what metric do you report?
(founder of castform here) - we didn't get to dive too deep into the dataset we were using for the retrieval in the blogpost for brevity, but we did link the training run (which shows the dataset) here: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...
the page shows the exact trace of all the models we are comparing against and the aggregate scores
we generated the question & answer pair from gitlab product handbook (https://handbook.gitlab.com/) since the point is to show that you can generate training questions from raw data corpus (something a company already has today)
Keeping track of any AI progress is becoming harder by the day, because there's ambiguity around common/clear/consistent benchmarks. Everything is constantly skewed into favourable directions.
Bit unrelated, I realized that z.ai gives you access to deepseek 4 flash. It's incredible how well it performs when given a detailed spec. I'm not sure I've seen a model one-shot like that, and I was already impressed by gemma 4's speed and efficiency.
Deepseek flash (especially after the recent update) has to be one of the most slept-on models. Price-performance is ridiculous, and its available on a number of cheap coding subscriptions
One thing that plagues [insert current FAANG] is the large amount of corpus knowledge that is outdated/misleading or just plain wrong. I'm curious how this addresses that if it's deriving the reward function from the corpus itself.
(founder of castform here!) - having worked at FAANG / big tech, i totally get this. our example was on gitlab's open source company handbook but i think a real company's corpus is way more messy and has many sources of truth.
a few ideas i have yet to validate are:
- prioritize recently updated docs when generating the training questions (assumption those docs are more correct than others)
- actually including contradicting documents that talks about the exact same topic might be a good training example - ideally the model should surface all the relevant info it can find, and explain what it has found. (usually contradiction comes from the fact that the later document is the updated stance)
- you could also mine high quality Q&A from public slack / communication channels where questions were asked and someone else in the team linked some docs / answer. those are strongly validated "ground truth" answers
The big lab models are academically interesting but business wise they seem toast long term. There’s no way for these model companies to compete when the models are becoming a pure commodity and others offering options that are orders of magnitude cheaper.
It’s not that the big labs couldn’t theoretically just also put out 100x cheaper options but their business model requires them to generate huge revenues from higher priced tokens or they’ll implode.
castform founder here. despite our bet on fine-tuned smaller open-source models, i'm still quite bullish on the big labs. i think scaled closed models will continue to dominate for more general purpose use-case like codegen, search, etc. but intelligence has lots of long-tail applications and i think for these longer-tail applications, finetuned custom models will rule
Product search model:
https://www.linkedin.com/posts/introducing-ontology-1-ugcPos...
Edit: more direct links, sorry:
https://onton.com/research/ontology-1
https://onton.com/research/ontology-1-benchmarks
this is really cool. i'm sure some of the larger e-commerce companies are already using clickstream data to tune better query rewriters/ranking models
There is so much opportunity for purpose built models like this. Ideally a harness should spin up a subagent to offload to targeted models for specific tasks like this. I know this is not a novel idea. Claude code does some of this by handing off the "explore" agent work to haiku. I just love seeing that specialized LLMs are being developed.
> Claude code does some of this by handing off the "explore" agent work to haiku.
That is not handing off to a specialized model, its just handing off to a lighter and interior model (compared to the parent model). That by itself can create issues like the lighter model not capturing all the data that the parent needs.
The idea is that we get specialized models that are better then general purpose models. But its rare for a specialized model to beat a strong general model.
There is a reason why we hear less about this idea of smaller expert models, because large strong models to the tasks just as good.
And if the tasks is repetitive to the point that specialization is useful, you can get into a situation that your better off having a program written for that reputative nature, then delegating to other models. And then have the main strong model, deal with the (semi)cleaned up data.
> There is a reason why we hear less about this idea of smaller expert models, because large strong models to the tasks just as good.
Smaller models are cheaper, sometimes faster. I agree that the “we’re an LLM fine-tuned for X” hasn’t worked out because you can just train Claude to do X (and Anthropic will), but not burning Opus/Fable tokens on dumb-but-token-heavy tasks is good sense.
As we move from “integrate AI into Y” to “optimize the ROI on Y”, we’ll see more of this.
castform founder here. while it's "rare for a specialized model to beat a strong general model" today, i think the tech/knowhow on how to do so is getting there. we see some early signs of this in industry e.g.
https://shopify.engineering/sidekicks-continual-learning-loo... https://thinkingmachines.ai/news/learning-to-replicate-exper... https://castform.com/blog/elsa/
at the end of the day, models are only as good as the data they're trained on. and if one has access to proprietary data, it should yield specialized models that do better than larger general models
You can register models with mcp. I think it’s an expensive solution, but it is available in the framework. I use a light weight bus protocol that lets agents interact and pass short messages with pointers. It’s very efficient.
There are! Chroma has Context1, SID has SID-1, and you'd actually be surprised at how easy it is to post-train your own with pretty good pass@ recall@ ndcg@ etc.
There's also Hornet who have shared some interesting talks & blogs lately. I don't know that I'd exclusively use agents for retrieval the way Neon outlines here as well. I think distillation similar to what ZeroEntropy has done for bespoke retrieval & reranking with _some_ agent manipulation on top-k results works better (IME).
> Claude code does some of this by handing off the "explore" agent work to haiku
This is no longer necessarily true. As of 2.1.198 [0] (released July 1st): "The built-in Explore agent now inherits the main session’s model (capped at opus) instead of running on haiku"
[0] https://code.claude.com/docs/en/changelog#2-1-198
I feel like the future is people building applications with tightly integrated LLMs that work hand in hand with the application's own lifecycle and code.
I also didn't realize that people were using agentic harnesses for search, it's an interesting idea. If the context length is short enough it should be fairly cheap compared to running "normal" agentic coding workloads where you have O(100k) context length for doing almost anything.
castform founder here. that's a future we are really excited about too :) ideally, you can post-train the llm within the application itself, as it's being used. both interesting infrastructure & algorithmic challenges here
There has been an over-obsession with frontier models and benchmarks. Most of the work will be done by task specific models. You don't put Phds on the factory floor.
Models keep on improving though, so doesn't fine tuning become an ongoing task with ongoing maintenance burden?
(one of the blog post authors here) -> once you set up a finetuning pipeline, it's often trivial to rerun it on top of a new open weights model. so, it's orthogonal to base model improvements
this is exactly what leopold talks about in situational awareness
yes, and this is why we need model routing
castform founder here: totally! we also think model routing is also a post-training problem i.e. getting a model to predict the difficulty of a task and match it to the right model -> we're gonna be sharing more on that soon :)
> There is so much opportunity for purpose built models like this.
OpenAI etc could themselves do this, and maybe they already do? Where the public-facing interface delegates to multiple little goblins behinds the scenes
castform founder here. openai actually deprecated their finetuning apis a few months back weirdly.
Exactly. There could be a lot of value for inference companies to do this. Could save a lot of money being able to hand off highly repetitive known tasks to far smaller specialized models.
There is a more serious question in here that's not being answered. How effective is the retrieval in finding buried needles in larger and larger haystacks. And there's a correlary question, how effective could you be in finding paired needles in that haystack where you need to hold a needle to unlock finding another needle.
Considering the state of the field ( RAG/retrieval/evaluation) I have 0 trust in it, even more if it's closed source with bullshit claim like that.
Everything is vibe sloped to death, and dead after a few months to a couple of years (and not hard to be 100 cheaper than GPT-5.6 sol ... DS is basically free and I guess already 100 times cheaper or more, and here another slope ).
castform founder here. we should have made it more prominent on the blog but here's the full code example: https://github.com/castform-ai/benchmax/tree/main/examples/n...
I use detailed project files. It has data regarding the project and subtasks as well as task status. It doesn’t depend on agent context and it’s managed to keep the agent on track. Feature creep with the new models is a very real issue. Capturing principles and how to reconcile tasks helps too. Even today it brought up a source of truth issue it had detected. There were multiple authorities born out of a patch and it used that principle to highlight and resolve the problem.
This feels like the database equivalent of "use the right data structure." We've spent two years assuming the biggest general-purpose model should do everything. It makes more sense for retrieval, reranking, reasoning, and generation to each have their own optimized model if the routing cost is negligible.
Nice, but there's no mention of how Luna or DSFlash perform on the same task? (Being 25x and 50x cheaper respectively.)
Nor of how much faster their custom model performs?
I have done my own testing and found that smaller models can beat their larger siblings on fact retrieval from documents. I haven’t investigated it in depth with a large enough dataset but my guess is that larger models overthink it while smaller ones just do it. I would like if they compared this with 5.6 Luna instead.
we actually have the test benchmark against luna too! it's just not in our title but you can see it in the first diagram below the title. luna does pretty well tbh but sol is just a tad bit better. but luna is way cheaper.
if you want to dive down into the various traces of the benchmark, you can check this out: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...
- founder of castform
Anecdotally, it feels like Opus, Fable, and Sol "get distracted" when you use them for writing code. Great at reasoning and coordination but they will go off on a tangent and refactor half the code base. I only use them for reasoning (of course) and coordinating subagents.
Just an anecdote but thats why Deepseek v4 flash 0731 is my current favorite model. It's really not very "eager" and stays on the task at hand.
Have been feeling the same. There's a sweet spot that threads the needle between "too dumb to search the right thing / relay the correct results" and "too smart to just stop overthinking and just report the damn thing"
Any data or public links you can share? That surprises me
On what do you guys test the model. Its very dubious that there is no common retrieval benchmark such as browsecomp plus or similar tested. And what metric do you report?
(founder of castform here) - we didn't get to dive too deep into the dataset we were using for the retrieval in the blogpost for brevity, but we did link the training run (which shows the dataset) here: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...
the page shows the exact trace of all the models we are comparing against and the aggregate scores
we generated the question & answer pair from gitlab product handbook (https://handbook.gitlab.com/) since the point is to show that you can generate training questions from raw data corpus (something a company already has today)
Keeping track of any AI progress is becoming harder by the day, because there's ambiguity around common/clear/consistent benchmarks. Everything is constantly skewed into favourable directions.
TBF gaming benchmarks is not something new to AI
I am not sure about GPT-5.6. It usually 10x more verbose for no apparent reason than GPT-5.5. Maybe it is only me.
Have you tried Claude? 5.6 feels less verbose, and less messy to me.
Yeah Opus 4.8 / GPT 5.5 what I use. Fable is okish, the coding experience is a bit weird with it.
it is definitely more verbose.
Bit unrelated, I realized that z.ai gives you access to deepseek 4 flash. It's incredible how well it performs when given a detailed spec. I'm not sure I've seen a model one-shot like that, and I was already impressed by gemma 4's speed and efficiency.
Deepseek flash (especially after the recent update) has to be one of the most slept-on models. Price-performance is ridiculous, and its available on a number of cheap coding subscriptions
Flash is the most used model in the world since last week
Maybe, but a specific example showing how to do it would have been a more compelling argument.
(castform founder here) we should have made it more prominent on the blogpost but here's the github repo: https://github.com/castform-ai/benchmax/tree/main/examples/n...
One thing that plagues [insert current FAANG] is the large amount of corpus knowledge that is outdated/misleading or just plain wrong. I'm curious how this addresses that if it's deriving the reward function from the corpus itself.
(founder of castform here!) - having worked at FAANG / big tech, i totally get this. our example was on gitlab's open source company handbook but i think a real company's corpus is way more messy and has many sources of truth.
a few ideas i have yet to validate are: - prioritize recently updated docs when generating the training questions (assumption those docs are more correct than others) - actually including contradicting documents that talks about the exact same topic might be a good training example - ideally the model should surface all the relevant info it can find, and explain what it has found. (usually contradiction comes from the fact that the later document is the updated stance) - you could also mine high quality Q&A from public slack / communication channels where questions were asked and someone else in the team linked some docs / answer. those are strongly validated "ground truth" answers