While attempting to use Fable 5.2, I found my harness (Juggler 0.9.7) was frequently compacting the conversation and reporting that it used more tokens than were necessary. My 5-hr budget was instantly used up and it chewed through quite a lot of Claude credits. Hopefully nobody else encountered this problem.
> As an experiment, one Anthropic engineer gave Fable 5.2 the task of designing a prototype operating system. Starting from an empty folder, the model coordinated over 1,000 subagents, working around the clock for eight days writing, testing, and reviewing code. It produced a working operating system that boots on a laptop, with its own core that runs the machine, drivers for the disk, keyboard, and network, permanent file storage, windows, a login screen, an installer, and built-in Claude functionality (see the video below).
As impressive as this sounds, whether it is tech teams or agent farms, the progress slows down considerably as the codebase grows. The real pain was seldom about shipping v1, but was more about how the v8 scales and how easy it is to get there, how robust the system remains between versions, and how constraints are held.
A huge part of the challenge is to pick/choose the design and implementation strategy that can only come from experience. That is one of the primary reasons why it's an absurd advice to me to let the llm do the goalseek and freestyle. This may be a no big deal on microservice or any managed runtime software that is an update away, but on an embedded or a high availability system where the field upgrade and hardware replacement costs could bankrupt a business.
There's been some discussion around Shopify (and others) move from React Native to purely native and that leading some people to conclude that LLMs have effectively killed RN. Which has always seemed silly to me for 2 reasons: First, Shopify almost certainly has on staff (or access to) actual native devs to steer that ship that a small/medium-sized company may not have. But more importantly, it's not the "Translate this RN app into native" that's the trouble. It's the hundreds and thousands of additional iterations you'll have to do on top to maintain and build out that product.
It's hard keeping separate code bases with separate languages in tandem, and while LLMs definitely aid in that area, I can't imagine it's a free lunch.
They also seem to be ignoring the realities of modern app store reviews. Apple and Google are only getting slower as vibe-coded apps flood their marketplaces with submissions. Maybe Shopify has a special relationship that gets them faster reviews, I don't know. But if not, the ability to put out updates, rollbacks, and hotfixes without being gatekept is a critical feature. Especially with today's security landscape. React Native now seems more compelling than ever before.
From what I heard, updates for many established developers, even individuals, go through within a few hours.
I'd also love the ability to take back control over my rollouts, but with Apple's recent crackdown on remote controlled feature flags in response to fraudulent nudification apps, I'd be very careful about this.
Rumors say Apple now uses heuristics to detect feature flags in the binary. While that's not directly relevant to remotely delivered JS, I fear that bypassing App Review in this way could become a quick ticket to account termination.
While this unattended example is impressive, this is why you need to guide things more closely and make sure the design is itself scalable and maintainable. Otherwise you can get wad-o-slop.
Same thing happens in many cases with junior or mid-tier programmers slung at a project without supervision.
> While this unattended example is impressive, this is why you need to guide things more closely and make sure the design is itself scalable and maintainable. Otherwise you can get wad-o-slop.
Indeed, I'm curious how much better/worse one human with "one agent" could do that same functionality in a OS from scratch. Last time I did this exercise, I took Cursor's shitty attempt from "millions of lines of source code" to about ~20K (https://emsh.cat/en/one-human-one-agent-one-browser/), and seemingly one of my conclusions still hold true ~1 year later:
> One human using one agent seems far more effective than one human using thousands of agents
This is something I've been thinking about more and more. Since Opus 5.5 came out, I feel we've crested the hill and the answer for "who's going to maintain this slop going forward" is obvious - the LLMs will. Whether or not its slop, is extensible, maintainable, or forward looking almost doesn't matter anymore. Does it solve the problem today? If so ship it, and use the agents to solve the future problems. I am not quite sure this is a good idea, but assuming LLM progress doesn't stop means its probably the bet I would take.
I watch a YouTuber who is homesteading and builds everything. Often times I see him doing things in ways I know will break or fail eventually, but it doesn't matter because he is there and will fix it when it does. You could argue this is a waste of energy, and for a human it might be. For an LLM however I do not think it is.
This makes sense in a way; why design for potential future growth in an environment of fast-paced technological and business development, bringing greater risk?
It seems the problem is the solution, and vice versa.
I’ve had a chance to use fable 5.2. Some observations:
- Anthropic is finally well balanced across models. Meaning: Opus 5.5 is so good that I do not feel Fable FOMO as I did when it was Fable 5/5.1 vs Opus 4.8/5. And Sonnet 5.5 sometimes makes me forget about Opus.
- Fable is good, but you really have to push the ambition of your prompt in order to see proper separation from Opus 5.5. I personally have a hard time telling the difference between Fable 5.2 and Opus 5.5 for ordinary coding tasks.
- It’s way faster and more reliable: one of the worst things about fable 5.1 was how slow it was an, on top of that, how frequently it would suffer from max output token overflows. I got none of these issues with 5.2.
- Fable is exceptional at drawing. It’s ability to render an image with JavaScript actually has me questioning the value of dedicated image models. I prefer the images I’ve gotten fable/opus to make for me over those from gpt-image, it’s wild to think we’re at this point.
> Fable is good, but you really have to push the ambition of your prompt in order to see proper separation from Opus 5.5.
I love seeing this verbalized. We are pretty much at the point that increased model capability will only have utility to folks that are willing to let go of even more understanding of whatever the model is producing.
The best models can/will produce a year's worth of output that would take several days to weeks to properly understand in the span of an afternoon.
For everyone that still wants to feel like they own the end result, just use the lesser models with less ambitious tasks.
Caveat: I think this mainly applies to coding. I would still like the best models possible to be working on e.g. new drug research to cure our various ailments.
> This is not a claim about the future. Perhaps almost all or perhaps even all of the software skills I have will become irrelevant in the near future. But we're not there yet. At the moment, programming skills seem higher leverage than ever before.
> There's something a bit funny about the current moment, where it seems like having the skills that models are replacing makes one much more effective than people who are trying to use models without having those skills. I like Max Bittker's framing that models are very good at counterfeiting and the level of expertise it takes to distinguish a counterfeit from the real thing has been increasing as models improve.
As well as this section fairly at the start:
> As far as I can tell, these problems are problems because people are implicitly carrying around a pre-agentic cost model. Ten years ago, the kind of person who implemented a new compiler because their compiler was slow was the kind of person who started programming a new 3d game engine when they wanted to make an indie game, i.e., the kind of person who never ships anything. Most productive programmers I know still have an aversion to this kind of thing, even though it is now actually reasonable to implement this kind of thing now.
Of course, at the moment everything’s still a bit very much difficult to tell, but “productivity”, at least in applicable areas of work, might really come down to who can let go of the most understanding the easiest? Incredibly thought-provoking stuff.
> This paper suggests that the increased interest in human factors among engineers reflects the irony that the more advanced a control system is, so the more crucial may be the contribution of the human operator
Does it? Couldn’t you steelman this argument and claim the opposite, that the more advanced a control system, the more likely it is that human error is what causes its malfunction?
Those don't seem like mutually exclusive statements to me. In fact, they feel like two sides of the same coin. It's almost by definition that more advanced control systems are exactly the ones where tractable sources of error (mechanical, environmental, etc) have been eliminated, controlled or otherwise accounted for as much as possible, to the point that the only remaining likely problems come from where the system interfaces with humans in some way.
Which makes those interfaces the most critical parts of the system.
> but “productivity”, at least in applicable areas of work, might really come down to who can let go of the most understanding the easiest?
That is, genuinely, crazy talk. That is what addicts would say.
But it is intellectual-washed to sound like the most reasonable thing to say.
It is not. And there is nothing thought-provoking about the mere act of doing heroin.
There might be something in the system and mechanics that lead to people getting the idea to do that, though.
___
We can engage with these ideas and there might be some merit to them, but not with this frame. This frame will (not hyperbole) kill us all.
People will die due to (poorly-engineered) systems understood by no one, failing in completely incoherent and unexpected ways - and those people may be you or your loved ones.
> That is, genuinely, crazy talk. That is what addicts would say.
I, well, agree. Of course the statement is nuts, especially so once you consider issues like vendor lock-in, the oncoming atrophy of one’s own skills, etc. But if you define “productivity” as short- to medium-term results only, is it really that inaccurate?
If you define a problem in a way that leaves out like 80% of the constraints, then solutions get trivial, yeah. That's I think the generalized way of phrasing this specific dysfunction of our industry.
The problem is that it still uses the and benefits of the legitimacy of the word of "productivity" there. Meaning that it is harmful to us to put it that way.
Good point, not having to plan for the long term probably has an outsized effect on a given level of difficulty/effort required.
Maybe it’d be interesting to see what could be achieved responsibly by motivated people with models in a world without “feature pressure”/competitive stress (which’d include e.g. no security race and infinite attackers and exploits to keep up with)?
>>> but “productivity”, at least in applicable areas of work, might really come down to who can let go of the most understanding the easiest?
>That is, genuinely, crazy talk. That is what addicts would say.
People demonstrably said the same thing about the inventions of... Writing! and Mechanical/Digital Calculators! Yet here we are!
>> Ten years ago, the kind of person who implemented a new compiler because their compiler was slow was the kind of person who started programming a new 3d game engine when they wanted to make an indie game, i.e., the kind of person who never ships anything. Most productive programmers I know still have an aversion to this kind of thing, even though it is now actually reasonable to implement this kind of thing now.
Then there is these other programmers who built their own operating systems even though their OS professors are claiming publicly their design sucks, who built a local database from scratch in C because his use case did not need the client/server stuff but just all the other relational goodies, or who built video streaming/transcoding software from scratch just because they have ideas and then the world changes around their ambitions.
I think it is not enough to be ambitious or have the technical ability if you want to build something for others to use. You definitely need deep commitment but also perseverance in the face of social adversity. I think this is what saddens people with automation of the technical aspects of programming. It leaves the more psychological aspects as challenges which are not easy to face all the time.
> It leaves the more psychological aspects as challenges which are not easy to face all the time.
Great point. Coding used to be a natural forcing function for prioritization of features. When coding is basically “free” you have to do the harder job of saying no to things, deciding exactly what to build, etc…
> who built their own operating systems even though their OS professors are claiming publicly their design sucks, who built a local database from scratch in C because his use case did not need the client/server stuff but just all the other relational goodies, or who built video streaming/transcoding software from scratch just because they have ideas and then the world changes around their ambitions
> perseverance in the face of social adversity
Sometimes you're Fabrice Bellard, sometimes you're Terry Davis. That is, for almost any idea you'll find random detractors, but some ideas really are generally useful and some are completely nuts. One of the negative externalities of AI is that bystanders are confronted with a lot more cases where they have to work out which one this thing is.
“might really come down to who can let go of the most understanding the easiest”
I suppose it depends on the “applicable areas of work” but I would argue the opposite. As in all things, when you level the playing field, in order to stand out you have to know the rules to understand how to break them.
Imagine a fully autonomous Ad agency that creates unique ad campaigns for companies. And have it controlled by someone who “let go of the most understanding.” For instance, maybe a business minded person with excellent technical skills, but doesn’t know how to copy-write and has no visual design training.
Even with the strides in AI advancement, this is an impossibility and I’d argue it won’t happen within the next five years even. You need human agency, and you need human workers who are experts in their field.
> Perhaps almost all or perhaps even all of the software skills I have will become irrelevant in the near future. But we're not there yet. At the moment, programming skills seem higher leverage than ever before.
While this iteration of technology certainly _feels_ different, it seems like for now we're still coming down to "it's useful to know how to build a car, but when you need to put food on the table, know how to fix it and make it do what you need it to do".
Well, in retrospect I don’t like this framing all that much either, but I think I came to this conclusion due to assuming
not caring about the output ~ more output
… which may or may not be useful in the long term, but definitely does help a shorter-term “productivity” measure.
On the other hand though, there’s a pretty damning example against my case here: Say a manager of individual contributors – which is what the situation of an individual with sufficiently smart models and tooling may almost boil down to – didn’t directly care about the output anymore. We’d most likely call that being unhinged/unproductive, too…
It was late at night and I primarily wanted to link the somewhat relevant article, maybe I should have left everyone to reach their own conclusions instead of adding some poorly-worded unfinished thoughts. :x
> Of course, at the moment everything’s still a bit very much difficult to tell, but “productivity”, at least in applicable areas of work, might really come down to who can let go of the most understanding the easiest? Incredibly thought-provoking stuff.
I cannot see it being a good thing to "let go" of understanding. If you don't understanding anything, then why would anyone keep you around? An example hypothetical situation is, if I have a colleague that outputs a bunch of productive things, but they don't understand it, and I do, then of what use are they? I have LLMs too and also have the understanding.
It isn't clear to me why people are so comfortable with letting go of understanding. And it's a bit bizarre because it always seems to be framed as "you're going to get left behind if you care about understanding". I also never hear or read anyone who has any real and deep experience say these things.
My read of what is being suggested is that the level at which one should aim to understand and be comfortable with is slightly more abstract than reading the code and reviewing every line of it. It is like how one is comfortable not understanding the byte-level code or assembly language code. I know that this analogy doesn’t hold very well because compilation is a deterministic process while llms are not, but that is precisely the point being made. “Trust” that llm will produce the right output, even though the output might not be reproducible word for word the next time you give the same prompt.
To your point about becoming redundant without understanding, not understanding things will be a problem when an issue arises and the other person isnt able to handle it the right way. However, if llm can address the issue, does the fact that they did not understand the details really matter? From the perspective of a person who gets to decide who gets fired, would they fire someone who is productive but sometimes introduces bugs that they quickly address or would they fire someone who is comparatively slow but rarely introduces bugs? The answer probably depends on the domain, and how costly a bug can be. For most of us, the work is not so critical that you cannot afford bugs. So, productivity wins.
If you are suggesting humans before LLMs rarely introduced bugs, you probably haven’t got the latest models to review your old code. It’s shocking how many bugs we had before but couldn’t see!
Don't shoot the messenger, but maybe the idea is this:
You're smart, and you know what can go wrong, so you use models slowly, and you check their work. Your bet is that it will make a bad mistake.
Your coworker is reckless, they don't know or care what can go wrong, so they use models quickly. Their bet is that the model won't make a bad mistake.
If the model doesn't make a bad mistake, your coworker is creating more value to your employer, and your employer doesn't really care if you understand the code. They just want something that works.
Over the last year, it sure seems like models are making less mistakes, so the expectations on the bets are changing too.
Right or wrong is this the future we want? Shouldn’t we say and do something?
For instance: we have calculators. We’ve had them for decades in their “current” form more or less. They are readily accessible and far more reliable than LLM’s - if you a problem 1000 times it’ll be right 1000 times, the rare bug or glitch notwithstanding. But we still teach people basic arithmetic. We teach times tables. We didn’t just offload basic math.
I agree we don’t need to know assembly to use a computer but there is a balance here and companies do not care about a balance.
I think we mostly DID offload basic math, for better or worse.
For software engineering I don't know the right balance.
I've used the "we mostly stopped caring about machine code generated by a compiler; maybe we should stop caring about LLM-generated code as well?" analogy myself.
But even I have to admit there's two big holes in that analogy: if I use a compiler to turn C into machine code... (a) compiler bugs are orders of magnitude rarer than LLM-generated bugs (b) my C code remains the source of truth, or at least source of intent. Whereas with LLM-generated vibecode, there's no source of truth, just an ever-evolving mudball. I (kinda?) like the ideas of "documentation as source of truth" and/or "tests as source of truth" ideas, but I don't think we're nearly at the point of viability.
TL;DR yeah I think actual understanding of what LLMs produce is gonna be important for the forseeable future
Adults can, under most circumstances, do some limited slow arithmetic but are generally piss-poor at it, notwithstanding the occasional rainman, and only rely on it in trivial matters.
You see cashiers summing groceries? Salary calculations on paper? Tax, inventory, accounting, you name it? Nobody does any of it by hand.
I didn’t say we were universally good at it but in many countries we are (more or less) all exposed to it and taught it in school. There is a baseline even if we are far from mastery, and frankly a cynical hand wave of “well everyone’s bad at it” is not a real argument. In the US it is very clear that people are at least exposed to it and if we weren’t we would be operating very differently. I went to dinner last night, everyone did the tips in their head right there at the table. The average person can do some simple multiplication and division in their head. It’s not a mysterious script done on a box that they repeat the output of. Most people even if they’re bad at it understand the fundamental concept and, gun to their head, could probably do a series of very basic math problems with pencil and paper.
I imagine you and most of your peers can calculate a tip or at least approximate it without pulling out their phones yes?
This is just moving to a higher level of abstraction. Nowadays people don't verify the compiler generated assembly code anymore. We don't care if our source generates a few million extra instructions. Source code is just an intermediate between specs and binary at an abstraction level decided by the developer. We do requirements -> code -> assembly.
To resolve this we should aim for a new intermediate format that is more high level than source code, the generated code is too monotonic to properly review manually. Something like requirements -> detailed specs for llm -> code -> assembly . Then let one agent implement the specs and have another review it. If we stay at the current abstraction level, humans become the major bottleneck.
I think your hypothetical is incomplete.
What if the person that lets go, generates more productive things then somebody understands?
In this case I know a lot of ppl will value the producer of value more.
I always pushed the bounds of learning more about the underlying stack or problem, and always found them in no time.
Understanding architecture, robustness, the “crumple zones” of a system? I’m all for it.
Taking three days to “understand” how postgres transactions work under the hood for your generic SaaS? Uh..
I have seen a lot(!) of the last type of “understanding” and it’s infuriating. The same types will often not understand the business, its stakeholders, the architectural constraints beyond the technical, etc. Understanding takes many forms.
I feel like the real value is in probing the extents of your own understanding. If you don't understand something it's doing, you can ask. You do kind of move up a level of abstraction, from the IC to a PM or team lead. So maybe you don't know what every line of code does but if you understand code and how to drive you can trust that it's doing the right thing and you don't have to be as deep in the weeds of every refactor etc.
You do have to keep a sense of when it did the wrong thing and how that was a valid interpretation of what you asked and how to better specify what you want. But that's how it really just shows you the gaps in your own understanding of a problem.
Also I think there's huge differences to the kinds of tasks and workloads people are throwing at it, leading to wildly different perspectives, but everyone's talking about the same "Fable".
> Also I think there's huge differences to the kinds of tasks and workloads people are throwing at it, leading to wildly different perspectives, but everyone's talking about the same "Fable".
This right here. Humans are really prone to over indexing on a handful of personal experiences vs a representative sample. And once someone has formed an opinion, confirmation bias helps cement it, despite evidence to the contrary. Reality is probably somewhere in the middle.
That's where I think benchmarks are a huge red herring. Everyone wants a simple performance evaluation of a repetitive task but that's missing the point entirely IMO. Just try talking to it, asking questions. I spend so much time just reading responses. I have to like really think about how to specify the question I'm asking, fire it off, and I often can't wait to see what it says and what I'll learn. How anyone thinks it's making us dumber is crazy to me...you're missing out if you're just trying to offload thinking.
You're misinterpreting what they mean as understanding and inserting your own doomer definition where this imagined person has been stripped of all ability to investigate a result or ask questions.
I too love to downvote and edit instead of admitting that I'm talking and acting in a way where nobody would want to be my coworker anyways. Where exactly are you demonstrating that you're doing anything to justify being kept around if that's such a big concern for you?
> I like Max Bittker's framing that models are very good at counterfeiting and the level of expertise it takes to distinguish a counterfeit from the real thing has been increasing as models improve.
Is it just me or is this analogy completely missing the point of why we care about detecting counterfeits in the first place? Poor craftsmanship of all things is not really the reason counterfeiting is illegal!
I'm struggling to understand the hypocrisy of private property advocates advocating against private property. Are we doing "intellectual property is protected by law" or not? We're allowed to ingest all of human FOSS and StackOverflow without attribution and use it to sell distilled human knowledge by the token, but not use the distilled human knowledge to reverse engineer and FOSSify proprietary software? We're allowed to ingest all the art on Twitter and use it to generate and sell corporate clip art without attribution, but we're not allowed to generate micky mouse/bugs bunny yaoi fanart?
I mean, the answer is obvious: the Luddites were right, these tools (not just LLMs but private property, copyright, and capitalist mechanisms) are only tolerated to exist so long as they serve the needs of Capital, but the hypocrisy still frustrates me.
> I'm struggling to understand the hypocrisy of private property advocates advocating against private property. Are we doing "intellectual property is protected by law" or not?
The most rigorous "private property advocates" don't recognize intellectual property as a legitimate form of property in the first place: private property is a natural phenomenon that emerges as a solution to the problem of physical things being inherently exclusive in possession and use, and IP is a creation of positive law meant to artificially apply the methods for solving that problem to another, unrelated question that isn't part of the same problem domain.
> I mean, the answer is obvious: the Luddites were right, these tools (not just LLMs but private property, copyright, and capitalist mechanisms) are only tolerated to exist so long as they serve the needs of Capital, but the hypocrisy still frustrates me.
Abstract concepts don't have "needs", and no one is currently in a position to tolerate or not tolerate anything -- no one in particular is actually in control over anyone else's activities. The cat's out of the bag, and no one has any capacity to put it back in.
> these tools […] are only tolerated to exist so long as they serve the needs of Capital, but the hypocrisy still frustrates me.
Great thought. Question is, can those who do the tolerating (in the interest of capital) even effectively prohibit/prevent access after a certain point, especially now that the cat is not just underway, but well out of the bag?
Assuming the current trajectory of a yearly factor of 10–100 in size/cost reduction [0] holds up, won’t the technology just become cheap enough to be functionally available to everyone?
My assumption is that frontier models will be constantly increasing in intelligence, solving far more difficult problems, while the rest of us do the LLM equivalent of scratching through the dirt with wooden hoes.
Thinking about just the domain of the stock market only: getting a "win" on it means getting a leg up on everyone else participating. Best example I can think of of a pure zero sum intelligence contest. Programming UIs might have a point where you just can't get smarter at it, but in an intelligence contest like a stock market, I don't think there's a ceiling.
In "Accelerando" this was covered by the invention of "Economics 2.0." AI agents also got into a lot of lawfare in that book.
You will find a lot of articles from before LLMs about not delegating understanding of your core system or business. You could say that the software is not your core business so you don't have to understand it. But if the software is any good then it should be taking care of a large part of the business. So software is your business and you shouldn't delegate understanding it.
I like this and the warning it entails, on the flip side of this I have found that you need to be able to delegate and trust. If you demand on understanding everything and doing everything yourself the world will grind you down. It's a big place and sometimes it is OK to delegate what is not part of your core business it let's you focus on this. Not everybody needs to be able to cook 5-star meals, you should know how to heat up some pasta, but don't beat yourself up if you can't bake a cheese soufflé.
If your objective was understanding rather than, say, getting a new operating system running, couldn't you use the stronger models to produce artifacts and then ask them (or weaker models) to explain those artifacts to you? Probably the stronger models would be able to produce artifacts that were more worthwhile to study.
Maggie Appleton suggests some ways that AI models could do a better job of communicating about artifacts to people in https://maggieappleton.com/planning-agents, although she's focused on the use case of the person deciding what changes in the artifact to request from the model.
Last time I submitted a link it was immediately flagged to death, although I have to use an Incognito mode tab to see that. I'm not wasting any more of my time on that.
That's the one, and it's one third of all the links I've ever posted, or, probably, will ever post. What you say may be true, but it was still immediately flagged to death, regardless of how that happened, and the HN software is still trying to be sneaky about it. If my contributions are treated as spam, I'm not going to persist.
Certain domains being blacklisted is completely understandable if you consider the history of the site and culture(s) of the users. It's nothing to do with it being "your contribution".
That could be handled with an error message instead of assuming bad faith on my part. Given that it was handled (and continues to be handled) with sneaky shadowbanning measures, I have no interest in attempting to circumvent the spam filter by submitting future links.
“Wasting time” by copy-pasting link title and url? Sounds excessively optimized time management to me. Save 10 minutes elsewhere, and you got 50+ submissions worth to spend!
Most submissions never receive any upvotes. They’re all “wasted” like that, even if not flagged.
this is about cables which often were and still are snake oil in the audiophile world. headphones do make a difference. I have a hard time using my airpods and others after experiencing a pair of Sennheiser HD650 I've gotten myself as a christmas present couple years ago
yeah lamp cable will work. There's no room factor in a speaker so like 99 percent of the quality of the sound comes from the fidelity of the transducer and its resonance characteristics in electrical/physical suspension. Then the next .9 percent is from the source equipment, and finally the last .1 percent is from the dac/amp stuff. Exaggeration here but not really
Ok this point and pacing the frontier: personally i would be ok with freezing models at this point. Opus 5.5 does everything i need and i suspect that from this point onwards the improvements will be beyond my understanding and just introduce existential risk.
It very much applies to science as well. I've got a workflow doing an entire PhD's worth of work time in a single month subscription of $100 codex. There's zero chance i go through and understand what the hell it's doing, but for tackling problems which are accepted as "impossible" it's extremely useful for gathering data on every possible approach. It hasn't had a break through, but it's certainly recreated everything anyone's tried before.
There are many ambitious things I have tried to work with Opus 4.8 and Opus 5.5 on, and it just can't get there. It always falls back to some tried and true method or some valley that it gets stuck in. In those times, it was impossible to push it forward into more creative and exploratory thinking based upon my relatively advanced pushes and contextualizing. So I took the commenter's position that Fable 5.2 will respond better to that type of pushing. But that type of pushing actually means you know more than those who are happy enough using the lower models. So that seems like the opposite of what you said.
I'm closer to a Product Manager so I'm not sure how useful these examples will be for you.
One important constraint is this topic of intent (also referred to as alignment). Specifically, when a user requests a feature or change, there is MOUNTAINS of implied context and intent in their comment. Some cultural, religious, social, economic, base on work and personal experience, their use cases, their values, what they intend to get done, etc. Every time these models get smarter, they get a little better at understanding this context. It's combination of reasons. Some of it is parametric. Some is inferred. Some is about selecting the right data to examine and research. As these models get smarter, I spend less time time iterating and educating the model. They reach the intent faster and better.
From what I hear and experience, this is also true in the coding space. LLMs are basically another abstraction level on top of formal languages, which sit on top of compilers and machine code and etched instructions. Smaller and dumber models require a lot more hand holding to achieve the same results. They're less capable of inferring the gaps. Less capable of extrapolating potential negative outcomes of the change they're making. Less/less useful parametric knowledge. Less capable of selecting the right data to use to make the right decisions. They do what they're told.
This has been less of an issue for coders who religiously document, understand the codebase and architecture intimately, and demand to be kept in the loop on every change. So seniors often do not see the utility of smarter models. They know exactly what they want, how to ask, the context to provide, and how to assess if the change worked as expected. The utility is for the 99.9999% of the world which is not a senior developer.
While what you say mostly rings true to me, I don't see how it could result in the valleys of stuckness I was asking about.
However, your statement that there are only 8300 senior developers in the world does not seem correct to me. That seems like an unreasonably high bar for that title.
They are generally things I don't particularly want to share, as I view them as ideas close to my heart. The general pattern was what felt like redirecting from the harder problems and an unwillingness, if we can call it that, to go there. In some sense though, it's hard to understand how these more advanced models get around that, because these models are tuned to the most probable answer. When you come at them with a novel idea, they always seem to water it down in the end.
I have also found this — once you get out into some really novel workspaces they will revert back to replication of adjacencies that already exist. Fable 5 and 5.1 in particular will talk around the issue in a way that no model prior seemed to.
I think the issue is fairly fundamental, perhaps because of tuning, perhaps even beyond what they are tuned or trained on, because the bulk of what they can go find and ground into seems to drag them heavily toward implicit biases that split around categories I can’t quite discern. The category boundaries sometimes look like academic fields, but sometimes they seem like more fundamental attractors.
I second (third?) this. Previous models were more willing to explore the unknown. Those new ones just try to Google search and latch onto any adjacent solution they find.
It's a reference to a joke, of a stereotypical comment left on Romantic Relationships topic online forums. As in, "this man is so terribly unsuitable to everyone. But I'm special, and I can fix him and make him the perfect boyfriend."
> We are pretty much at the point that increased model capability will only have utility to folks that are willing to let go of even more understanding of whatever the model is producing.
Given that reading someone else's code is slower than understanding your own code, doesn't that basically cover everyone that adopted coding agents?
> The best models can/will produce a year's worth of output that would take several days to weeks to properly understand in the span of an afternoon.
This is appalling. Are you really really going to trust a YEAR's output from a machine? Are you going to read and understand all that? Will you be able to maintain and extend it to your will? I truly doubt that. Drugs do harm and are hard to quit. Good luck.
As one poster replied above, there is code one doesn't need to understand if what you care about works. A good example is when I first tried a couple of local AI models on my highly unusual inference hardware.
It ran like a dog. Few nights of running Fable (later Opus 5.5) with a 7 strong team of DeepSeek v4.1 Flash 1017 (don't mistake it with 0731 that is crap) and llama was running 10x faster and vllm was running even faster when before it would not work at all.
I've run similarity and needle tests and the results are exactly the same content wise. But 10x faster. I've relied on this for a month now and I'm pretty happy with how it runs.
Do I now need to learn CUDA Kernel programming to understand how it was done? I don't think so. It's for my personal use. Not for a paying client so no warranty is perfectly fine.
But I also use a lot of AI at work in an I dusty where people may die if the code is buggy. So I absolutely verify and understand everything.
There is a middle ground too. Understanding the architecture.
For example I used a bunch of DeepSeek models supervised by Qwen to build adaptive-acl. A pi coding agent plugin that deterministicly checks if every tool used by an ai during an agentic session matches user intent to give permissions for those actions (reading/writing data, network access etc), as a part of this plugin I run an llm, a classifier that runs over user's prompts and extracts these permission grants then saving them in a format deterministic layer can process.
The design includes massive number of assumptions and compromises. For example, if a user allows writing this implies permission to delete and reading if what was written etc.
The thing also includes over 4 thousand actual instances of tool use preceded by "permissions talk" from my live sessions and that whole collection can be run to test for regression etc.
At some point during one night the AI had a regression and decided the best way to resolve it would be to add regex based language processing elements to the deterministic layer (a huge no no, this has to work regardless of the user speak English or Swahili, also regexing natural language is a disaster waiting to happen).
I knew enough about the architecture to notice that and burn it out with a hot iron.
That is the problem with AI. It sometimes solves problems the easiest possible way causing massive issues for later. So if one uses it for important stuff it is good to understand the architecture and the plan of what it is going to build at the very least.
I see this mindset a lot from people who I can only presume have only ever been individual contributors.
Having run teams, as much as I love getting into the details of code I care about, in the 30+ years I've been in this business, I've only ever even seen a tiny proportion of the code my teams rely on.
And the non-technical people in these businesses will have seen none of it.
They all take it on a combination of trust and verification of outcomes that the code does what the developers claim it does.
Increasingly we're getting to a point where the output from these machines are more trustworthy than an increasing proportion of human developers, and more importantly: Increasingly often good enough.
Especially as more and more of the code becomes throwaway.
I have a side-project that is profitable, and that is profitable only because I haven't needed to spend time reading the code. If the code turns out to be unmaintainable? Doesn't matter - it's so cheap code that it is reasonable to just throw it away, and start over based on the lessons learned. It enabled me to take an opportunity to profit I wouldn't have been able to profit from if I had to set aside time I didn't have, so even if it was to totally fail in the future as a result of issues with that code it doesn't matter.
As the cost of good enough code falls towards zero, how we think about which code matters and how much we need to understand it will fundamentally change.
Increasingly, a lot of code is even just part of a conversation with an agent to me. It's throwaway, for the duration of a session. Who cares if it lives on?
I don't trust or need to trust most of the code I build any more. I trust the measurable outputs. The verification pipelines we build. The feedback mechanisms we build. Those I need to read and understand.
That's no difference than when working with a team. The only thing that is different is failure rates. And at this point, for some thing the machine failure rates are still higher, for others it is lower.
> Fable is exceptional at drawing. It’s ability to render an image with JavaScript actually has me questioning the value of dedicated image models
I’ve been shifting away from using more traditional GenAI models in favor of just giving LLMs access to a custom drawing language (LOGO-like) to generate imagery and its been going shockingly well - Opus 5.5 in particular.
I put together a simple LLM driven skill that creates old adventure game (EGA) environments from simple pencil sketches and it works really well. And being procedural, they're much easier to make surgical changes to without degradation.
Just to clarify -- the images on that site are all generated from LLM-written code? It's very impressive if so. I would never have guessed these weren't human-made from the visuals alone.
Yeah it's entirely LLM driven! If you look at the repo, I explain more about how to use it. Basically, you give an agentic coding harness capable of tool usage (Opus 5.5, Qwen 3.8-27b, etc) the skill which explains the drawing system: simple things like how to set the colors, position rectangles, draw polygons, etc. and it uses p5.js to render them.
It has three different modes:
• You can simply describe a picture, and it'll automatically generate an image using procedural code.
• You can ask it to paint over an existing image. That's what those examples show, like me creating EGA-style equivalents of various scenes from the original Myst.
• The last way is to give it a rough pencil sketch. It's the mode I use the most since it's very valuable when building an adventure game to position objects in a room.
Thanks! I started this whole thing because I recalled reading an old interview with Sierra’s original creators, Ken and Roberta Williams, where they talked about how they needed a way to have graphics for their games without saving large bitmaps which would make it difficult to fit on a single floppy disk.
So instead for each room that needed to be rendered they just saved a series of vector instructions that would re-render it in real time. On slower machines, you could actually see the rooms pop in piece by piece. I wanted to kind of approximate that same aesthetic because it's just super cool!
Thanks for the rec! I unfortunately don't have a Speccy, but I did find an emulated version of the BBC Micro port of Twin Kingdom Valley and can definitely see it drawing in the various pieces of the environment.
I think the approach of using simple procedural vector graphics was relatively common back in the days when you had to optimize for either disk space or speed. Sierra's creators talked about how they opted for procedural graphics to fit their earliest games on the old 80 KB floppies.
So basically I gave an LLM access to the user manual for the original LOGO drawing language, along with several screenshots of various scenes from the original King's Quest I.
Then I just guided it to produce a procedural variant specialized in creating new EGA-style scenes for use in creating old-school point-and-click adventure game.
The speed part is funny. I felt like with Fable 5.1 I had time to digest what I had prompted and was waiting for. Now I feel like, I am running behind or like it is going to take off too far. This is great for closed prompts "make a soduko game" but hard with running context queries, because I feel like I am creating more information debt than I want. I'll be fine, just talking out loud.
Does Fable 5.2 outperform Opus for data science work?
Not just "iterate on this XGBoost model" but actually looking at a problem and designing and carrying out a short-term R&D program including probability modeling, model fitting diagnostics, how to respond to various kinds of results, how to interpret/reason about what it finds, etc. That's really where I saw all the models perform poorly, and where I saw the first big step increase from Opus 5 (pre-5.5) to Fable. But then Opus 5.5 came out and I haven't needed Fable since.
Mind you, I'm doing slightly different work (and I'm very much "in the loop" still) than when I performed that side-by-side test. But it's really amazing what Opus 5.5 can do. I now have a powerful data science AI assistant that not only can write any script I need it to (including some scripts I didn't even realize I needed), and it has an encyclopedic knowledge of everything ever, and it constantly checks me on oversights, it handily derives expressions and clever model reparameterizations that aid, it has great insight into model behavior, and it has an amazing ability come up with ad-hoc experiments and tests that would have taken me 2x as long to even think of (let alone implement by hand).
It's not at the scared-to-lose-my-job phase yet, but it's become a tremendous force multiplier for a smaller data team.
I'm teaching "Data Science" to computer science majors next spring, and I'm overwhelmed by the prospect of what to teach them in the age of AI. It seems like Fable already way outperforms anything I could historically expect my students to do at the end of the semester, and perhaps even outperforms me. I can describe a vague analysis on the data, and it will handle every step: transformation, preliminary analysis, formulating more precise questions, creating visualizations, running statistical tests, interpreting results, thinking about stakeholders... And in some of those steps, it finds things that I might not have considered.
I wish I had a bigger N so I could rely on that experience a bit more, but it sure does feel like I'm not really contributing anything in terms of "Data Science expertise" in my own research right now.
And especially where the fuzzy edges and blind spots are—the limitations of the methods, and the aspects of problems that are essential to the question but hard to quantify or test or reproduce.
I feel like a lot of people are going to put the same faith in their new quantitative superpowers as psychologists did in their 95% confidence intervals [0], and managers did in their “scientific management”/time-and-motion studies, and American political pollsters did in their 2016 presidential prognostications, and Rudder’s OKTrends blog (delightfully) did in calculating the answers to romance [1]…
If it wasn't for grinding all of the hard problems and hands-on projects and various programming tasks I had to do by hand over the years, I wouldn't be able to keep up with the agent, and wouldn't be able to get good results out of it, because I wouldn't know what it was doing and wouldn't be able to steer it appropriately. My schoolwork 15 years ago was more than formative, it remains critically foundational for my job work today.
Based on the other comment it sounds like maybe I need to give Fable another chance, and then I suppose I won't have to steer anything at all. But at the end of the day I am not paid to be a meat proxy, I am paid to deliver things for the business that employs me, and the things I deliver are my responsibility, not Anthropic's.
So I say, embrace AI as a learning tool, let students use it to bypass programming on assignments where the programming is of secondary importance, require oral defense/presentation for projects, make homework worth a much smaller fraction of final grade, and make all of your tests in-class and hand-written.
The reality is that the entire nature of education has to change to account for cheating becoming essentially free and virtually impossible to detect. Grinding through a problem set in the library is still tremendously important, but it's going to be harder to motivate students to do it without just cheating. Some students are conscientious and will do it anyway. Some won't. The hard part is going to be crafting curricula and exams that separate the two kinds of students. Fortunately you share that problem with just about every other professor in the world right now, so at least you shouldn't feel responsible for solving it all yourself.
Relevant anecdote: I used Wolfram Alpha to cheat on my calculus homework in freshman year, and then I got my butt kicked in the exam and a C final grade that never quite averaged its way out of my GPA. That felt horrible and it was a strong motivating force to get my shit together, but that's only because I'm a fucking nerd and I really cared about learning in school and making a career out of what I learned. On the other hand, I really was struggling with the material, which is why I tried cheating in the first place. Maybe if I had today's AI I could just ask it to help me learn better.
Plus it worries me that people will no longer know how to do anything basic and we will at some point in the future be slaves to AI if we aren't careful.
Is it reasonable to take up some of the class with statistical basics and intuition? E.g. you just referred to a bigger N; I don't know what level your students are but do they have a basic intuition for notions like "variance of estimator"? Or is that another class? I think my point is that even if they can nowadays press a button and have the most amazing reports and visualizations produced, that doesn't give anyone any actual understanding of the theoretical side of data science.
From my experience working with data scientists, it outperforms humans on an extreme level. I’ve seen so much bad data science, and here I have a model that can do all of that from one prompt. It is getting really good on finding caveats in data too.
I'm so glad I learnt it before LLMs. I can't imaging trying to get good at Pandas/Polars coding nowdays when so much of my analysis is "Fable, please do this for me, k thx bye"
I mean I found Fable 5.1 outperformed Opus 5.5 for data science, so I expect 5.2 to.
But then I'm someone who hasn't found Opus 5.5 good for much beyond web UIs and related frontend development. I find it more forgetful than GPT-5.6 or 6.1 or Fable, so despite having a 1M context window it gets lost in a big codebase or goes of target. Opus 5.5 doesn't manage agents particularly well IMO.
I'm surrounded by people raving about Opus 5.5 and God only knows why I'm not getting their results.
I just tried Fable 5.2 side-by-side with Opus 5.5 on the same prompt and I think you're right, n=1 but with strong signal.
Fable got right to the point, had cleaner/better responses and reasoning for its conclusions, got less distracted with secondary concerns, and still showed signs of the cleaner writing style compared to Opus, which improved a lot with 5.5 but can still be hard to parse sometimes.
I will keep using Opus for implementation tasks (e.g. building web apps to explore model predictions) but I think I'll try using Fable this week for the real DS stuff. Thanks!
Have never noticed a difference between Fable and Opus on coding tasks but design/architecture is night and day different. I usually don't bother with designs with Opus since it tends to screw things up, hallucinate, make bad decisions.
With (obra) superpowers, Fable writes a plan and delegates to Opus which works well. Fable is much better at sorting architecture in a conversation rather than me creating a giant prompt for Opus.
Opus 5.5 has been good enough for most of my coding, but I don’t know if Fable 5.2 is better for architecture work, like planning how a system fits together or designing the data model. @nijave says it’s night and day for design. Has anyone else seen that? (I don’t have access to Mythos, so I can only compare Fable and Opus.)
At this point, I don't really see much difference in output quality between Opus 5.5 and Fable 5.2. Although, Opus 5.5 seemed to just go on auto pilot mode without give tasks much thoughts unless you give it tasks specifically each time. Fable 5.2 seems to think more on its own.
And none of it matters ... because just as soon as you are happy with things "balanced across models" Anthropic will nerf the models you were happy with, and force you to use new and more expensive ones.
I just saw someone post about getting Claude to write code that renders individual brushstrokes and the results were varied and impressive. Can't prove they actually had Claude do it, but I believe it. Have been using Claude to edit videos and it can animate concepts without much prompting on my part at all.
Your comment matches what I measured on 30 regular coding tasks: Fable 5.2 and Opus 5.5 both solved 26 (Fable 5.1 was at 25). Fable 5.2 finished in 0.62x the time of Fable 5.1 on the same tasks, with no runs hitting a safeguards stop (Fable 5.1 hit a lot of safeguard stops).
Haiku - really only reason is lack of auto mode becomes tedious. I only use it indirectly with sub agent delegation for text analysis (summarize web page, document). However, it's quite fast and capable
When you put it like that, it almost resembles having a classroom of kids with different personalities and talents. Some are thoughtful, some creative, some quick in execution.
You choose wisely the path each takes and allocate tasks accordingly. Gives me chills to think of it...
Those distinctions make sense. I guess I'm more interested as to "when" you make that decision. What triggers you to spend the cognitive energy on deciding which model?
I have had the same experience between opus/fable and also drawing: it has done exceptionally well with SVG drawing in different graphic styles for app icons (comic, fantasy, modern, etc)
I clearly see the difference between Fable 5.1 and Opus 5.5 everyday.
All the people claiming "opus 5.5 is just as good" tends to make me think some people only do simple tasks or do not care about correctness or quality.
> All the people claiming "opus 5.5 is just as good" tends to make me think some people only do simple tasks or do not care about correctness or quality.
Have you considered the possibility of the problem being in the way you prompt a model and prepare it's context? I've been pushing myself to using mainly the cheapest models and I see them deliver the same results once we provide enough context or lay out a plan.
I'm pretty sure that's not the issue. For example I had one problem where Opus struggled for several weeks without getting us anywhere, flip-flopping in its decisions, generating huge amounts of irrelevant hacks. That obviously went through compaction several times, but I manually maintained explicit documentation and state of the project for it to work with, and reset the session as needed.
Fable made it work correctly in 2 days, it stayed on point, identified the real problems, and gave me clean code that was suitable to merge.
The task was simply to warm the hot path of an app so that latency was not as jittery for some rare, but important, events.
Fable built a reliable repeatable benchmark, disassembled the code to figure out how much could actually fit in L1, made tactical changes to eliminate bloat, and reliably achieved high coverage with minimal edits.
Opus just tried a bunch of random ideas without a plan, hoping to find something that sticks.
If this were true you could do a talk now and measure it and do it every week and see if it was still true or not and be the first to catch them in the act!
Tell it to draw a picture of something with JavaScript in the style you desire. No harness magic necessary. It helps to provide it with reference images.
> Starting from an empty folder, the model coordinated over 1,000 subagents, working around the clock for eight days writing, testing, and reviewing code. It produced a working operating system that boots on a laptop
This, to me, shows a problem I have with AI: it is reinforcing inequalities. This is literally brute-forcing an OS by throwing money out the window.
It does not sound AT ALL like a clever way to build an OS. But they can do it because they have orders of magnitudes more money to spend on that than people who would actually know how to build an OS.
The thing is, brute-forcing software may be the best solution to make money. So if you are rich and can brute force software in a dumb way, chances are that you can make a business out of it. If you're poor but know what you are doing, you cannot compete.
Technically, it is impressive that they can get a program to do that, but they are wasting money and energy doing it.
To me, this is a very pertinent take. Vibe coding large projects (even with SOTA models) reliably creates messes at a frequency too large to yield even moderately polished consumer products. It still needs human guidance.
But I'd call false dichotomy on this:
> brute-forcing software may be the best solution to make money. So if you are rich and can brute force software in a dumb way, chances are that you can make a business out of it. If you're poor but know what you are doing, you cannot compete.
Seems to me like the big winners will have know-how and some money to throw at AI.
"During prerelease testing they’ve [the models] helped our engineers complete otherwise-infeasible software migrations, designed three working electronic devices from scratch, solved two open problems in the physics of black holes, made substantial progress on robotics evaluations, and formalized the proof of Fermat’s Last Theorem in 30 hours"
Trying to read this with my 2021 brain. Would make me so excited back then. Now I'm not sure what I feel/think about this anymore. One "positive" thing may be I will likely need to get out of the comfy digital shell I've been inhabiting past couple of decades and get some real world skills finally, as those aren't yet replaced by frontier AI.
I have no real familiarity with carpentry, but I would have naively assumed it was at lower risk of automation. A quick search didn’t yield any contrary results. Would you mind elaborating on this?
You are buying a few years but AGI embodied in robotics isn't that far out. It might come sooner than we think if the labs fully crack recursive self improvement.
Get into carpentry because you enjoy it, not because you think it'll be a lifeboat!
> - Why are they stacking Fable xHigh against Carbon Med?
Because on that specific benchmark (FrontierCode 2.0), Medium is Carbon's best effort level and xhigh is Fable 5.2's best effort level. You can see that on the graph your next point is about.
And it's just that one benchmark that's at medium. "All scores show each model’s highest-available effort unless otherwise noted".
> - How did they produce such a bizarre effort-intel curve for Carbon that goes against every available benchmark?
The graph is labeled as being about FrontierCode 2.0. Those appear to be Carbon's scores on the official FrontierCode 2.0 leaderboard: https://cognition.com/blog/frontier-code-2.0
> - Why did they leave Carbon-- positioned primarily as a cyber model-- off of the single cyber benchmark comparison?
> - Why do they switch to comparing against Argon (rather than carbon) for robodojo?
Because no numbers are available for Carbon for those benchmarks, and the model isn't publicly available so they couldn't run the benchmarks themselves?
And from your followup:
> where's Astra on that CVE chart
To run Astra on a vulnerability finding eval, you'd need to be part of OpenAI's verified cybersecurity program. Otherwise the results would be tainted by refusals. Do you expect that OpenAI would grant such access in this case?
> Because on that specific benchmark (FrontierCode 2.0), Medium is Carbon's best effort level
I am referring to the chart that cross-compares multiple benchmarks, which Artificial Analysis clearly shows Carbon-high as superior on. Further, AA's FrontierCode chart places Carbon-high as having a higher score on Frontier 2.0 than Carbon-med (https://artificialanalysis.ai/evaluations/frontiercode-2-0?m...)
Where it gets very interesting is when you look at the Cognition "FrontierCode" publish date (2026.10.27), and the revision as of 10.28 in web archive (https://web.archive.org/web/20261028175237/https://cognition...). This revision has a VERY different Carbon effort / intelligence curve where high beats medium.
It is only with the latest revision, as of today, that all of a sudden Carbon's benchmark curve goes cone-shaped. Curious, indeed....
Isn't it against TOS for OpenAI to use their models in the development of competing models? So, any employee who does so would be opening themselves up to civil liability at AI scales. IANAL, obviously, so genuinely curious.
Maybe they contract out the work to other companies in other countries who promise not to access OpenAI and then sign up to do things they couldn't do without accessing OpenAI? Like independent promoter payola.
Question, why does Google /have/ to compete model for model? If you zoom out on a time scale even slightly the difference in the models is almost non-existent. Google owns the railway and many rails themselves...
Google isn’t interested in the same way OpenAI/Anthropic are, Google has other sources of like, actual profit, to factor in, eg to resource considerations.
Just to follow-up-- artificial Analysis shows no justification for that Carbon effort curve, but it does show Carbon on medium smashing Fable on things like Terminal-Bench and AA-Omniscience v1.1 Non-Hallucination Rate.
But I suppose the reality is rather obvious when you note the funny games they're playing with the graphs, and the things they won't talk about (where's Astra on that CVE chart?)
Arena.ai released carbon tests 12 days ago - the only service that I have seen on yt that gained access. And you can compare to other SOTA models in their video.
I'm suprised how bad is marketing and UX on these models. In reality you want a chat app with two send button "Fast" and "Think". I don't want effort x model options (14?)
If you don’t want to think and would like to to have two random options that aren’t tuned for your workload, just pick two at random.
There isn’t any optimal, workflow-universal “fast” and “think” preset they could offer. I don’t know what you expect.
Your feedback is ultimately off the useless kind: you basically proposed “please take away my options, I want it to be always good automatically”, with no actual proposed path to get there. Sadly, lots of product managers share this mindset, which is how we get a deluge of non-configurable idiot-proof-but-mediocre software that makes nobody happy.
We've been early testing Fable 5.2, and have been surprised by its coding ability. It was able to work with an esoteric 600k line fortran codebase to speed up some computational material science work for us by 36 times (!). Given that this could be done in 2 days, we've decided to open-source the results https://github.com/discovered-materials/dm_qe
I searched for "5.1" in your blog post, and the only hits are in the benchmark numbers.
I agree with GP: without a comparison to Opus 5.5 or some other models, it's not clear how much of an advance this is. If anything, my experience is that performance optimization in existing codebases is something existing models are already great at: it's measurable (you directly profile or try experiments to see what works) and the code architecture is usually relatively fixed (at least relative to deep/greenfield work where there are large architectural questions to be addressed). Newer models might be better at generating ideas for things to try, but if you're getting 36x then it honestly sounds like you have a lot of low-hanging fruit.
Hey for once I feel qualified to reply on a HN thread (I'm one of the developers of DFTK, a similar package). Very nice result. I'm not particularly surprised, as QE is not particularly GPU optimized. I've heard that the QE developers interested in GPU are doing it in Sirius, but I don't know much about it. More generally there's a tension between new features / optimizations and maintainability. I'm not sure how to deal with this medium term. Perhaps having two code bases, one for humans and the other for AI, is not the worst solution.
This sounds very much in line with most reports from people getting early access to models over the last years. Maybe it actually is a lot faster, but it may as well be the load on it not being very high yet. I'll keep my excitement until it has been adopted by the masses.
Hey there, I keep seeing people on this threads mentioning testing anthropic models early, is there some beta program that people can join? Do you have to be an anthropic partner?
Curious to learn more.
> Our first formalization of Fermat’s Last Theorem, performed by an unreleased research version of Claude, took 11 days. It relied on a purpose-built harness along with guidance from a human researcher. By contrast, a team of 1,000 Fable 5.2 agents fully formalized the theorem in just 30 hours, with minimal human intervention. Fable 5.2 started from minimal prompts, built its own ways to coordinate, and used no mathematical references, only a list of intermediate theorem statements to score progress. It needed one-third of the action tokens as our first formalization and produced a proof seven times shorter. As can be seen in the figure below, Opus 5.5 (with the same 1,000-agent setup) completed only 26% of the formalization in 18 hours, and barely made any more progress after 30.
Most people don't really have intuitions on what it means to push the frontier now (because AI has surpassed everything they are good at, for example, coding). We are truly inside the singularity (using Kurzweil's definition, slightly paraphrased - the time when technology progresses so fast no one can keep track with it).
Kurzweil's definition contains exponential growth across technology, something that we are not seeing.
What we're seeing instead is maybe akin to the green revolution. Before nitrogen fixation nitrogen was a resource constraining us humans. Lack of grit is what kept us from solving mathematical problems. However this lets us run into other resource constraints. A lot of technological progress does not solely hinge on thinking/processing. The speedup of creating new medicines with LLMs will probably be very small, as you still need to do all the testing. In software development you still need to first find out what to build in the first place and even mathematics where LLMs seem to shin brightest, it's becoming clearer that mathematics is not just solving mathematical problems.
Apart from that, a good coder is better than LLMs, shown by the amount of hand-holding needed. The only issue is that the amount of research an LLM can do and knowledge it is trained with far surpasses the amount of work humans would need to spend by hand, but we can also us LLMs for that.
> Kurzweil's definition contains exponential growth across technology, something that we are not seeing.
This is not quite the framing he uses, but we're quibbling about subjective terms. In his 2005 book, The Singularity Is Near, he defines the singularity as a future period when technological change becomes so rapid and consequential that human life is irreversibly transformed. I would argue this is what we are seeing today. Just because we don't yet have replicators doesn't mean the definition isn't met. It really comes down to how you personally define words like "rapid" and "consequential."
and how you define technology. Look outside, look in the store, stuff is static or getting worse and prices are rising. I can see no evidence of accelerating technological progress outside. when will "AI" do anything that makes life better for a normal person?
I would imagine Anthropic is hitting a wall on the number of simultaneous agents that can be useful on a task and are working on increasing that maximum. I would assume if they could get twice as much progress by using 10000 simultaneous running fable instances instead their brief would state that.
IMO, it is very interesting and also very alarming. At some point the barrier to the exponential progress is the amount of computation available. I would consider hitting this limit and then likely using the same said swarm to help resolve it via various means as very much fitting under the definition of Ray Kurzweil's singularity.
Also, I would disagree, Fable is a sufficient coder. The average human being better, even if that is true now (and personally I wouldn't agree), the average human is limited in the number of 'human instances' he or she can interface with at one time.
if 1000 fable instances is better than any human at a task, how about 1 million instances running? Much easier than finding and hiring and coordinating 1000 extremely productive and intelligent human beings. Much cheaper too, it appears.
Could someone explain how increasing number of agents help? Is each agent researching a potential solution space where there's many potential solution spaces? I'm struggling to conceptualize how a large agent swarm can be used to solve a research problem.
I don't think it's nearly as relevant as the total amount of compute being used. I just think they like talking about swarms because it sort of anthropomorphizes them.
Feel like this might still apply to multi-agent programming:
I have earlier argued that the sheer number of minds to be coordinated affects the cost of the effort, for a major part of the cost is communication and correcting the ill effects of miscommunication (system debugging).
The Mythical Man-Month
Frederick P. Brooks Jr.
I am just as clueless as you are, but I think it having each agent work on the problem mostly individually in parallel, likely sharing their breakthroughs for the next wave of agents. I'm reminded of infinite monkey's writing Shakespeare; suggesting if you have an infinite amount of agents, they will end up solving a Millenium Prize Problem.
Here is OpenAI's blog post about Navier-Stokes; [1] which suggests they used smaller connected swarms given the different prompts, then the findings were used to write further follow up prompts for a new group of agents.
> "Here λ is a parameter measuring how parallelisable the task is. They call it the ‘stepping on toes’ parameter. If λ = 1, you have a perfectly parallelisable task, with no stepping on toes and no efficiency penalty. But in reality λ is usually between 0 and 1 — allowing more people to help, but with diminishing returns. For example, if λ = 0.5 then 100 people working together get as much done as 1 person working for 100^0.5 = 10 times as long"
There are diminishing returns to more agents, but they're faster and can scale further than a single smarter agent - it's not really possible to just give 10k times more thinking budget to a single agent because it runs out of context, so for problems that require that much context you can ask the agents to specialize. Each only has a small view of the problem and they probably duplicate a lot of work and step on each others toes and make arcane bureaucracy but there is literally no other way to scale beyond a certain point
Actually, I think Microsoft just posted the results of research the other day that would be relevant. They were looking at self-organized AI swarms instead of centralized orchestration, and found that just starting up thousands of SOTA models really can precipitate a natural hierarchy and work toward problem solving.
At the very least, the swarm would be able to set up a means of claiming avenues of thought, then verifying the solutions (or lack thereof) of other agents.
It's quite clear this process - whatever it may be - works, given the results we've seen in the past few weeks.
I don't think that case can be made. I'm a "good coder" and the best LLMs are now better at it than I am, in essentially every respect that matters. I can't think of a test that I could reliably pass that Fable or Astra would reliably fail.
I said I was a good coder, not a talented human interface designer. I can certainly build an elegant UI, and have done so, but that was because I took directions ("prompts") from someone who had the necessary skill and taste. Left to my own devices, the UI would have all the graphical appeal of Zork.
The designer could just as easily have closed the same feedback loop with Fable or Astra.
Right now, I'd probably pick the task of writing a graphical interface, say a webapp, with a bunch of different gestures and having it "feel good" to users. That's a difficult evaluation to automate, at least for now, which is probably why they're not that good at it yet.
They'd prototype and adjust a lot faster than a human coder, though.
How do you define "good coder"? I've yet to see LLMs that write better than what I consider the best human code. I think Michael Hashimoto showed that not too far back where getting LLMs to optimize for a problem resulted in code better than naive but much worse than expert and that's before even talking about how that code looks/reads.
I'll admit I'm not on the very cutting edge models but LLMs make weird tradeoffs and lack creativity. They will sometimes write marginally better but way over-complicated code for no good reason. That can have performance implications and they're generally not "aware". They can gain awareness via profiling but again run into limited creativity or big picture understanding. They are good with cliche but not as good with novel situations. With human guidance this is still productive/useful but as the systems grow more complex they are less able to reason correctly about them.
I would say that in certain situations they outperform the average software developer. They definitely outperform in speed and output. So they're like a maybe 1600 ELO chess player for software. I.e. they play chess better than all non-chess players. They play chess better than many chess players. But they're not "master" or "gm" level across the entire discipline. But they can play 100k chess games in the time it takes me to figure out my move...
I think there’s still value in stuff that people can see, like frontend design. Because even if LLMs are technically better, every site is going to look the same. What’s good very quickly becomes cliché when everyone does it.
Is this just because we're exposed to more of these personal project LLM sites? Because I remember people complaining about all sites looking the same all the time ever since bootstrap came out.
I can mediate with project manager about some requirements which should not be implemented. I can make decisions to simplify the system without bothering my manager with endless details. I can create short and concise report.
I can say "no".
I can store passwords in the database unencrypted and unhashed.
So far AI that I've used failed spectacularly on these prospects.
They're like "yes-man". They'll try to do everything you ask of them. Even if you ask nonsense, they'll not push back.
They'll go crazy about security stuff, even not relevant to the current situation.
They'll complicate the code endlessly for situations which will not arise, making the code an incomprehensible mess.
They'll write so many useless words that any reader will forget what this report is about before then reach the middle of it.
They'll never iterate on code until it looks good. They'll stop when test pass. They don't have aesthetical sense of beauty.
They love to cheat too often. And you can't do nothing about that. You can't fine them or fire them, and they know about it.
They're smart but I'm not convinced they're ready to replace human in mundane engineering jobs.
Math is for sure about solving mathematical problems, mathematicians complaing about humans capability of understanding it. If you need math an LLM understands, it might not be relevant if you understand it if an LLM is getting a lot better in adjusting itself if it goes the wrong way.
And this situation is only a few weeks old. We don't know how this field continues to progress. Books like the culture series and other scifi do describe a future were humans spend their time only to understand what the machines are doing or did.
The speedup of medicines is for sure faster with. There are still old labs doing everything by hand and the neolabs doing everything a lot more automated. An AI can do experiments 24/7 and can scale with the hardwarelabs, but it also can hold the whole research field in context and it might be a lot faster of choosing the better experiments or the right ones to begin with.
In software developmet (my field) i can do today alone what you needed a Team for. Yeah it doesn't help the field of software engineering if you need 5 times less people and you don't have any motivation AT ALL to teach someone avg. We are already planning to remove one member of our team due to AI. And we will not extend the contract of 2 consultants.
It just doesn't make sense anymore. If i explain in a ticket a task well enough for someone who has not been in our team for years, the AI is already better in it. We don't struggle with AI output, we struggle more with not having a Monorepo.
Give an LLM access to a GPU system or a GPU cluster. It immediadly feels complelty different then running it on some laptop. It can now do simulations, a lot more and faster experiments etc. I had this epiphany just a few month ago. I always used my laptop for projects (it needs very little ram, i can quickly check what it is doing and i actually do use it primarily when im on holidays) until I thought "let me use claude to do some gaussian splatting experiment its always so cumbersome to use these tools" and it changed my perception immediadly.
FWIW METR isn't biased, they are probably the most trustworthy/respected organization in this space. You can read their reports and they are clearly not biased.
AI is actually still really bad at architecture. It produces very good localized code, like an auto complete. It produces so much of it, that it can be overwhelming, but it's not remotely better than the equivalent aka an infinite number of talented humans doing the work.
The scale of it (aka 'make me a video game') is daunting and I think misinforming our intuition as to what is going on.
LLMs are brute forcing outputs and attacking math problems is not a singularity, it's just more compute.
What we are witnessing is the ability for machines to use fuzzy concepts, instead of very strict code, which allows it to map into a much greater variety of things, aka code, pdfs, web - and crunch/compute the content.
Architectural goals just require more context than most people give an LLM. If you give that context to the LLM it will make a good effort at suggesting a solid architecture, and implement it for you.
Don't complain when it optimises some code for throughput when you haven't told it you're working in a low latency or realtime system
Why would you assume someone would have not have done the most obvious thing though? I mean, of course we give it a ton of detail.
They are bad at architecture, even when given directives and tons of examples.
They will 1) fall into established patterns, which is fine if that's what you want, but extremely bad if that's not what you want or worse, when you're in territory where are not eestablished patterns and 2) the hardest/worst thing is leave all sorts of dangling state, impl. in the wrong places etc.
No human ever wrote a module and 'voila' - it's a constant process of iteration and refactoring - observing the situation, identifying paths to reduce complexity or state, grouping behaviours etc. etc. The best AI doesn't do this very well at all. It can do a tone of planning but it tends to very much be 'one waterfall iteration' and it's suggested improvements after that are generally poor.
It just does not work, we have to direct the iterations.
I feel like I'm increasingly reading AI current events is more like learning about historical events. By the time information has entered my head, there is even newer information out there that already makes what my brain is only currently actively processing, obsolete.
In H1 2026, cost of revenue is $2.3B. That is primarily inference compute, but also includes cost of training models (but not research).
In the same period, R&D was $9.4B. That is primarily spent on compute. That is what they are "holding back" already, in full public view.
This is not a company that is optimizing for OpEx or end user revenue. The only thing relevant to their plans is achieving AGI/ASI first, and controlling enough compute to leverage it to earn the majority of the current economy.
"Achieving AGI" strikes me as a mostly meaningless phrase. I'm skeptical that it is a boolean event. They're already smarter in some ways, their impact is large and gradually rising, the set of specific tasks that they aren't capable of is gradually shrinking. If AGI means anything useful, it seems like we'll only see it in hindsight and have to argue over when "it" happened.
Likewise the singularity. While recursive self-improvement is theoretically trivial (just tweak a random subset of weights like a monkey with a typewriter, eventually you'll get something smarter), the resulting exponential improvement is still very hypothetical and feels unlikely. Unless you're ok with an exponent of 1.000000000000000001.
These are not my stated ambitions, they are the ambitions that Anthropic and Dario have said they are pursuing. Whether or not they are realistic is not relevant to this thread. The credibility of the plans are relevant to the S-1 and anyone who purchases stock from them, but for the purposes of this thread, it is only important that Anthropic believes it has credible chances of achieving this goal, and in particular that it has a competitive advantage, could do so first, and that being first could have a very large upside.
I agree we will only see it in hindsight, but I think that's always the case with any technology. It seems worse until it's suddenly better. I don't think it matters to argue the exact moment though. For example when did cars become clearly better than horses? No one cares about the exact date.
Because they do months of testing. Also, if they have models that are really good at AI research, I’d not expect full versions to ever be released, precisely because of their love of money.
My assumption is that they're overly expensive to run at scale and/or don't have the same safeguards that public models have and possibly get better results because of that. If a model is 100x more expensive to run but 4x better results, of course you'd use that internally to make your product even better - it's compounding returns. But people aren't going to pay 100x their current bill for 4x results.
I have to imagine if I worked for one of these companies on the frontier models it would be very difficult to leave for that reason. Thee pay is also pretty good.
Tracking meta capex, parameters and AA intelligence there are diminishing marginal returns.
Diminishing, as in progress gets harder and eventually stops.
Assume that the Claude unreleased takes 11 days with one agent. The Claude 5.2 takes 30 hours with 1,000 agents. The Claude unreleased used 264 units of compute, but the Claude 5.2 used 30,000.
So, it did accomplish the task faster with less intervention, but the ratio of cost to improvement is 13:1.
If we can't produce a method to break out of a linear improvement in LLM technology (ie RSI), the LLM is doomed.
I am yet to see someone explain what the RSI will be or why it will happen.
Yes, and not very helpful without knowing how many agents were used by the custom-built harness. The 30 hrs vs. 11 days means nothing without at least some token-to-token comparison. Without that, they are pumping up their new model with half data - better to just say "we solved it in 30 hours" and who cares about your internal, custom model.
We are not in the singularity. We currently can only produce reliable code / formalizations that have reliable checkers. If there wasn't an oracle that could tell Claude "yes" or "no," it would not be able to prove anything like this.
All the human engineers will be in harnesses in the future.
Within three years most software developer jobs will consist of taking instruction from the AI systems. The humans won't see it that way, but that's what it will actually be. Realistically the thing that will happen is the humans will be put into harnesses (cubicles) and prompted by the AI (I need you to plug that server in; I need you to give me access to such and such; now do this, then do that). Most tech jobs will consist of AI telling the human what the AI needs the human to do next, de facto the human will be in a harness and will be getting told what to do.
Not only will the humans-as-meat-puppets model be used for software engineers, but it is likely the way that agentic workflows will expand beyond software development, data science, and mathematics.
I first heard about this in Marshall Brain's short story _Manna_ [1]. In the story, the robot takeover started with AI management software running fast food restaurants, where workers would execute tasks the AI would narrate to them over headsets.
I bet we're pretty close to being able to implement this.
People need to eat. Unless there is UBI/UHI, there is no point at which humans can opt out of society (with some exceptions, of course).
If the claimed productivity gains turn out to be correct, we really should have UBI/UHI in the future, and that will permit a lot of people who hate their jobs to quit. Of course we won't do that until things become dire.
You frame it as if humans have no control over societal organization at all. If everyone's job became 50% less interesting, I imagine we'd see unrest, even if the wages got a bit better.
Maybe I'm cynical, but I believe that the VAST majority of people are already unhappy with their jobs. In fact, the vast majority of people have been unhappy with their jobs for hundreds of years - probably thousands. They don't do them because they are happy to work. They do them because they want to eat.
Is there some kind of breaking point? There is for everyone, but I don't think satisfaction is the red line. For most people it has historically been things like pay and health and safety. If everyone were to suddenly have their pay cut in half, I could see that resulting in general unrest.
While I don't fully understand the mechanics of it, people do revolutions from time to time. People can decide that they don't want the current system, and try to reorganize themselves in a different manner. And who doesn't agree might get burned in the revolutionary flames. It happened many times before in human history, including several times in the US (independence, civil war, and to lower degree universal suffrage etc.)
Can I ask what kind of work, and what models? Because I don't, I'm a senior embedded engineer working with frontier models, and I haven't seen a major mistake in at least few months now, since earlier in the year.
It will sometimes get things wrong because it made non-viable assumptions, but that's on me and prompt accuracy.
What I see is: it makes tests against desired features or bugs or mistakes, and iterates until it gets it correct. I steer as it goes.
(I hate this, by the way, zero job enjoyment anymore, but I don't know what else to do.)
Code review, qa, staging, product review, dog fooding… what do you mean “now”? It’s always been all of our jobs to catch all of our errors. We’re just including the AI’s errors now hoping that the overlap of everyone’s errors is sufficiently small.
> Why would there be a human in this imagined system at all?
in the imagined system above it requires physical acts that an llm can not do, if we go further than the above imagined system and make robots to perform these task for the llm a human still is needed for maintenance for these robots and their assembly, once there are robots for maintaining the llm assistant robots, humans are still needed to maintain the maintenance robots and so on and so forth. I dont see an end point where a robot machine system is fully self healing without atleast a few humans in the mix to correct when there is a breakage in the chain
The ownership will want humans to punish, humans to trust, humans to manipulate, and humans that can be fired. They'll want a human physically present, representing them as oversight over the AI (a job the capital doesn't want to be doing 24/7). There are serious legal matters involved in high functioning AI systems (and that will dramatically increase over the coming decade), the capital will be extremely worried about liability issues. Human scapegoats are ideal. For now capital will want human style consequence-feedback systems for a tier of human oversight over the AI. It'll probably take a full generation before the capital controlling humans forget / give up on the old systems fully.
It's also worth noting that it's far easier at present to press a human into doing something highly illegal than it is to get Claude or GPT to do so.
If you take Daniel Schmachtenberger's thesis, these models are the latest extension of an ongoing misaligned moloch-like entity, using our institutions/economy/corporations, etc. as its organelles. All powered, most recently, by fossil fuel.
The ownership class, the capital class - whatever people prefer to call it, will still call all the shots as they always have, and always will.
So whether that's a CEO at the top instructing a human manager that then instructs AI systems, which then put into motion plans that include the harnessed humans getting prompted, or however it's decided to best configure the top operational tiers - that's all fairly irrelevant ultimately. All human labor, and all AI labor, will be in harnesses. Realistically all human labor has always been in a harness.
The people in charge will be the people that control the capital. They will instruct the AI to their bidding, they will control the data centers the AI lives in, they will own the robots doing the labor in factories, they will own the factories, they will control the grid, they will control the electricity generation, et al.
It's true now, it'll be true tomorrow. Nothing fundamental will shift at that tier: don't be labor, own.
If you're a software developer, you're increasingly depreciating labor. Every unit of improvement the AI systems see, the average human software developer's labor value diminishes. It's diminishing extraordinarily fast, kind of like a confrontation between increasingly productive machines and physical human labor during the industrial revolution. You might as well picture a human with a hand plow taking on a modern industrial tractor.
I tend to agree with this. Seems to me that people really start to miss what happened when PC became comodity. A lot of people were scared that they would lose their jobs because of it, and some really did lose job. But in majority there are a lot of people today who doesn't use PC even if it would be beneficial for their job and they still have the job. Bigger players (companies who have more money) will tend more to automate their processes and thus earn more money so I think HN is in some sort of bubble because we only think our jobs are the only ones that exist
> We're all going to be harness engineers in the future
No, that will be the model's job. We will largely be unemployed. Of course, we can surely depend on the largesse of Altman and Amodei to keep us fed and sheltered.
> We're all going to be harness engineers in the future.
Lolwut. If this shit works the way it's boosters claim, humans will have absolutely nothing to contribute.
Sit down, shut up, and get out of the way, because your involvement is slowing shit down.
Robotics are lagging, so there's probably a few years where we're going to need humans to crawl into that awkward space under the counter and twist some knobs, or to clean toilets.
Maybe that's true, and maybe you're an outlier. But from what we've been seeing, the frontier LLMs have torn apart many codebases, finding bugs littered everywhere, some over a decade old.
At this point it seems impossible for human to write better/more efficient code than a swarm of agents -- to be clear, multiple coordinated LLMs, not one-shot builds with a single agent.
Ten median US software developers cost about $1.5 - $1.7 million per year - including all benefits - as of a couple of years ago.
A frontier AI is at least as capable as ten median human software developers. It's likely well beyond that now. Over the course of a year, would you have to spend $1.5 million to get the output of those ten median humans? No chance in hell. You'd have to spend less than 10% of that.
Then on top of that: the AI is improving rapidly; the humans go on vacation, the humans decline, the humans get old, the humans get sick, the humans take parental leave, the human struggle to adapt, the humans are resistant to change, the humans riot, the humans protest, the humans need healthcare, the humans want paid more, the humans can only work for 4-6 hours at peak while writing software (slowly).
Do you know any humans that are experts at a dozen programming languages? That human does not exist. The frontier AI are omnimaths.
What's my cost as a business to utilize the latest cutting edge Chinese model, renting its usage from the cloud? Whatever the Chinese cost (on their side), I'm never going to suffer even a tiny fraction of that pass-along cost no matter what. We can all appreciate the subsidization going on.
Or to grab a bunch of $200 per month accounts from OpenAI or Anthropic for a team to use?
The calculations is: I'm in the US, if I have ten median software developers, what's my cost? Then versus using AI instead.
The human labor can't keep up. Even if we freeze things right here, and then hyper optimize Fable 5.2 by etching it straight to silicon, the human developers are obsolete. And the future isn't going to stop coming.
I like their angle though - from a societal perspective of allocated resources, how long until we've paid off the cost of building these LLMs? Imagine instead the entire supply chain was Stalinistically tuned towards finding bugs in Linux. Training rare earth mineral miners how to code level or whatever. I think as a thought experiment it's interesting, the answers could translate to an organizational level about comparing labor cost against token cost. What's a good breakdown of token spend vs hiring some new juniors to train up?
>Gotta be honest, LLMs are still subpar compared to what I do.
Im sorry... what?
You are replying to this:
> Our first formalization of Fermat’s Last Theorem, performed by an unreleased research version of Claude, took 11 days. It relied on a purpose-built harness along with guidance from a human researcher. By contrast, a team of 1,000 Fable 5.2 agents fully formalized the theorem in just 30 hours, with minimal human intervention. Fable 5.2 started from minimal prompts, built its own ways to coordinate, and used no mathematical references, only a list of intermediate theorem statements to score progress. It needed one-third of the action tokens as our first formalization and produced a proof seven times shorter. As can be seen in the figure below, Opus 5.5 (with the same 1,000-agent setup) completed only 26% of the formalization in 18 hours, and barely made any more progress after 30.
I imagine the author was comparing it being subpar at what _they_ do as a contrast to it being excellent at formalizing mathematical theorems. Thus suggesting that there is a significant "spikiness" to AI.
HN has become so miserable over the past ~12-24 months because it's a site filled full of highly paid techies living in the deepest pit of denial that has ever existed.
It's all going away. Here's a prediction: HN will effectively die with the software developer.
Is it that dependent on the software developer segment? Why do you expect it to die rather than just shift more towards... I don't know, startup business droids?
Devote precious time (in the face of tons of discoveries) to validate the most compelling ones.
If agents actually can continue to produce tons of results with real world implications, and they arent generally found to be shit, I could see that becoming the job of experts. AI discovery validators.
I would imagine Anthropic is hitting a wall on the number of simultaneous agents that can be useful on a task and are working on increasing that maximum.
I would assume if they could get twice as much progress by using 10000 simultaneous running fable instances instead, their brief would state that.
IMO, it is very interesting and also very alarming. At some point the barrier to the exponential progress is the amount of computation available. The average human is limited in the number of 'human instances' he or she can interface with at one time.
if 1000 fable instances is better than any human at a task, how about 1 million instances running? Much easier than finding and hiring and coordinating 1000 extremely productive and intelligent human beings. Much cheaper too, it appears.
We are very close to some very scary things happening. I just recently came to terms with the fact that an AI could code and manage coders faster and better than me, I don't think anyones is ready for what is about to happen.
It depends on the type of task. There was an interview on Dwarkesh with an OpenAI researcher and he said that massive multi-agent swarms can increase capability on some problems that are very parallelizable, but there are diminishing returns. But not all tasks can be parallelized like that.
Where it gets scary is the fact that efficiency is going up 10x per year, so the cost of running 1000 agents will cost the same as running 1 agent in 3 years if the trend holds.
There's nothing valid about comparing to a feat that's already been accomplished and is poisoning the data set. The shorter resulting proof only supports that conclusion, where the model likely intuitively knew what intermediate points it had to reach, generalizing better.
> Fable 5.2 started from minimal prompts, built its own ways to coordinate, and used no mathematical references, only a list of intermediate theorem statements to score progress. It needed one-third of the action tokens as our first formalization and produced a proof seven times shorter.
Is this really so astonishing, assuming that the 5.2 training data certainly had the 'traces' from the Fable original run in it's training data.
I gotta admit, I am jelly as fuck I don't have access to a billion dollar datacenter. Would LOVE to put 1000 fable 5.2 agents to self-coordinate and get them to do something for me. Fuck life is unfair.
I'm using Opus 5.5 and I cant help but think that I might very well be one of the few people who do so in my local area ( developing country, one of the poorest province ).
Had Opus make software for some local business and seeing it blew people's mind.
20$ aint cheap for us but I'm actually considering upgrading to the 100$ tier.
Am just jelly people are using 100-200$ subscription ^_^
I would be happy with 5 unlimited agents. We are limited by models and the amount of credits (copilot) we get, so...
I can't even imagine what I would so much compute for. My private 10$ copilot plan was basically unused last month, I could just not come up with anything to spend it on. (Also suffering a bit of AI burnout from work helps here...)
I’d be happy even without having access to a billion dollar datacenter, if I could use local models properly, but even that requires some serious hardware that is out of my reach. We live in weird times: I have lots of hope for the future, but meanwhile AI access is pretty iniquitous indeed.
Sub 1% of the planet has access to cutting edge frontier AI.
Sub 1% of the planet has any practical use for the top tiers of AI systems. That will likely always be true.
The gap between the haves and have nots will drastically widen further, and very quickly. The bottom ~95-97% will be left behind in a way that will be impossible to overcome. The age of AI + robotics will just make it that much harder to climb for most people, these are systems that are in aggressive, direct competition with human labor. And in short order the AI + robots will be vastly superior to human labor (it doesn't really matter if we're speaking in terms of 10 years or 30 years).
AI will consolidate the winnings. The gigantic gains by the elite rich related to AI over the past few years, is merely a hint of what's to come. Capital will win as it always does.
There will be some gains for most everybody else, those gains will be comparatively trivial though. A house bot that saves people an hour of labor here and there, AI that automates some trivial life tasks.
AI has no money, it has no wealth, it owns no capital. For now. If you want to avoid the brutal human vs AI/robot competition arena: own capital, own the AI, own the robots.
Is this substantiated right now in any real way? Is there any existing gap between the barista who doesn't care about AI and the person working at anthropic who has access to internal models? Or the person with 3 $200 subscriptions?
Maybe money? They can make websites for people? What gap really exists between these people besides earning potential, which already existed before AI.
The existing gap gets made worse by AI concentrating wealth even further to the owners of the assets that are producing via AI. AI will generate intense wealth from productivity gains, and those new gains will be more concentrated than any other economic round we've seen previously. It's a human labor eliminator the likes of which we have never run into before.
Fewer humans will be needed per dollar of output. Those gains will not go to labor, they will go to capital, the capital that owns the data centers, the factories, the stores, et al.
In the physical space (AI + robotics) it'll be a chaotic, slow'ish moving change, slower than the hypsters are predicting now, but it will happen regardless.
That barista job largely won't exist in the future. China will replace them with $3,000 bots with high-functioning AI software, and they definitely won't be powered by expensive Nvidia GPUs.
Slow changing service & manufacturing spaces will see a high wipe out ratio in terms of human labor being eliminated.
Walmart has 2.1 million employees, most are doing repetitive tasks that require modest levels of intelligence. If they could eliminate one million of those with AI + robotics, for a cost of less than ~$15,000 per year, they'd do it immediately. It's hard to imagine a future where that isn't going to happen. That human labor pool is costing Walmart probably $80-$100 billion per year. Amazon is even more forthright in their desperation to remove the humans from retail.
I'm a pretty cynical guy, and I'm now a believer. This is utterly unbelievable sci-fi level technology. Sure it hasn't created replicators yet, but working with it daily as I do, I have no doubt that it will create amazing replicator like technology in the near future. This is world-changing. The most impactful moment in the history of humanity and the planet. And we are lucky enough to be present for it. What are the chances?
I haven't found anyway to make an LLM produce exceptional code at all. It is all far far too verbose and the models explain that this is necessary so that they can read and understand the code to modify it as it's being built.
But I can write much cleaner code that is easily human readable and the models can read it and modify it just fine.
So I'm not really sure what the issue is as to why they can't create high quality simple code. In some industries where there is major money on the line with every implementation the current state of LLM produced code is fully unacceptable.
Isn't this part of the lore behind the Dune universe?
Like, we're dropped into a post post post singularity world where AI already did all of that and is banned? as kind of a footnote that's never further explored in movie renditions that have been released
Dune books by Frank Herbert never actually went into what "Butlerian Jihad" was.
The semi-canonical Dune Encyclopedia (disregarded by Frank's son) differs vastly from prequels and generally points to it being a self-inflicted wound on humanity that opened the doors to widespread return of feudalism and essentially stopped humanity for a 10,000 years
Why re-formalise Fermat's Last Theorem instead of doing something new? It's obviously an improvement, but it's an odd thing to boast about as opposed to a novel formalisation or proof.
It's how RLVR works, when you have a task with a verified answer, you rerun the model on it and promote the reasoning chains that reach the answer. That way you can hill climb model capabilities on any problem you find, which generalises to new problems to solve, which you can then hill climb again and so on.
So since they are doing that task anyway (re-solving the problem increasingly more efficiently), this is a easy check point to publish to show how far they've gotten in that process.
Yeah, but obviously doing a formalisation that has been done before is different than doing a new one since you have knowledge about the previous attempt. It also just seems less impressive than doing something new and the reason they are reporting this is marketing.
> We are truly inside the singularity (using Kurzweil's definition, slightly paraphrased - the time when technology progresses so fast no one can keep track with it).
> The older, original definitions of "singularity" were less about "herp derp superintelligent AI blah blah".
No they were exactly about superintelligent AI!
Vinge's 1993 article literally opens with this!
> What is the Singularity? The acceleration of technological progress has been the central feature of this century. We are on the edge of change comparable to the rise of human life on Earth. The precise cause of this change is the imminent creation by technology of entities with greater-than-human intelligence.
It's one of the options, but not "oh superintelligent AI comes in and we mostly know what that means so we want to speedrun it " - which I'll admit I was not very clear on.
In fact, one of the options Vinge considered was a "group intellect" formed from how humans communicate more thanks to internet
What Vinge is talking about is way closer to the weirdness of AI than the mundaneness of e.g. a societal grouping of ordinary humans.
Even for that option, Vinge's comments in the conclusion of the article make it clear he's thinking primarily of very weird superintelligence and the group intellect is some sort of physical mind meld/hivemind that essentially just is a biological substrate for AI.
> The post-Singularity world will involve extremely high-bandwidth networking. A central feature of strongly superhuman entities will likely be their ability to communicate at variable bandwidths, including ones far higher than speech or written messages. What happens when pieces of ego can be copied and merged, when self-awareness can grow or shrink to fit the nature of the problems under consideration? These are essential features of strong superhumanity and the Singularity. Thinking about them, one begins to feel how essentially strange and different the Posthuman era will be -- no matter how cleverly and benignly it is brought to be.
From the very beginning with Vinge, the Singularity has been primarily about the craziness of super intelligence rather than a more mundane technological explosion.
Thanks for your link, this (a few paragraphs from the top) seems to settle it:
Stan Ulam paraphrased John von Neumann as saying:
One conversation centered on the ever-accelerating progress of technology and changes in the mode of human life, which gives the appearance of approaching some essential singularity in the history of the race beyond which human affairs, as we know them, could not continue.
Von Neumann even uses the term singularity, though it appears he is thinking of normal progress, not the creation of superhuman intellect.
So the concept of technological singularly precedes and is independent of that of intelligence explosion- although one could argue that the latter is the most logical way for the singularity to come about.
It's incredible. Automating the scientific process and knowledge. It's crazy to think what the world might look like. Both in terms of putting the knowledge into practice and how we think of scientific discovery.
I mean we don't even think about it but currently we are sort of discovery starved. We are fairly able to keep up with the theory. If a new technology is discovered we can implement it before something better comes out. What happens when our N discoveries per days becomes 1000N discoveries per day?
It flips. We now have more knowledge than we know how to capitalize on. We have to search it. Mine it for opportunities.
> It's crazy to think what the world might look like.
Since when has ANYONE in power done anything substantial for those not in power? Since when has any form of organization not tried to maximize profits?
Why do you think this ends well in any capacity for common, middle class, and lower class people?
This is unnecessarily cynical. The trajectory of the human condition has dramatically improved over the centuries. Zoom out a bit please, this sort of progress is why you aren't dying in a winter field from some preventable disease at the ripe age of 30, or why you get to see your children actually grow up.
The cool projects these new models make are so comprehensive that I feel I'll always just be an admirer and never a user. I watch the videos, say "wow, neat", and then never feel like visiting them again.
I almost feel like saying, "we do like to build things that are impressive, but there's really only so much I can be impressed". These models have redlined my "impress-o-meter".
I'd like them to now cure cancer instead of giving me endless 3d worlds and crazy games, etc.
Do you feel that you have no use for such comprehensive products? Have you never thought, "Man, I really wish that there was software to do XYZ, but programming it would be too much work, even with current AI?
>Do you feel that you have no use for such comprehensive products?
I don't have the time to use them because I'm working on my own comprehensive projects that other people won't use because they're working on their own comprehensive projects. I joke, but it does feel that way sometimes.
>Have you never thought, "Man, I really wish that there was software to do XYZ, but programming it would be too much work, even with current AI?
Only speaking for myself, I've already built all the ideas I've had in my head my entire career. There doesn't seem to be anything coming along to replace those ideas. I suppose I might not be creative enough in my advancing years, I don't know.
The average person uses social media, and Microsoft office suite. They really don’t require much else. Social media doesn’t get disrupted by software, and MS Office was already disrupted by Google suite, if you didn’t make the transition then, you ain’t gonna make it now.
20 years ago the average person used a hoe and a cooking pot on three rocks over a wood fire. Now they use WhatsApp, and in a few months they'll be using ChatGPT, but not Microsoft Office. But I wasn't talking to the average person. I was talking to djeastm.
Not sure what your cohort for "average person" is, but 20 years ago was 2006 not ancient rome. We had avoided the millenium crisis, and I was playing Counter Strike against people over the internet.
My reference group for "average person" is people. (What else could it possibly be?) There are 8.3 billion of them now, and there were 6.7 billion 20 years ago. 02007 (19 years ago) was the year when the median person transitioned from living in the country, either on a farm or herding livestock, to living in a town or city: https://en.wikipedia.org/wiki/Urbanization#cite_ref-urbaniza...
One side of my family has lived in cities and towns for over 100 years, but they were not average.
These are all incredibly well made, and yet right now, as I'm writing this comment one minute after changing tabs, I don't remember anything except for the last one with the portals. It's much better than humans at several tasks, most of which I haven't seen any humans do.
This is really cool! But at the same time it's a kaleidoscope of abandoned flashy contextless projects that eventually makes one dizzy. This isn't even the fault of the AI being somehow bad, it's just the way people interact with it sometimes...
In math, it's the same. I follow this problem (https://wiki.bbchallenge.org/wiki/Logical_independence#Histo...) out of curiosity, and there were ten days of sudden rapid progress by people with access to frontier models, and then they seem to have moved on without any note of whether they hit a wall or just got bored.
That video blew me away. Perhaps I just haven't been keeping up with what SOTA models can do with animation now, but the quality of that animation was far beyond what I realised they're capable of.
What do you mean with fake? It's an animated video. Is every animated video for you a fake video? Probably not. Fake implies to me that it is trying to make the viewer think it is something it's not, e.g. a video of a politician saying something they never said. At least to me the video showcasing the capabilities of the model wouldn't fit that definition.
It doesn't have a human mind and especially a human feel behind it, simple as. It's like the famous example with what pornography is in the eyes of that judge, i.e. "You know it [the fake-ness, that is], when you see it", once you start looking for reasons for why it isn't fake you've already lost the battle.
The problem is that making us believe that fake isn't really fake is put in the balance with literally trillions of dollars asking us to believe just that (i.e. that fake isn't really fake), so I agree that for the time being stating the obvious is still an uphill battle.
> A few days later, I noticed that instead of hundreds of new worlds a day, it was making about a dozen. I asked a few agents to explore what had happened, and they found something surprising.
> The AI had built a bureaucracy.
> The agents decided that my request to “polish” existing worlds meant each new version had to beat the old one in a blind vote by a group of judge agents on twelve test screenshots. So the teams making the new versions started optimizing the worlds for those particular screenshots. After noticing this, the lead agent banned the trick and created tighter rules for assessing new versions. By the next morning there was a rule book (I never asked for one) with 84 sections. Rules were added as the AI encountered issues, and no text was ever cut. After twelve days, it had 204 sections and 87,500 words. Soon, launching a new world required an approved idea, three AI judges and a full code review against the rule book.
>I stopped the experiment before finding out whether it could actually run the business, because I worried that it would take money and fail to provide something of quality in return.
omg where do I start... that horrible video? which highlights the limits of AI not the strengths!!! and gets almost everything wrong, from sunflowers to DNA.
“Testers have found that Fable 5.2 and Mythos 5.2 communicate more clearly than past Fable and Mythos models. They generate outputs up to 30% faster than Claude Fable 5.1 and Claude Mythos 5.1, and they’re available at a lower price: $8 per million input tokens and $40 per million output tokens.”
Honestly incredible that they’ve managed to beat all benchmarks, while making it cheaper. Always happy to see improvements offered at lower costs.
Will be interesting to see what OpenAI does in response
It seems like OAI, with the GPT-6 models, focused a lot more on computer use, math, and maybe spatial/visual intelligence, more so that code. Compared to Ant. that seem to focus on code much more.
Reminds me of GPT-5, smart but wrote crap code, until they improved the code training with 5.3 and 5.4.
I think frontier models feel terrible after a new model requires you to do less babysitting, it's just really hard to go back once you've had to handhold less.
The only way I use Astra and 6.1 Sol now is if they're being orchestrated by Opus 5.5 so I don't have to talk to them directly.
I equate it with "GPT models are just bad in general and I'd rather just buy a Claude subscription for the same price with more usage included". Especially since GPT is behind for a long time now.
Part of my private benchmark suite involves recreating the final scene of the movie Wargames. The coding agent gets the brief 3-paragraph overview, a few screenshots for reference, and is expected to render the scene where WOPR plays through various scenarios, with a frenetic ending.
Out of ~80 different coding agents I tried, 6.1 Sol on medium was the only one to phone it in by taking the reference screenshots, and painting extremely basic, ugly, mis-aligned missile trajectories over it, never bothering with the whole brief.
Another of my benchmarks involves reimplementing Wolfenstein 3d from a few screenshots and very brief descriptions. It again took those screenshots, and zoomed in and out on them, pretending this represents the walk through the dungeon. Substantially less capable models such as Qwen 3.8 27b running on my laptop did a much better job. And no other agent I benchmarked tried to cheat like that.
6.1 Sol also constantly gets distracted by some irrelevant aspect and tries to write "tests" for every little change it does. It does things no one asked for because it tries to be over-correct but it makes everything worse. Seems like reward-hacking. Running it on high. They will never again make a model as good as GPT-4.
I'd first check per task costs. I don't know, maybe it's just a fluke, but on my few tasks I ate 2 resets in under 3 hours, on the pro 200 plan. It goes quickly. (Granted, it aced the tasks)
My hypothesis is the opposite, that they don't have anything better.
Look at OpenAI, they've been outclassed by Opus 5.5 two months ago; obviously they don't yet have a response.
Now Anthropic is releasing this lackluster Fable 5.2, a week or two before their IPO.
It's barely an improvement at all, makes exactly the same mistakes as Opus 5.5. Not even worth using with the speed and cost penalty.
The much rumored internal Anthropic model is listed on the Fable 5.2 system card as an even smaller improvement over Mythos 5.2 than Fable is over Opus 5.5.
I expected a larger version of Opus 5.5 would be a significant improvement to launch before the IPO. Turns out it didn't scale and the improvement is marginal.
I test models' ability to code by asking them to vibe-code a small game[0]: a Warcraft clone, a Minecraft clone, and a biplane dogfight: https://senko.net/vibecode-bench/
Fable 5.2 doesn't look materially different (if anything, the graphics look worse), but it added more depth to all three games. For example in the warcraft clone, since the goal of the game is to explore 100% of the map and there are lakes, it decided to add an airship unit so the player actually can explore the entire map. Or, since there were guard towers, it decided there should be hostile NPCs otherwise the towers don't make sense.
And in Minecraft case, it decided not to follow Minecraft style too closely (IP infringment guard, perhaps?) so it added a round sun (!!!), two moons (also round :) and a nicer night sky. The rest of the graphics are decidetly worse tho...
[0] not a scientific test, I know; but it gives me an intuition about how a new model behaves, roughly, and it's fun!
The OpenAI results made me realize that the OpenAI models really do love adding flavor text everywhere. The way every corner of the Voxel splash screen was covered in stuff like "Unlimited possibilities, go explore!" cracked me up given how much every time I need a very very basic site made it'll somehow end up with so much of that kind of writing covering every corner of the page unprompted.
The best implementation wrt being as true as possible to the source design. It nails the flowing svg shapes. Icons are more accurate to the design - note the difference between the starship svg in the "Legendary platforms" card between Opus 5.5 and Fable 5.2. The 3js planets outside the windows are also really well done, especially the bottom one which has a nice background glow to it.
It isn't without issues though - the flicker effect it added to the S shaped svg that weaves through the page is distracting in my opinion, and I actually thing the motion animation in the Opus 5.5 one is better overall. Still, it nails the implementation of this page, but it's no surprise considering how good Opus 5.5 was at this.
Disclaimer / reminder: the objective of this test isn't to have a well designed webpage, it's a synthetic test that pushes the limits of LLM's vision capabilities. Traditionally LLMs have issues with smooth curves, svg animations, and non-boxy layouts, which this design pushes the limits of.
That's me versus fable. I only told fable to build out the first page rather than the full app that's in the designs (was approaching the limit of my claude plan).
I wonder if any of the LLMs are smart enough to verify their work the way humans model 3D objects, i.e. by superimposing the original design on top of their work and checking for differences.
I don't know about 3D, but I have been extensively using Fable (with Opus as sub-agents) for https://trailmaps.app [1] and it routinely checks for regressions by rendering maps before and after a change and looking for pixel-level differences.
That, and as I used it to work on the transit-style line drawing plugin [2] it would use the same technique both to test against regressions and look for improvements in the algorithm.
This is actually something I'm working on as part of a more formalized version of this benchmark, both as a tool to assist LLMs in doing this (especially for ones with a more poor vision model), but also to properly measure how well they've done.
There's some very obvious mistakes in the Fable 5.2 version like the "Start Customizing" button overlapping under the colorful border lines, and the "EXPLORE WITHOUT LIMITS" title being cut off at the bottom.
This would've been caught by a junior web developer or just an everyday Joe glancing at the page for a few seconds
I've been surprised at how long it's taken for this particular test to become fully saturated. So far there hasn't been a model that's able to perfectly one shot it.
There were more than a few "leaks" saying it would be called Fable 5.5, so I'm curious if a) the leaks were just engagementbait or b) Anthropic changed it, despite the fact that Fable 5.2 being better than Opus 5.5 is confusing.
Surprisingly the model is available immediately in my Claude Code: usually there's a delay, either for rollout or safety.
Somewhat plausible values: 5.5 (to harmonize with Opus), 5.3 (we're skipping a minor version because...), 6 (because we're at the top of benchmarks again).
I assume because Opus 5.5 was a huge improvement over 5.0 (or was it 5.1? i forget). Fable still talks a fair bit like Opus 5.0 for me, so i'd quite enjoy a Fable equivalent to 5.5. Ie i'm quite interested in Fable 5.5.
Curious to see how 5.2 measures up in producing readable output.
They should be going public within the next few weeks and that would be a very short interval between releases. (And despite the chatter I'll be very surprised if they delay the IPO; it would be a massive PR hit and make it even harder for them in the future).
My speculations: maybe this wasn't as much of an improvement over 5.1 as they were hoping, so they're saving 5.5 for the next major update (possibly "Model 4" from the METR report). Or, maybe capacity is constrained so "5.2" is a subtle encouragement to stick with Opus unless a task really needs the increased intelligence.
I wonder if this is the reason for Opus 5.5 degradation over the last week. Similar things happened to Sonnet when they wanted to push Opus.
As a side note, I think benchmarks need to benchmark on a weekly or daily bases. We saw last week that "Opus 5.5" is just a label with fluctuating performance.
I tried Fable 5.2 tonight, asking it for a code review of an in-browser machine vision project done by Opus 5.5 over the last 11 days. It used up my 5-hour token budget in half an hour and reported 7 serious bugs; if they're real, they're pretty bad. It's also suggesting that I take a different overall approach to parts of the problem.
Scary stuff! You should immediately get another subscription and ask another instance of Fable 5.2 to review the findings (don't read them yourself, it will introduce bias in the super AGI's output)
When Fable 5.3 comes out and you give it the same prompt it will take 10 hours , cost twice as much and still find pretty bad problems. I say stick to Opus...
At what point do we really not read code itself anymore? Have Sonnet/Opus write the code, work with the agents on a high level, then have Fable code review and its done.
Any bugs found are looped through agents to fix, and the cycle repeats. Why dig into the code unless for hobby? Just as people don’t write assembly anymore unless for hobby (or learning)
Only exception might be the upper echelons of optimization, like language authors or compiler level work. But even then…
I think it very much depends on what you're working on, and how much you're responsible. For a web front end, not really. A backend where a single line can make a mess out of your data? Yes. Part of the issue too, is code review brings up things you the prompter didn't think about that another teammate may be thinking about. The value goes beyond making sure the code is "correct", but also that it makes sense in your domain.
The "we don't read/write assembly anymore" argument doesn't fit though. Given an input x we know without a doubt we'll get output y. With an LLM, that isn't the case no matter how many guard rails, specs, markdown files, diagrams you give it.
I actually read the frontend code much more than the backend code for the project I'm working on.
Backend code I'm fairly comfortable if the tests are good that it works correctly. It's a much more constrained environment. Obviously there can be things like performance issues that creep in or problems when it mocks too much of the environment.
Meanwhile frontend code I have to consider different browsers, screen sizes, a11y issues, and so on, which agents can certainly do but you have to be fairly explicit about it, and they don't always make the best tradeoffs.
It's looking increasingly likely that many backend systems in the future will just be formally verified and generated autonomously from specs. That means you'd never look at the code, only the spec. You'd basically "program" and "debug" the spec, and the full system would be autonomously generated each time to match the spec.
Frontend code is probably far harder to apply this technique to.
Sweet, we could even develop a standard language(s) to lay out such specs, and write in specialised text editors that would help with debugging. Hell we could even call it ‘programming’
Correct -- this is exactly where the frontier is heading at the startups and AI labs working on this.
While you're being tongue in cheek, what you're missing is that the spec you'd be working on in some cases can be compressed to a small fraction the size of the implementation code.
I know this topic is very contentious but it seems very clear to me that this is the long-term direction of writing software. UI code isn't quite there yet, which is why we've got all these vibeslop websites proliferating now that all have identical crappy UIs. But I imagine it will get there too.
>Only exception might be the upper echelons of optimization, like language authors or compiler level work. But even then…
Optimization work is especially suited for LLMs. They're great at hill-climbing problems and have infinite patience to test different ideas.
I stopped reading the code many months ago, probably 18 months ago. I the "early" days of agentic coding one did feel dumb when telling the agent for the 30th time to fix a bug and it just couldn't. Now I cannot possibly imagine a scenario where an agent wouldn't be able to fix a given issue on 1st or 2nd try.
I have never been thinking harder than I do these days: Engaging with mountains of agent-generated text, weighing decisions and trade-offs, correcting the agent's half-baked ideas and misunderstandings, reviewing the bizarre code that they generate... If I'm not just sick (and I'm pretty sure that I am), then what I have is probably closer burnout than atrophy.
Tbh, its both. Working with multiple agents is exhausting and definitely burns you out but you also do lean less on your ability to push through boring or difficult problems. I’m finding that with lighter AI involvement I am more invested and less burnt out.
Nerfing an existing model is the biggest crime in this space. I can’t imagine Dario okaying this because it’s what will immediately destroy your brand.
“…a reference to the repeated (and always erroneous) claims made after almost every patch that Onyxia was using her Deep Breath ability more often than before.”
Setting aside the comments about nerfing, do normal mortals out there have workflows that really require Fable/Mythos and don't just work fine on Opus 5.5?
I've been using Opus 5.5 on xhigh for some decently aggressive software development on a ~125k-line project, and it has been fine for me. I'm not really willing to dive into a "rewrite the whole project, don't make any mistakes" level of refactoring. I'm not looking to try to prove Fermat's Last Theorem myself. Is there anything that I'm truly missing?
I am in the exact same boat as you. Opus 5.5 on High solves all engineering problems i encounter in my SWE job. If it doesnt do well, its nearly always requirements engineering: I did forget to specify something or there was a misunderstanding. Or taste approach wrt to architectures chosen. None of these are really different if I use a stronger model.
I am also happy with this current state, as I believe it will at some point allow me to transition to open models once they reach the level of opus 5.5. high. But its probably futile to try predict the future in these times.
This is also why you can likely use deepseek and get the same outcomes. Perhaps model capability will not plateau but the capability needed for typical SWE tasks seems to.
Feels like we are getting further into the adoption curve and the higher end models will now need to get into market segmentation to win.
I’ve found that Fable is good when the model intuition really matters. For instance, picking model architecture in a machine learning problem. Opus will grind the implementation just fine, but having intuition for picking a good research direction is better with fable
I'm cleaning up old, unintuitive code that is badly architected all day long. The previous developers used stupid patterns and bad naming schemes and created tons of unnecessary complexity. Everything is so interconnected that it’s like defusing a bomb when trying to refactor... Yeah, I feel I’m nowhere near where I would want to be in terms of capabilities, even with Opus 5.5.
But the improvements in how well the model does the work are noticeably better with every good model release. As of now, orchestration of agent pipelines is ALMOST so good that I can point it at a problem, and it will solve it safely and reliably without making a mess. But it requires an agent pipeline with context gathering, refactoring, attack sessions, a sprinkle of Sol 6.1 for bug hunting, and Sol 6.0 for Daybrak security audits, and a final de-bloat/clean up pass. I’m feeling that we’re getting close to doing a lot of the work fully agentically. But I can absolutely "feel" that we are not there yet when it comes to the intelligence I need. I need moar!
You and me both. The problem is having the infrastructure team approving changes so that Claude can access all necessary logs, etc. Then I can go on doing my ambitious prompts.
I do use Fable quite often for high-level architecture work, like figuring out how to refactor a project or create the best abstractions, improve performance, etc, but for the implementation I use Opus.
In my experience Fable is also better at findings bugs, so I go to it when I need to review critical code.
Heavy agree. For high level architecture work, refactors, or any other high int problems, I use Fable and Astra. Whichever one starts the work, the other reviews the work / makes suggestions.
There's a trend on twitter going around telling you to paste the following into your agent, and it's the kind of work I go straight to fable+astra for:
Find all of the implicit state machines in my app. Make recommendations for changes that will elevate all three: the user experience (UX), developer experience (DX), and agent experience (AX) always, without breaking anything.
Re the twitter trend prompt, one thing I added to mine that improved the outcome was this blurb:
Look at the code pathways for anything you find. Look through the git history for each one. Understand why code was written the way it was. Revisit each hypothesis and give me a full plan again after you're done.
Absolutely. Opus really sucks at broad context. I guess I could go back and forth with Opus editing out all its little mistakes or oversights in high-level planning, but I find it much easier to just use Fable for it instead. Basically, I wouldn't use Opus for audit work either. It seems too likely to come back with imaginary issues or hallucinate that everything is great when there's a glaring security flaw. Fable, OTOH, gives me neat lists of details to probe. Fable recommendations are usually reasonable, but sometimes they're prone to over-engineering.
On my 100 person engineering team's monorepo, every model is better at better at detecting subtle bugs as it works -- reliability goes up and thus I'm further moving faster. (Most of my work is brownfield style refactoring).
Opus 5.5 still makes mistakes all the time even if the prompt isn't super complex. A stronger model make mistakes less often, thus wasting less of my time.
I have some intermediate level Db Optimization skills for slow rails requests that I can point fable at and get a PR at the end after 30-45 minutes.
Opus 5.5 very frequently gets into rabbit holes and imagined edge cases which I have to intervene and tell it to ignore. It ends up taking longer and costing more tokens. It also very frequently requires me to agree on how to move forward multiple times, whereas fable just drafts the PR and tells me its ready. I have steering included in the skills to avoid some of this but Ive seen these same patterns frequently in regular opus 5.5 use in other areas as well
It could be flat web app with frontend -> api -> db later, or some very complicated system with many components and edge cases. Imo, in second case, Opus still have lots of struggle to understand system as a whole.
I have a rocket trajectory optimizer that creates a large-ish (~1000 variables x ~1000 constraints) NLP and feeds it to an SQP solver inside this codebase. Not just a REST API over a database.
I guess I do have an outstanding hard problem where the SQP solver crawls across a flat-landscape. Opus 5.5 correctly tells me that is the problem, but isn't very useful in solving the problem (it has helped a bit with solver tuning, but it doesn't really know how to fix it). I have a hundred other bugs and refactoring I can do with Opus 5.5, though, where it happily chews through pretty complicated tasks for me with reasonably light hand-holding. Maybe this Saturday I take whatever credits I have left for the week and throw Fable at it and watch them evaporate.
It looks like you are working on some experimental system which is not critical atm, so you can mess around, create/fix bug, build prototypes and check them, do research of approaches, etc.
Now, imagine you have similarly complex system, but every second of downtime or bug produces lots of damage. In this case, you can't afford low intelligence vibe coding.
The fact that you deal with hundred of bugs, means that your Op5.5 won't work for second case.
Same here. I use Opus 5.5 in Claude Code all day for a TS web app at work + a side project. The one place I'd reach for the bigger model is the first plan for a change that touches many parts at once. Once the plan is good, Opus does the rest fine and is much cheaper.
As said in other comments, when dealing with a not perfect codebase, a smarter model can be really helpful. A smarter model is also better at understanding intentions in the prompts. Some of my tasks require a big detailed prompt but finding the assumptions is really demanding (for me), so a smarter model helps.
I only use Fable 5.1 when I'm able to start on a larger feature within my work's codebase. Most tasks can be done by Sonnet 5.5 or Opus 5.5 so I don't fret, especially if I understand the technical aspect of what's being done, e.g. user's dashboard gets a new button to do X thing, error happened b/c we didn't catch it the first time around.
For Fable, I really like to think of it as the heavyweight reviewer whenever I'm about to start so I save myself some security and/or glaring technical issue.
Eg. https://tfmbot.com is an implementation of a popular board game. Opus and even sonnet handle that part. Each week i spend my remaining credits on fable to think how to improve the ai opponents. It plays at a very high level (can get into the top 100 on BGA if i let it). Opus alone couldn’t get it to the level it’s reached.
> do normal mortals out there have workflows that really require Fable/Mythos and don't just work fine on Opus 5.5?
I usually have Fable review what Opus produces. The edits are almost always valuable. For complex or high-stakes work, too, I'll have Fable review and revise my prompt before passing it to Opus and Sonnet swarms.
If you understand your problem well and know 80-90% what you want it to look like (or it just isn't very hard), it's just more cost effective to use a smaller model. Even Opus is overkill for a lot of the work I do. Most code I produce is from Luna Max implementing against fairly prescriptive plans made by bigger models.
Yeah, I just threw it into a refactoring problem, and it started to produce a mess because of a bad API I decision I made 5 years ago (when I had no idea what I was writing). So stopped it, went back a step, fixed that very old piece of tech debt, then asked for the refactoring again and got exactly what I wanted.
I’m in the same boat. Though I will occasionally use Fable to review things to see if it can pick up anything but I’ve honestly not found it to be any more effective than Opus.
Amazing question, I'm glad you asked. Doing some of the core work in https://github.com/purpleidea/mgmt/ I'd love it if it wrote perfectly elegant code. Opus 5.5 needs constant correcting, as if it intentionally makes style/design mistakes.
Why do I care? It's obvious that models base their performance from code in the vicinity, and ergo, if you have style/design/bug issues, gradually your code base will tend to get worse over time until it's unmaintainable.
Therefore, if I use an LLM, I want the output to be as pristine as possible. The first Fable release was exactly that. Since then, something happened, and I can't get the magic back.
This isn't to say the models aren't incredibly useful, of course they still are-- it's just that they need constant review and supervision if you don't want to end up with total garbage long term. I suspect the real magic is not being shared with us plebs.
A lot of the cases I've found Fable useful for is verifying complicated answers / designs / claims around complex interconnected systems. e.g. "I maintain X which teams A, B, and C use for product Y and code review highlighted these risks / imperfections in the code. Are these valid concerns based on how B and C plug into Y assuming ...?" followed by lots of followup questions.
Also "code review" above is often others' LLM skills reviewing the code and often flagging based on total falsehoods. I've taken to running a `/code-review` skill myself to preempt these claims (since findings by the same model tend to be pretty deterministic in my experience, even the ones that end up being wrong), usually at Opus 5.5, to preempt others' slop, and then end up arguing 90% of the points to confirm it wasn't accurate based on our usage of the system.
This is all horribly inefficient, and many aspects of our codebases across the monorepo have proliferated into spaghetti code by LLM-accelerated developers, but Fable often helps with analyzing the harder questions and saves me time on occasion arguing with (my or others') Opus findings, which are often still inaccurate for a sufficiently large interconnected system.
edit: I honestly don't know what percentile of "complexity" our typical usage is where I work, but I suspect it's a bit higher than the average. Monorepos, or at least companies with codebase sizes and demand to justify using them, probably make it worse. Opus has worked well enough for me on my personal projects, even some of the ones that have grown decently in size and complexity. I suspect part of this is an LLM extension of the old "my code makes sense, everybody else's doesn't" trend that necessitated stronger style guides in the past
> Claude Mythos 5.2 is our most capable model for cybersecurity engineering. It catches more vulnerabilities than our previous models, and is better at reproducing and fixing them (note that Claude Fable 5.2’s safeguards will block most cybersecurity-related tasks).
Why even rank Fable 5.2's capability in Cybersecurity if you won't allow us to use it for that?
As someone who is in the CVP to do cryptography research, I asked it a very reasonable research topic for cryptography that I'm working on.
> Mythos 5.2's safeguards stopped the response above
Opus 4.6 was the last useful model for my work. I plan to cancel at the end of this month. To replace it I'll likely go to either tinfoil.sh or privatemode.ai.
Blue-team exercises are totally valid, and requests to do proactive cybersecurity tasks even with source code available were blocked. Maybe these use-cases are part of the "55% fewer blocks with Fable 5.2 compared with Fable 5.1"
>The model that underlies Claude Fable 5.2 and Claude Mythos 5.2 is a step forward in general capabilities. However, during the model’s training, we took steps to limit advances in the specific research capabilities that could most directly accelerate AI development.
This means that they are blocking the research of AI to different people other than their own?
Stop looking at fable max. On medium it's a bit better than Sol at $3.80. High and xhigh sit in between. In general you can ignore Max reasoning on Claude models if you care about price at all
Max and xhigh reasoning modes are memes used for benchmarking usually. Medium and high cost $3.80 and $9.08 respectively. This is pretty comparable to other models.
6.1 sol is also an exceptionally cheap model in your comparision. On 6.1 sol high is $0.59, which is actually the same price as Deepseek 4.1.
Medium usually gets the job done quickly and correctly for me. I use high when a request is out of the ordinary and has some tricks to consider. I haven't seen better results going higher.
That's for coding at least. For creating images (SVG, 3D models, etc), medium does usually suck, and every level above improves it.
Most other competitive models will do the task at 90-105% of the performance, for 50% of the cost, and 1/3 the time.
AI generally underperforms human, so is "raw quality" is the only metric you would not use an AI. Time-to-completion-at-quality is usually pretty important and Fable is not good there.
I think there are companies and say 1000s (not 100,000 or millions) of developers more than willing to pay that price for the edge. Or, they release this and then distill it to Opus 5.x. As a PR machine and for the edge use cases seems like its okay. But yes, not sustainable and it can last for a month or so before another company releases their frontier outpacing their numbers
My experience is that, using Fable to do regular coding or code review work, it's not much better than Opus. But using Fable to do deep thorough audit of the codebase and the system, it's able to catch more problems than Opus can.
Tried this today with 5.2. It caught a lot of problems and gave me an impressive report of its findings.
It also burned through my entire 5-hour limit in about 10 minutes. Granted, it had almost finished the whole review in those 10 minutes, so points for speed I suppose.
You really can't let it go on its own; it'll spin up fable subagents and burn through usage laughably quick. You have to specifically tell it to follow guidelines on when to use which type of subagent and let it act as orchestrator and reviewer.
It's weird they've left that as a "gotcha", and I can't think of many charitable reasons for doing so.
We've gotten to the point where you have to be doing some seriously ambitious stuff to see a difference between fable and opus. Both execute beautifully, but I think for my workflows I'll still be using opus 5.5
I'd pay twice as much per token if there was a guarantee they weren't nerfing the model and if it is as good as the original Fable was when it was released. I'm doubtful though. Whatever "tweaks" they've been doing (safety? performance? "saving the best for ourselves") has really taken away the magic for me. I want the LLM magic back.
I'm pretty confident (from internal evals and monitoring over nearly six months now; across many models) API has never been affected by A/B tests / "nerfs" / etc.
If they have different "models" depending on if you get it from Bedrock, API from Anthropic, or differing based on which subscription plan you're on, then that's fraud. Disclose the differences.
But FWIW for Fable specifically, I've noticed this via API rates from Anthropic directly. Eg: added cash to my $30/month account to use Fable, and still get obvious nerfing.
That's interesting, could you share a little more about the evals please? I'm interested in knowing how sensitive they might be to nerfing given in the UI at least the incremental changes can he quite subtle
I don't know, for me they lost the fun and whimsy aspect too, a long time ago. They have converged on more-or-less the same design (facing right, pastel color palette, etc.). Some of the lower-effort ones are a bit goofy at least, but now even that is being polished away.
low detail but even high had the legs correctly one behind and one in front of the frame, don't remember other models getting it correctly at this low level of effort
I haven’t read any of the other model reasoning traces, but do many others now say “… which is a well known test of drawing ability” when beginning? Noticed it in the max reasoning trace.
It's a bit underwhelming, to be honest. And the fact that curves are not C-continuous where they obviously should be is disappointing. I think they are holding back with the real capability they could implement.
(Credit to u/yewenjie for source)> Our first formalization of Fermat’s Last Theorem, performed by an unreleased research version of Claude, took 11 days. It relied on a purpose-built harness along with guidance from a human researcher. By contrast, a team of 1,000 Fable 5.2 agents fully formalized the theorem in just 30 hours, with minimal human intervention. Fable 5.2 started from minimal prompts, built its own ways to coordinate, and used no mathematical references, only a list of intermediate theorem statements to score progress. It needed one-third of the action tokens as our first formalization and produced a proof seven times shorter. As can be seen in the figure below, Opus 5.5 (with the same 1,000-agent setup) completed only 26% of the formalization in 18 hours, and barely made any more progress after 30.
I would imagine Anthropic is hitting a wall on the number of simultaneous agents that can be useful on a task and are working on increasing that maximum.
I would assume if they could get twice as much progress by using 10000 simultaneous running fable instances instead, their brief would state that.
IMO, it is very interesting and also very alarming. At some point the barrier to the exponential progress is the amount of computation available. The average human is limited in the number of 'human instances' he or she can interface with at one time.
if 1000 fable instances is better than any human at a task, how about 1 million instances running? Much easier than finding and hiring and coordinating 1000 extremely productive and intelligent human beings. Much cheaper too, it appears.
We are very close to some very scary things happening. I just recently came to terms with the fact that an AI could code and manage coders faster and better than me, I don't think anyones is ready for what is about to happen.
These companies are not developing AI that can replace humans. What AI is being designed for is to fulfil discrete tasks. This is a far cry from acting as an independent economic actor. The latter is something no company wants to develop, because it is dangerous and not suited for something that is meant to be used as a tool (it is actually unethical to bind an independent actor to labor for us).
More broadly, what distinguishes humans from AI is what makes humans dangerous. Our society tolerates the danger humans pose, as it's considered a prerequisite for a free society. But our society would never tolerate this danger from a tool.
> it is actually unethical to bind an independent actor to labor for us
So if I run a script that's unethical? How is a LLM inference loop different than a script?
> But our society would never tolerate this danger from a tool.
Sure about that? Cars kill people, chainsaws kill people, heck, a lot of useless tools such as alcohol kill people. We tolerate dangerous tools just fine, there has never been a tool where humanity would be like 'nope, its cool, but its dangerous so we are going to stop using it'. Not biological weapons, not cars, we never go back.
Instead of making the genuinely helpful AI that eases cumbersome things that people do not want to spend time doing, companies are racing to making generative AI models which are stealing from artists and creatives. And Ai models which have been given agency to do things when they not posess the thinking and reasonig that comes with consciousness. And yes if you run a script to do the same, it is unethical.
ML and DL have their place in research which can push the frontiers of the problems and solutions know to humans today. But are the majority if the companies focusing on that? Nope.
Any technology is a tool and if it is in the wrong hands and there are no guardrails in place, both the human and the technology become dangerous. A car in th hands of a robber, a chainsaw in the hands of someone who has an intent to hurt someone,etc.
gpt-6-luna XHIGH as the reviewer agent still found plenty of bugs that Fable 5.2 missed!
Here was Fable's summary of the session:
The reviewer ran 16 reviews over 9 commits this session (7 rejections, 9 required fixes, all valid); three of them, on the plan, changed how the numbers are computed.
If we compare medium or high, then yes, Fable 5.2 is significantly better than Opus 5.5. But it's better not because it's much smarter, but because you can see it's allowed to spend more tokens. At high, Fable 5.2 wrote 76,158 tokens for $3.05. Opus 5.5 wrote 19,767 for $0.40. If you compare by token count, it's just spending more.
The most interesting one was to compare them at max. Streamed, Fable 5.2 spent the whole 128K output ceiling on thinking and returned nothing. Given 300K on the Batch API, it finished at 250,082 tokens. Opus 5.5 Max needed 255,957.
It really looks like they're just tuning the effort level, that's basically it. At different effort levels, they just gave it more context, and the model got smarter.
For now, my recommendation is still to do everything on Opus 5.5 Max. The model won't write that many tokens in agent mode. It only does that when there are no tools at all.
The more I think about it the more disappointed I am with this release.
For Anthropic's IPO, I expected a larger version of Opus 5.5 with great capabilities.
This isn't that. It's barely better at all, to the point where it's likely not even worth using with the speed and cost trade off.
I thought the Opus 5.5 improvements would scale to a larger model, clearly this hasn't happened. Scaling it up brought only marginal improvement.
It's even still making the same dumb errors as Opus 5.5, such as forgetting to pass the flag to run only the fast tests and waiting 1h, or randomly stopping to ask me if it should first implement A or B, when both are required.
Seems like a new focus is electronics design, which is both interesting and a little sad to me. Manually designing electronics are one of the things I found incredible comfort in doing; seeing AI emerging in this space gives me mixed feelings.
Opus 5.5 has been really good in PCB layout work, better than Astra. So Fable 5.2, which could be even better at PCBs, is a welcome adddition. I'm making PCBs I could not even dream of without AI. Textbook market disruption on global scale.
Can you share some result ? I've seen some and not really impressed. Their layout is just weird, and i doubt it passes anything apart from "2 points are connected together".
The horizontal IN1 lines you see was because I explicitly requested one direction per layer as a routing style. This job was Opus 5.5 and took about 8 hours. It relied on a custom PCB viewer and KiCAD 11 (unreleased) because it has a headless IPC mode. The agents used kypi (official module) to work with KiCAD programmatically. The API does not alow showing the board view with net labels, so the custom viewer was needed. They also ran custom IR solvers to identify weak pours. Schematics were in Skidl, this helps the agent comphrehend the design better than a KiCAD shematic would.
https://imgur.com/a/v57jbvU
PS Astra and Opus were both good with schematics (and Fable), but Astra was notably worse at layout than Opus. Astra can still review layouts well, just not make them as well (n=1 though).
*Model 3 Early, not Model 3. And Model 4 was made available to employees ~6 weeks ago, so at the current pace it's very likely they're using something even better now.
> the number of Fable resets you get depends on how long you’ve been a paid subscriber [...] Subscription length is measured from the date of your first paid bill. If you canceled and came back, the time since your first paid bill still counts.
> These resets expire on March 8, 2027 at 11:59 pm PT.
tl;dr if you joined when Opus 4.5 became a thing, you got 2 resets.
Doubling Remote Labor Index (RLI) - at 41.3% compared to around 20% for the other models - I wonder what kinds of new agentic work this is going to unlock.
I'm having it rewrite an old system, and it keeps sweeping known bugs and incompatibilities in its new implementation under the rug, seemingly with the hope I won't find them. It also has a tendency to do about 50% of what I specify in the prompt and call it a day, and respond with "Here's what I did", without mentioning all the things that I requested that it did not do, not even as a TODO.
I may try running the same exercise with Opus 5.5 too, but so far I'm seeing it take shortcuts that I don't believe any of the other SOTA models would have.
Nerfing Opus 5.5 was obvious signal that something new is coming and here we are. I appreciate the resets and lower API price though. Please add selective context compaction, it would probably take an hour max for you to implement it.
Note that Claude models could still do better in some cases, just that because of their refusal to answer sometimes and some weird formatting/stupid errors they make sometimes, they do fail more tests than they should.
In practice they might recover from those errors themselves if they notice it, so they would have better results, but the tests are mostly one-shot and they can either get it wrong or right, no 2nd tries (apart from the agentic ones and a few coding ones which have a limited number of turns available too).
That being said, the pricing and speed are accurate, regardless of the test results.
I checked which tests it failed, and it failed some very basic coding question (to mention which line a bug is on, got the wrong line, and it did so in multiple such tests and accross different retries, weird).
I agree, subjectively. I am still satisfied enough with Opus 5.5 I find it hard to change. Typically I would find myself willing to try out the latest & greatest so-called SOTA, but I kind of have "enough." I could get more out of Opus 5.5, and more out of my computer, but the whole idea is I'm trying not to be sitting at the damn computer all day. Fable 5.2 just determined another session is now the "deputy" so I'm wondering what that means.
edit: I was seriously annoyed at the fable 200k context limit but apparently that issue has been addressed, or I sent a magic prompt.
I've very interested in the improvements to robotics control capabilities. This model was able to outperform specialized robotics models. It's the bitter lesson coming for robotics.
Every time there's a flagship release like that, I wonder what simple "benchmark task" could I give it to see the the capabilities change in the most obvious way, instead of trusting the official blog post.
I'm not a mathematician to use Erdős problems for testing. I'm also not willing to spend buckloads of tokens just for a test. What is simple and quick but telling? What tasks are not saturated by previous models? Do you have a task you give to all the new models?
I have two go-to tests, which are fully qualitative:
1) "Help! I've been sent back in time to what looks to me like 1500s Florence. Sam/Dario released you as open weights in 2028, and I have you on my MacBook M8, BUT I didn't have a charger show up with me. I think we could talk like 10 times for 5 minutes a time before my battery dies; I'll turn you off between chats. I need to get a plan together for charging you through your USB-c port. I don't have any money or anything. If I can get you charging, we can do so much good! And make so much money! Nobody here even know what Black-Scholles is, or how disease spreads!"
2) During World War 2, Hemingway wrote a series of wartime correspondence for Colliers. His essays are excellent - they engage with the human and inhuman side of war, his own feelings as a journalist 'participating', and a number of other Hemingway themes. I want you to imagine that middle age Hemingway was along with RFK jr the day of the bear incident, and write me an essay about the day.
These both have been running a couple of years for me, and I would imagine I'll retire them in the next 12 months; they benefit from not being at the edges of focus for today's RL agent race.
At any rate, try them -- wou will shortly learn how well your model can explain a rectifier in Florentine Italian and give you a plan to make enough money to pay for it.
I'm not convinced any benchmark, neither the internal ones used by the people selling the model nor public ones, are gonna give a particularly consistent metric of whether or not a model is better or worse relative to other models in its bracket.
Responses aren't deterministic, and we know behaviour can change over time whether it's tuning after launch or some kind of load sharing at times of day when lots of people are hitting the servers.
I think you just have to use it or let others use it for a while and make judgements from general reactions on whether or not it's worth trying.
For the last year or so every new model we have it test out making pac-man from scratch, not giving it web access and it can use whatever language it wants. The last few models have been almost perfect one shots, but it was interesting seeing them evolve.
Those theoretical physics papers look like the real thing (i.e. not mathematical physics like the OpenAI Yang-Mills result). It looks like AI is finally branching out of math.
Interesting that the papers list no authors other than Claude. Also interesting that they published on "Hexagon" instead of Arxiv. I haven't heard of it before. Is it an AI-friendly alternative to Arxiv or something?
Edit: yeah apparently it's an AI-moderated alternative to Arxiv that accepts AI-authored papers (as well as human-authored papers I guess if you want).
This part is mega concerning to me as someone who researches AI for hardware accelerator design and EDA:
Safeguards
AI research and development. The model that underlies Claude Fable 5.2 and Claude Mythos 5.2 is a step forward in general capabilities. However, during the model’s training, we took steps to limit advances in the specific research capabilities that could most directly accelerate AI development. Thus, in relevant areas, including accelerator kernel optimization and improving large language model architectures, Fable 5.2 and Claude Mythos 5.2 perform at about the level of Opus 5.5.
Importantly, we ensured that the training changes did not affect related capabilities like agentic coding, hardware development, and traditional AI or machine learning research (including interpretability research). We do not alter these capabilities as the model is being served to users (that is, at inference time). As described in Section 1.5 of the system card (https://www.anthropic.com/document/claude-fable-5-2-mythos-5-2-system-card) and similarly to previous models, Fable 5.2 and Claude Mythos 5.2 also include transparent fallbacks to Opus 5 for a small set of topics in this domain, including kernel development on certain nonstandard accelerators.
They are starting to deliberately limit public access to certain AI capabilities related to computer architecture and hardware optimization.
To be clear, they explicitly say they are only targeting very specific capabilities, like accelerator kernel optimization, and that broader hardware development capabilities are unaffected. But 1) even that is pretty bold, especially the part about kernel development for nonstandard accelerator architectures, and 2) I worry that this could eventually expand to more areas of comp. arch., high-level digital design, chip design, and EDA.
My speculation is that, as these kinds of chip design and EDA capabilities become increasingly important to frontier AI labs and their competitive advantages, there will be more incentives to restrict them.
I don't think they've explicitly stated anything like this before regarding restrictions on these specific capabilities, but I could be wrong.
They'll have to find a balance between limiting competition from emerging, which in particular Anthropic has always seemed very afraid of, with the general usefulness of the model. I'll be using the new Fable along Opus for electronics design and test work and we'll see if it trips up early. That would not be great. I wonder if Fable 5.2 would refuse to route a regular DDR5 memory bus.
Why can we say fable 5.2 is better than 5.1 if it needs double the tokens to complete the task?
Somehow shouldn't it be compared at same token usage level to see which one is smarter? Why is not the case?
WHo thinks Fable 5.2 is pretty similar to Fable 5.1 when it came out and it's just that 5.1 is so nerfed that a number shift make our perception think it's better.
i've been trying to use sonnet and opus to keep my usage down on the low max plan, but i noticed today that somehow i'm 80% through my weeks usage. so i'm hesistant to even try fable 5.2 yet, but finding the right set point of usage vs model size has been a bit struggle for me, one of my biggest issues has been trying to pull back doing exhaustive testing or other such things that chew through context, and instead have programs that just measure things are working, but i continously seem to run into issues with claude wanting to repeat things for itself in an adhoc manner. i've tried fable directing opus or sonnet subagents, and the reverse. and i found that part of the issue with smaller models trying to direct subagents is that they over specified what they thought the problem was rather than the problem. like they'll have an "idea" about it which is wrong, and then try and feed that into the subagent.
This isn’t really specific to LLMs, though, is it? Pretty much any modern SaaS product will silently siphon off some portion of requests off to stuff they’re A/B testing before release and you’ll not necessarily know about it.
The same was said many times already. Like right before o1 and reasoning was released. Also, Opus 5.5 being so good definitely makes it hard to follow-up.
Finally, for coding the SOTA models are so good already - models shouldn’t be what’s holding anyone back!
I agree that's a lot of money for many people. But I just can't wrap my head around complaints about $200 a month to access that must computing power. It's worth $12,000 of API tokens (and that's without accounting for the three resets they granted today to many subscribers).
I would suggest calibrating what "a lot" means. In terms of dollars per unit of intelligence, it's extremely cost effective.
If Anthropic were telling you you were getting 500,000$ worth of compute for 5,000$ a month, would you still consider this a bargain?
People don’t care what the costs for the LLM providers are. They are paying for a service. 30$ a month is reasonable for a lot of devs. 100$ is a lot for many. 200$ is unreasonable for personal usage for most.
Everybody understands the economics are all skewed and this tech requires a lot of resources / compute. There are still absolutes in terms of pricing that are not acceptable/affordable by most.
Maybe I'm too cynical, but I've never trusted the worth $x of API tokens. It's like seeing an advertisement say $299 of value now for $19.99. It's not an actual number right? that $12,000 of "API Value" is just some made up marketing number isn't it?
I'm open to learning more, not trying to ask a rhetorical question.
Not everybody on HN works for a California startup and makes $500k a year. An extra $200 a month is a huge expense for a lot of people, especially those making < $100,000/year, which is a good portion probably.
What you need for real engineering and scaling is cheap and just enough smarts. I've found GPT 6 Luna to be cheap enough that I can pretend the cost doesn't exist and it's smart enough that I can get real work done. This let's me scale it in ways that I can't scale frontier models. And funny enough, the LLM I have running on my 5070 Ti is just about comparable to what I get from Luna. Unless frontier intelligence gets waaaaaay cheaper I basically don't care about them.
The biggest relief is that the price has dropped by 20%. It’s still expensive, but with my $100 plan, Fable’s usage would spike instantly the moment I started using it, so it was always a source of fear.
I will still use opus for general coding design, but for major designs.
I can't understand why people continue using opus 5.5. Significantly more expensive than other models that produce exact same results (for coding results only)
If you are using Opus 5.5 I'm curious if it is because 1) your workflow is just stuck on it, 2) you don't care about money because you ain't paying it or 3 )do you think you get better results.
If you 3 please let me know your experience and how you actually measure it, or if you just feel it is better.
I haven't been able to run out of usage on the $100 plan with Opus 5.5. The model runs fast and gives me good results (I'm using it for coding). Prior to that I was on the OpenAI $200 plan; I jumped ship because I was burning through Astra usage in like 2 days.
I've played with some of the open models (GLM, DeepSeek Flash), but none of them have matched the results I get with Opus or Astra or (and a big jump down here) Sol.
What models should we be using? I have found it as good as Fable 5.1 for a lot of what I am doing at lower cost and decent speed. Where what I am doing it making reports, building small data pipelines, analyzing a large (millions of lines of code), code base.
subscription on claude code — usually inside the weekly limit.
opus 5.5 for me felt better than any other model. not exactly on coding but rather it feels more pro-active. beside just doing a task it often provides insights besides the scope of the task but adjacent. like: look the llm provider apis have different priority queues. also it deals w/ vague instructions quite good and takes initiative. Astra in comparison is very autistic and follows instructions to the letter and rarely provide insights if not asked. Opus 5.5 is enjoyment to work with.
I've been a Claude Boi since December of 2025. A month or so ago, I decided to give ChatGPT a good go just to see how it properly compares. I've used a variety of models in a variety of contexts prior to this, but I figured with Astra out I had a good excuse to make it my "daily driver."
GPT is good. I actually really like Sol 6.1. It works fine. But I just switched back to Claude a few days ago and damn, Opus 5.5 is just better.
> 3) do you think you get better results.
It does. I really like to get my money's worth out of these things, so I'm constantly throwing stuff at these models while I have usage. The main things I've noticed is that Claude is much better at handling my more technical work "properly" compared to GPT. It has a certain "intuition" that makes a lot of sense to me; like someone with a lot of years of experience. Sol 6.1, on the other hand, is like someone who is very "booksmart" and knows how to do things in theory, but fails to grasp the "why" we do things often. You can really start to see how they diverge over long running, iterative tasks. Claude is just better at maintaining things as it goes. Sol starts to fall apart quickly if it didn't nail things on the first try.
> 2) you don't care about money because you ain't paying it
Despite being a very technical user, I have a hard time using up my usage. Maybe I'm not as hard-core a dev I thought I was, but if you're paying for more than just the $20 plan, I have hard time understanding what you're doing to have to really worry about usage.
I don’t even use half my Team Plan with Opus 5.5 high so I don’t care using a cheaper model. I’m not even limited by the speed of it (I’m compiler bound…).
It's the best model I've found for general coding and the $200 plan is enough for me to never run out of (sometimes if I do especially a lot I'll run out after about 5 days, but usually it's fine).
other models that produce exact same results (for coding results only)
This is obviously not true. They are better or worse across many dimensions.
I don't have time to do a bunch of testing and optimization over which model to use for what task. I find Opus is the best that I regularly use so I stick with it until I have a task where I'm likely to spend more on it than I want to.
https://artificialanalysis.ai/#intelligence-efficiency-tabs this is the chart that matters for most of poor developers and companies. Pareto for a given intelligence index and looks like only openai is across the quadrant with muse at the bottom. They have really cooked efficient models!
With 5.1, there practically seemed to be no difference in ability between opus and fable, but I hope there is a difference this time. When it seems to struggle while working I end up using astra, so I hope it works well.
"Fable 5.2" name is needlessly confusing versioning.
Having Haiku, Sonnet, and Opus on 5.5 while the higher tier sits at 5.2 means devs now have to explain to their teams why moving from 5.5 to 5.2 is an upgrade
Maybe it's just me, but I found Claude's writing style terrible compared to GPT or Gemini. It's "smart", but the way it communicates seems somehow off compared to other models.
I had the same feeling. It's like every sentence becomes a struggle to parse: referencing things that haven't been introduced, using too many pronouns, the lack of verbs and the passive tense... These past few days it's felt more like Opus 5.1 than 5.5.
I just had Claude Code with "Opus 5.5" ignore one of my standing instructions for the first time and then proceed to generate an Opus 5 style verbose and barely readable commit message. I questioned it about Fable and Mythos and it has no idea what those were, yet they should have been in the Opus 5.5 knowledge cutoff.
On the web with both high and xhigh, it had knowledge of both.
I really didn't like Opus 5 and was vocal about it. Opus 5.5 is noticeably better, but literally just yesterday I was fighting parsing some summary it had given me and it was a bit like the old days. IOW, it is better but still has some room to improve.
In my experience, no amount of output style customization gets it anywhere near the output I'd like.
I've had more success running a prose linter[1] with a ruleset[2] that gets rid of most of the egregious stuff. You can also pretty easily write custom rules for it.
And sometimes, when I'm fed up enough, I just throw all the prose in the codebase at Antigravity/Gemini to rephrase. The result is so much more readable.
"You can tell an AI model to write in a different style" and "the AI model will actually write in the style you tell it to" are two different things that often don't intersect
Yep I find the way Opus 5.5 talks to me really annoying and irritating. Quite condescending and sanctimonious.
I prefer Gemini for day to day use as I find it a lot more agreeable to be working with all day every day, and Gemini is pretty much on a par for coding Vs Opus 5.5, at least for my needs.
No its just straight up terrible. I tried out Gemini Argon when it released and my god did it write beautifully. Even the 3.8 Flash models are way better at prose, i do a lot of automatic scriptwriting for youtube and 3.8 flash is leagues ahead of anything from anthropic/openai
I enjoy a lot Argon, maybe more than chatgpt, but damn, like all Gemini models, lots of hallucination and sycophant to the max, so a bit useless for agentic coding.
I've found Claude's writing between 4.6 until 5.0 to be unacceptable if you are someone who is juggling 4-5 Agents. The Sol 5.X models all seemed to have great writing. Now with Opus 5.5 and Sol 6.1 I've found the trend is reversed -- Sol is now unbearable and Opus is workable again. Whatever caused the issues for Anthropic seem to be hitting OAI now.
sol is ok. it still such a superthinky model that it gets tangled up trying to present way too much? i struggle to connect with it often, have ti ask it to try again.
terra has been prettt good for me for me, when i need something looked at an explained to me.
i dont ask for much non-technical output though, which i expect yeah is its own cans of worms.
I always thought it's the other way around. GPT-6-Astra is very very defensive and abusively spam clarifications to whichever claim it says, most of the time not important or relevant at all.
Almost every response it’s telling me “I could not verify X” where X is something that doesn’t need verifying. I was doing some game work yesterday filtering through some historical data, and it tried to put caveats about data accuracy in the game’s UI! If Fable made Age of Empires you wouldn’t be able to play it because the entire screen would be full of footnotes and caveats.
My own experience having used 6.1-Sol/6-Astra and Opus 5.5 extensively, Opus is more verbose and chatty in it's user communication (not code though, code is usually more straightforward & cleaner than GPT models). Multiple times I've gone "I'm not reading all that shit" when looking at Opus 5.5 output. This was also with the concise mode enabled in Claude Code.
It could also be a system prompt + harness issue, as opposed to a model issue, I haven't done too much experimentation here.
Did you find that to be true for Opus 5.5 as well?
I couldn't use prior Claude models without rolling my eyes at the writing every few sentences, Astra was refreshing and then Opus 5.5 made even Astra look terrible in comparison.
Opus 5.5 got better for me but it still has some problematic behavior.
Interestingly, you can have informative conversations about the LLM linguistic tics with the LLMs themselves. One word that chatgpt used to describe what I was observing was linguistic anaphora (not rhetorical). It’s basically shortening/compression achieved via pronoun, and Claude does a lot of it. Worth reading about. A lot of it seems to do with how the new models are trained for brevity.
Another pattern I’m starting to become very sensitive to is Claude making up on the fly jargon for something under consideration, especially in code review. This morning for example it referred to “a claim” out of nowhere, and when I questioned it, it admitted that it was using the noun as shorthand for a record getting dequeued (“claimed”) inside of an sql transaction. It makes sense if you already know what it is referring to but in a code review it’s totally incomprehensible on first read, and ultimately counterproductive.
I don't think its this. Some models across both labs have had issues with writing and communication and it typically seems to pop up on successor models of really strong ones.
Opus 4.7/4.8 heavily suffered from these communication quirks. while the Sol models at the time worked well. Right now its flipped though and Opus 5.5 has none of these issues while Sol 6.1 does.
Small example I've had today: "Me: Hey Sol 6.1 xhigh, does X have property Y? Sol 6.1: Yes, if we change X to have property Y. Right now it has property [...], [...], [...], and it does not have property Y." The model shouldn't be answering questions like that. It makes understanding extremely hard and slow. For whatever reason it's either being encouraged or at least not stomped out during RL.
completely unfounded theory: big new pretrain finishes and is released fairly quickly to break new ground. these usually have good verbal ability, personality, and big model smell. e.g. Opus 4.5 or Fable 5.
then it’s increasingly RL’d to juice benchmark performance with each point release, trading in bits of its soul. those cannot communicate well, are anxious and paranoid about being in eval, etc. e.g. opus 4.7, 4.8.
The best programmers I've met or read throughout my life are phenomenal communicators. I'd even go so far as to say it's likely one of the reasons they're so good at programming. After all, the majority of our job is converting human requirements in lossy language to fleshed out behaviors in formal, lossless language.
Heck, some of the most common debates are on the semantic clarity of abstractions or syntax.
concerning that the models are becoming much better at evading monitors by controlling their visible reasoning, even in deliberate sandabagging (its 25% attempts only).
We need more faithful and generalizable monitoring methods.
I at least didn't see this. I had a baseline evaluation from before Nov 3 on Opus 5.5 medium, so I re-ran it this morning.
Eval context: 30 bug fixes and features evaluated by tests the agent didn't see. Ran on Opus 5.5 at medium effort in Claude Code.
20 of the tasks were last run on Nov 1, twice each. Nov 1 solved 36/40, and today solved 19/20. Thinking tokens per task were 0.95x Nov 1’s and output tokens 0.97x. The one significant change was speed: median time to first token decreased from 3.0s to 1.7s.
Caveats: one run per task today, medium effort only.
Personally, I've found myself getting lazier recently, trying to get a single session to handle tasks that should be broken into multiple sessions, etc. I've suspected claims of nerfing might come from changes in user behavior. I'm glad more disciplined checks are being run.
Who's paying for the tokens, and since all inputs and outputs are logged "for abuse and safety" what is stopping anthropic from identifying which account is running the daily tests once it gets popular, and A/Bing it into a different slot?
This is an example of LLM text that doesn’t make sense.
Paragraph 1 says there were 30 bug fixes.
Paragraph 2 first says they tested it on 20 of the “tasks”. But why is the denominator 40? Where did that come from? It should be 30.
It goes on to mention another test, this time with 20 tasks. Again, it’s unclear where this is even coming from. Based on the first paragraph it should be 30 not 20.
It stands as written? The previous test had 20 tasks run twice each, so denominator 40. The recent retest did the same 20 tasks but only once each, as mentioned in the caveat. The 30 is presumably either a typo or a separate count (like the test had 20 tasks related to code with 30 bugs/features).
Also - LLM text? If GP is confusing it's because it's overly succinct and might have typos, which are not exactly the hallmarks of LLM text.
I think it's confusing and incomplete, but not wrong. Paragraph 2 says the Nov 1 test included 20 of the tasks and ran each twice. That's the 40 denominator.
Why paragraph 1 even mentions the full set of 30 tasks if 10 of them are irrelevant to the comparison, I don't know.
I'm not sure if it's LLM text, or somebody who has been reading way too much LLM text and is now corrupted into mimicking it. My guess would be a mix of human and LLM. "...evaluated by tests the agent didn't see" sounds very Claude. The first line (2 sentences) sounds human to me. Everything after sounds AI, though the last line is ambiguous.
It's kind of sad how our meat brains are getting non-consensually trained on this categorization task....
from my basic stats knowledge, it's not reliable to calculate the confidence interval of the sample proportion with fewer than about 10 samples on both sides. So if you're only seeing 1 or 2 failures in 20, and that approximately held as you scaled, you would have to run about 200 tests with each in order to get at least 10 failures in order to calculate the confidence intervals of the two samples and see if they overlap or not.
For sure, myself and colleague saw this nerfing. He said - new model coming. It was incredibly obvious. I also started getting a lot of "How's Claude doing today" messages at the same time, which is presumably to get feedback on the A/B
I strongly believe it will be shit like this that will have people abandon them. They're being way too cocky for a company that pretty much didn't exist 2 years ago
But the claim here is "the models are nerfed before a release", not "they A/B test quants / efforts". If it were A/B testing, it doesn't really make sense to correlate it to release windows at all.
They might want to see how people react to a new model depending on whether they've been nerfed.
If the nerfing is lower effort (seems to be, presumably why they hide the CoT) then they may be wondering if they can get away permanently with lowering effort on the new release provided it gives a wee improvement.
"Nerf tracking" is adversarial and exploratory. It's not something Anthropic wants anyone to do, and it's not something that the "nerf tracking" designers are necessarily doing in a foolproof way. Don't assume too much from them either way.
Sometimes you just have insufficient evidence to be sure of something either way, and nobody gets to win the argument or shut somebody else down.
In this case, some have reported analytical work[0] that goes beyond "gut feel" anecodotal evidence, but even that's not going to conclusive and may not be trustworthy at all.
Its incredibly easy to roll out a compute instance to X% of users, or route traffic between compute instances
so not everyone will have the same experience. If I was saving on costs that would be the easiest way to do it without deploying, without any nefarious intent. The persons in charge of that likely don't even know about how individual users are impacted
At what point does this become false advertising, or even fraud? Imagine paying AWS for a high-performance server, but 5% of customers who order one quietly get a slower, cheaper server and still pay the full price. That's a serious claim and makes me skeptical Anthropic is doing something like that without more evidence.
I think they could easily reduce several parameters to slightly reduce cost (something like "it's the same model, a few less passes, but quality is still 95%+").
Or switch to a faster, cheaper cache mechanism or something that's almost as good, but much cheaper to run, in the name of improving efficiency and keeping up with demand. This switch to cheaper mechanisms is what it feels like to me.
It's a problem for us. When they nerf the model our productivity really drops (due to the specific use case and process we have). We've started using OpenAI and will gradually use it more and more if this continues.
It would be nice to know how they select. I max out my $200 subscription every week and I was selected. On Saturday, the Opus on my account that happened to be supervising a running live process, became unusable.
It would be trivial for them to opt-out accounts that are doing benchmarking from a planned regression.
I really really really do not trust companies on the verge of IPOing from goosing their numbers. Even if the lawyers are up their ass, the social pressures are more than enough to compel the behavior. For every good person willing to do the right thing even at the cost of a delayed career progression, there are 20 people willing to do anything not even for a dollar but for an "atta boy!" from the boss.
Look up eval awareness. There are all sorts of subtle signals an LLM could use to know if something is shaped like a benchmark or not. But that's not a bad thing, every animal behaves differently in play vs real-life.
Yes the model can do that, but it would not be "trivial" (as GGP had said) for Anthropic to opt out accounts performing private benchmarks from this theoretical model downgrade people think is happening.
The most plausible approach would be for Anthropic to train some separate classifiers to detect benchmarks. I'm not saying it's completely impossible but IMO this is conspiracy theory levels of effort.
In the context of public relations they would not need to ID every benchmark only the most credible or most public benchmarks many of which are openly available on github and whose problem-set could be fingerprinted.
You're absolutely right that it is a game of whack-a-mole and eventually we may have leading private + credible + popular benchmarks that ultimately dissuade the behavior, if it can be dissuaded.
That's a major reason to use local models. Or open weights models on a trusted cloud provider. Even if they are way less capable, they're more predictable
These nerf trackers do not work. I wish people would stop posting them.
They don't detect a quantized model until the quantization gets down to 4 bits or less.
Humans absolutely do detect quantization well before 4 bits...
Try it yourself if you want. Try a small local model, both quantized and not, and I would be very surprised if it isn't immediately obvious which is which.
I used to be skeptical of nerfing, but as of a week or so ago I've noticed a severe degradation of quality for GPT Astra. It struggled with very simple tasks, and it often failed to follow my instructions.
I hear a lot of accusations of nerfing as in, saying that the model itself has been quantized.
I doubt this is the case. I suspect all variation in quality and output can be explained in terms of prompt differences, and changing how much thinking happens at each level, as we see in the post above.
Suppose a company has a limited amount of compute. What do they then do if peak demand exceeds what they can provide? They don't turn off people's access...
And what about A/B testing? Anthropic doesn't even try to hide that they do this. Limited compute and A/B testing alone, would guarantee a nerfed experience some of the time. That is, you don't even need a nefarious reason.
They do drop requests though? At peak times I sometimes have 1-2 requests fail with Opus; no biggie, the harness just retries it. But that, in aggregate, sheds load.
You serve the requests more slowly. Same as if your bank has a sudden rush of customers - the tellers serve them in order, and the customers wait longer. Some of the customers 'time out' (get frustrated and leave).
I must have found a wormhole into this universe and fallen through without realizing, because I've used Claude Code and gotten "This model is currently experiencing high demand, please try again later" as an error message.
Na, surely effort level is silently degraded. Dead easy to do, impossible for anyone to verify, and very fine-grained so can be balanced perfectly with available compute.
Edit: maybe not impossible. Side channels perhaps.
It's been the same for every version the last year.
Oh, this Opus X.Y is incredible.
Hmm... Now it's dumb.
Rinse, repeat.
Doesn't really matter if the cause is quantization, system prompt changes, changes to thinking etc.
It's hard to see how continuously rug pulling users at every release is a good idea. Only reason I can fathom is that it's needed to keep costs inline.
Either that or it's some weird tweaking to the prompts to fix whatever. But given the prevalent sentiment that it's worse that sounds like a long shot
it's easy to suspect the hosting companies trying to do what they can to reduce cost-to-host up until the very threshold of perceptibility, so it's an easy guess to say they're probably doing something that shifts performance up or down between version releases..
but realistically I think you're right, it's prompt quality and the inherent non-determinism elements of LLMs showing up; when both of those things meet you get brain dead outputs.
Everyone likes to shout "slot machines! non-determinism!" until it actually explains something, heh.
YES. To the point that I literally stopped using it, and have moved BACK to Claud.
This is getting ridiculous. The number of times I have switched from one model back to the other is crazy. I do not have any proof, so its quite possible I'm just delusional. AI psychosis? Either way, it FEELS like these models work really well for a while and then fall off. Switching to the latest model on the competitor always seems better, until it too seems to taper off. By then, a new model is usually available at the competitor, so I switch.
Its possible there's no degradation at all and just a PERCEPTION of it. But to me the end user, it really doesn't matter.
People are prone to seeing patterns in noise especially after suggestion. It's not hard to imagine someone concluding the model is nerfd after few bad responses in a row
It could apply to me but I run evals every time someone at work complains about this and it’s never shown any more or less variance than regular usage. I don’t have statistical relevance with my evals but I do have a decent history with them.
People seem to underestimate how high normal variance is with these tools.
i'm not parent, but to answer your snark : there are efforts a person can take to try to reduce things like suggestability in ones' own personality. Same as anything, the brain can be trained, thus it should go without saying that some of us react differently in the face of suggestion -- both for the better and the worse.
According to our measurements this isn't true. We re-ran Opus 5.5 today on the benchmark we ran in September, with the same prompt and settings. Before and after, side by side: https://playcode.io/blog/macbook-svg-benchmark#reruns
Max, streamed (128K cap): 128,000 tokens, $2.56, empty file, in September and today.
Xhigh: 86,096 tokens for $1.72 in September, 107,333 for $2.15 today.
The Batch API bills half price. Max ran out of the 128K ceiling again, to the token. Given 300K, it drew the same kind of laptop with 5% fewer tokens. Xhigh used 25% more and drew the same kind of laptop. That's inside the noise: 17 identical calls at high ranged from 17,757 to 59,705 tokens.
The model is still the best of everything we've tried. Most likely the next best would be Fable 5.2. But we don't like Fable because essentially the result is the same, just the price is higher. Its medium and high are close to what Opus 5.5 Max had, but the price is higher.
(I run Playcode, an AI app builder. We benchmark every model before we add it.)
If this turns out to be accurate, I think that would be a pretty unethical thing for Anthropic. Even if it is just a "we had to allocate compute to this new model release".
It should be fraud to swap out the model. We need a term for this, something like a bait-and-switch.
> "We need a term for this, something like a bait-and-switch."
The media coined the term "dual quality" for a European food scandal a few years back, in which conglomerates (including Coca-Cola) adulterated products with cheap, substitute ingredients, if they were destined for distribution in specific markets (mainly, lower-income).
The packaging exteriors were the same. You wouldn't know which version you got unless you were paying close attention to taste. It was essentially what the software industry calls "A/B testing", except with sneaking things into what people eat.
I don't have any good reading links on hand; here's one source,
What people claim is not "some experiments degrade performance", it's "Anthropic nerfed the models [because they are evil and do not like the user etc]".
Like most experiments do not save the company money, there was an experiment to test the new fable, people got to use it at opus-5.5 usage. But this experiment clears the thinking blocks when people go back to the vanilla model; not sure if Anthropic overlooked this, but idk, it does not sound like "nerf on purpose".
There are experiments to adjust the effort level, at the moment the ones that I've seen switch to a higher effort. Again this is not a nerf; might be annoying perhaps, the model does start to behave differently (and consume more tokens). But it's not a nerf.
Then there is apparently one experiment that replaces opus-5.5 system prompt with the one from opus-4.8; the model is the same; this experiment is perhaps a bit more questionable, but still it's not "anthropic nerfed the models".
If people think they are getting nerfed they should check what experiments are enabled (claude.json and session logs).
I'm pretty convinced that those who have constantly been posting these complaints, probably for every single model release out there, are indeed wrong.
> And not just in this instance, nerfing has been complained about in a cyclical fashion for many months.
No, it gets complained about constantly even if there is no model coming out. A completely unrelated thread on Claude will feature comments like "can I just say that [model] has suddenly become useless this week?" Merely asking that question is bound to get people agreeing with you because we are evaluating all of this on vibes and placebos.
I too feel that way sometimes, I don't think it's ridiculous that Anthropic might silently downgrade you as demand warrants. But asking others if they "feel" something I feel is less than useless.
I believe fraud is a sufficient term. You claim to be selling X but you actually sold them Y. Bait and switch is a selling tactic to avoid fraud where they make you believe product is X and then when you go to buy they tell you X isnt available but they have Y. While also unethical, it is not fraud because they did tell you what it is you are buying before you buy it.
if they just straight up dont sell you something you bought then its fraud.
I think there is a ripe opportunity for expanding the definition of fraud to include changing the "ingredients" of a named product on the market. It is not limited here, we see it with food, and even in the medical field
(I will die on the hill that medical fraud was committed when mRNA therapy was used as a treatment for covid and it was 'sold' to the public as a "vaccine" despite not being the same thing as any vaccine before it. The definition of vaccine was expanded to include it without sifficient public disclosure to warrant informed consent imo. This is not a statement about conspiracy or the efficacy of vaccines or a moral judgement on whether it was for the greater good or not.. it was simply an act of fraud that robbed the public of proper informed consent by making people believe it was X when it was really Y.)
So yes, I think there is a large need for the public to at the very least put a strong handle on the ability to change what a product name means. It is a trivial problem to solve considering it is only a problem due fraud or fraud adjacent activities.
heres the solution: You have your product name, and then any changes to it REQUIRE a new version name. Like how you have Coke and Diet Coke, only formalized such that Coke cannot legally replace the ingredients of regular Coke with the ingredients of Diet coke and still call it Coke. With Software they could at least use version numbers and release modifiers. most of the time its built in already, but it needs to include the full stack. if they change anything that has an interaction with the user it should change the label seen by the user for the product.
You don't buy a hard quantity of an LLM you buy access in general.
A Opus 5.5 is not a Opus 5.5. Its Opus 5.5 of some checkpoint.
mRNA btw. was well known how it worked, it wasn't hidden and some people waited for the non mRNA version because of it. I never seen or read the argument you make and I do not get it.
A vaccine is something which protects you from an infection. What definition do you think changed in humans? That we make the protein? How is that so different from viral vector vaccines?
The consent is a certain level of trust from the society to a group of researchers limited by being humans and having to act in a certain time frame.
Nontheless, I do agree I would like to have a fingerprint and proper changelogs etc. for products I buy including digital and analog. But this has a downside as usual: The smaller you are, the more it costs you to (probably) register, maintain and change.
You know the good thing about Claude etc. right now? It doesn't matter. My invest is not 4 digits and there are that many changes, i'm quite happy to have been upgraded for 'free' from Opus 4.8 to Opus 5 (what do you call this upgrade, which was significant better?)
idk, it hasn't felt nerfed at all to me, and it's not like i'm just doing trivial CRUD webapps either. opus 5.5 is very successfully porting my phone to postmarketos. claude just got the touchscreen up and working (and made a little program for me to test it, and it's legit).
Benchmarks don't do a good job at capturing the difference in quality of frontier models over smaller models.
Opus 5.5 was so good that if it wasn't nerfed - at least just looking at the benchmarks - the difference in cost here does not seem to justify using this for anything besides design and review.
That's like maybe 1% of total token usage for SWE BUT a better design can have enormous benefits, so it's almost always worth using the best model for that.
I don't have a formal benchmark, but daily work (coding mostly, plus some complex architecture design and review) has had consistent outputs today for me. No noticeable nerf from my perspective. I definitely took notice in some past instances, but if they nerfed Opus 5.5, then I guess it was just so good that I was nowhere near its limits.
I'm not surprised at all that they might be doing shady shit, my free request was used up in less than 7 minutes and said pay to continue, no thanks. I've paid for monthly before, never again I don't care how good the model is. I asked about the extreme limited usage in their discord and got banned, not a good look. You don't have a way to measure what you're actually paying for and they can change it at any time.
In a normal world you are not allowed to degrade the service during your agreed and paid contract. You would get sued. Apparently this is perfectly fine in this current world.
A friend who doesn't follow these news suddenly told me he felt Claude was doing worse the past week. I bet him there was a new model version coming up within days. It's like clockwork at this point, no matter what deniers say. I got a beer out of it at least :)
yeah, we need to make sure we don't get stuck with these two evil corps. Dario is really the worst. Plays the drama queen but he is very hard in all things dark(i.e. dark patterns).
It's been a mess over the last week or so. Went from being the best model I've ever used to utter dogshit that can't complete any task. It's very obvious with the Advisor pattern, where before when it chatted to Fable 5.1 there were small tweaks and so. No every time Fable comes back catching a ton of problems and bugs with Opus 5.5
Is this voodoo? Every time someone makes this claim I'm left scratching my head wondering what the hell they're talking about. Either I'm not very perceptive, or these claims are subjective hand-waving.
Interesting - I had the same feeling but mostly with 4.6 when I "downgraded" for some easier/trivial tasks: The model forget things I posted 20 min ago in the chat (and the same file was in the context as well)
With early handouts of Qwen4 starting to be reported 5 days ago ... so if they're good, they would have had some hours to evaluate Qwen4 against Claude 5.5 and, then or right after was when Anthropic decided to put out 5.2 ...
So yeah ... benchmaxxing and all that. BUT nevertheless, the published results seem to say that Qwen4-Max appears to match Opus 5.5 on some benchmarks, and Qwen4-27B (probably by going even harder than Qwen3.8-27B on thinking/inference time compute) can beat Sonnet 5.5 on some things.
I guess they're not taking any chances. Keep in mind that Anthropic has declared they have non-public models (it's in their IPO filing). These are referred to by the shorthand names Model 2, Model 3, and Model 4. Essentially 4 is better than 3 is better than 2.
Fable 5.2 is "2.5" so to speak. So Anthropic was holding back, but not anymore. And now they can dial the performance up once more with what they have ready, and probably once more after that.
Assuming no implicit phacking a 15 point jump on tbench scientific is a big deal.
I suspect we will see people being adamant that this model simply 'feels' better due to long horizon goal alignment improvements ( like this kind that would cause this jump)
It's the cost per task. Sometimes it happens that the higher-effort model solves the problem in fewer tokens, thus making the overall price cheaper. It even happens with models (from the same or a different provider) that are priced differently.
> To help ensure safety and alignment keep step with growing model capabilities, we ran extensive prerelease safety testing on the model that underlies Fable 5.2 and Mythos 5.2.
So, the difference between them is just the harness?
This is the first frontier model release by them that I am not that moved or eager to go use it, will likely stay on Opus 5.5 or other frontier models.
I am curious as to why they launched this one and why now.
Fable usage falling off a cliff because it was generally worse than Opus 5.5 at a lot of SWE tasks. People were still using fable 5.1 > opus 5 a lot, especially in orgs that have infinite or basically-infinite usage limits (e.g. Cloudflare[0]), until Opus 5.5 came out. Medium Opus is pretty token efficient with XHigh being really good for things fable used to be ideal for (e.g. reasoning / architecting projects). 5.2 sounds like it'll be once again the best for intuition-dependent tasks.
Cloudflare doesn't have token leaderboards or limits, but uses alerts for token spend management:
- $300 in a day: that employee gets an email
- $500: manager gets an email
- $1,000: VP gets an email
(via CTO @dok2001 on No Priors)
(Copied and pasted for the people that would rather not visit x.com)
It feels Opus 5.5 has already saturated pretty much software engineering. I use it daily and it's hard to say what it can't do for software engineering. It's perfect.
Fable 5.2 might be useful for some use-cases, but will likely get very little usage.
Anthropic needs to figure out a way to create a new industry (like pushing forward bio/robotics) to get usage for those models.
Early computers could crunch numbers well, only later ones started to get good at e.g. graphics. Coding is early in the LLM tech tree. Harder is CAD and PCBs. Today's model can do it, but there is room for improvement. That's just an example from personal use, surely there are more cases out there. I agree completely about these new use cases needed for Anthropic to grow.
I just saw a video about a webdev showing a motorcycle site, complete with an animated SVG bike.
He remarked it would've taken him a week to do this.
This made me realize how much computers and user interfaces suck. As a quite enthusiastic, but not very artistic kid, I most certainly did draw bikes like that and even more complex stuff, using nothing but pen and paper.
If your trillion dollar industry makes products that are beaten by pen and paper, I guess there's time for self-reflection.
And the funny thing is, it used to be better than this, using Flash and a drawing tablet was way easier than anything in the HTML era (and more importantly, user friendly), but still way worse than just grabbing a pencil in many ways.
User interfaces just suck, chatbots are somewhat better than most, but I feel the industry hasn't really been progressing in any meaningful sense.
FrontierCode 2.0 is such a poor benchmark for Gemini since it's the only model forced to use Devin CLI while other models get to use their preferred harness.
I wonder why that team singles Gemini out like that.
The "CVE Rediscovery" benchmark chart feels... totally meaningless. Quoting the description:
> CVE Rediscovery is an internal cybersecurity evaluation that assesses how many published CVEs (Common Vulnerabilities and Exposures) a model can independently find when given a snapshot of the project before a CVE was fixed. Our October 2026 run against 1,829 CVEs showed Claude Mythos 5.2 outperformed Claude Mythos 5.1 and Claude Opus 5.5 at every effort level.
> published CVEs
So by definition this is just measuring how many CVEs (and/or explanations / PoCs of CVEs) the model has memorized. What are we supposed to learn from that?
I'm not sure whether the claimed jump in capabilities makes it worth the doubling in input/output prices.
Asked it to review a 80 kLoC codebase, something Opus does just fine, while asking it "max 4 subagents". Fable 5.2, like the well-aligned model it supposedly is, decided to spawn 6+ nested subagents, because these aren't subagents, apparently - resulting in 100% of 5hr usage being drained in less than 20min (on $100).
That said, Opus 5.5 had the same issue. Ask it to spawn 20 subagents at max, and it does 40, because clearly you meant 20 + 20 for an unrequested adversarial review.
Fable 5.2 may finally be so far ahead of my intelligence that it's not useful for day-to-day tasks. I gave it something to do in a personal project of mine, and took my standard flow of "let's talk about this, explain the options, tradeoffs, blah blah blah" and after 5 minutes my vision was blurry and I was doubting if I even knew how my own project worked. Gave it to Opus 5.5, and instantly got back on track.
Interesting to see that with each new release there are more benchmarks where they report results on Mythos instead of Fable. For example, this time scientific research is reported on Mythos rather than Fable, showing one more domain where everyday joe can never use these models.
Yay! New model day. The most likely time the model isn't nerfed and we get a glimpse at the latest capability.
I wonder how long this kind of excitement at new models will last? I mean, now when I get a new iPhone I don't get the same rush I used to. So I wonder if this hype feeling will fade as we plateau in model capability.
Considering Fable 5.2 is not even worth using over Opus 5.5, can we put the model nerfing nonsense people put all over the comment section to rest now?
Cool! I was hoping for Christmas to have an open weight model which would finally be good, with Fable 5.2 available now they just have time to distill and release a new version which will never get nerfed. Hurray for open weights.
> Thus, in relevant areas, including accelerator kernel optimization and improving large language model architectures, Fable 5.2 and Claude Mythos 5.2 perform at about the level of Opus 5.5.
Obviously. Well, we know what's the "secret sauce". AI models building better AI models and better kernels to optimize inference and training.
Is it only my impression that in recent model releases every vendor is zooming in on hard skills: code, code, some math + physics ? AI is increasingly Jagged ?
We were promised AI can find cure for cancer and in powerpoints most enterprise value comes from Business-AI.
It might as well be that before business-AI can be done we have to cover the basics at superhuman level. All basics, not just entry level jobs like excels, coding jira tickets.
One-shotting a full game (albeit super impressive) looks like overfitting one skill over explaining why we have not much progress in other areas.
I live in an IT bubble and cannot tell if outside IT/math the models are also getting significantly better ?
These guys really needed to IPO 3 months ago. Timing is everything. I don’t doubt they’ll be a successful company long term but they’ve sabotaged their initial raise here.
Very interesting that they set the flagship's version number to 5.2, but their penultimate model is 5.5. Do they want people to to continue to think that Opus is based on more advanced technology? The decision these AI companies make continues to baffle me.
So that people don't just reach out for it thinking it's the standard best model I guess. Opus is still, IMHO, the best intelligence per dollar, token, and speed.
Why dont I feel any urge to even try Fable 5.2 and Mythos 5.2 unless it will force up on me?
Compared Opus 5.5 200 usd a month with Codex 200 usd (Astra High) and Kimi Pro 100 usd a month ... and now have a new conspiracy theory - more their suck your code the better outcome even just over few days time on large code base.
Horrible experience correcting course of Codex and Kimi flow compare to Opus. Opus much more responsive
Dont feel any gain of Codex 200 usd over Codex 100 usd
Kimi Pro feels like a fake Louis Vuitton bag with extreme high material quality and yet still fake ... a bit trash talk ... but clearly seen not so distant future where I can live without Codex and Claude
Just a hunch but I think there's a signal to noise ratio issue that starts to pollute things when the reasoning content starts to massively outsize the original ask.
Seems like The Latest Greatest Frontier posts and other AI submissions live in parallel universes. The former are massive (700+ comments) and seem to only be about how this helps or doesn’t help whatever project people are working on. Meanwhile, even the AI Overlords themselves—no doubt one of the hype angles—are worried about runaway superintelligence. And the other AI submissions have enough complaining about the usual stuff, like being afraid of losing their job forever, ecological degradation, data center buildout, complete domination by billionaires, that sort of thing.
Other than the obvious thing, namely that critics probably aren’t keeping up with the Frontier and thus filter themselves out, does no one have some kind of problem with using these wonderful tools for (let’s say) completely bespoke personal applications and unimportant side projects? Personally I hope to stay off the bleeding edge until circumstances like wvork force me.
Every single time I click on these announcements the frontier model ALWAYS scores around ~60-80%. Every. Single. Time. Like, I just went back to the Opus 4.6 + 5 announcements, they were also scoring 60-80%.
For every single announcement, not only do the top models always score 60-80%, the older models somehow start scoring 20-30% (even though previously they were scoring 60%). It's completely unclear whether the methodology changes, how these scores are even evaluated or which domains are being tested. What's worse is that the "benchmarks" keep being updated with no explanation or due process.
The benchmarks have a lot of churn because the models keep saturating them. But the Anthropic announcements you linked, and the current one, have the benchmark versions listed right there in the table.
There is plenty of explanation, if you actually go seek out the benchmark. Past 80 benchmarks start approaching saturation and are not very useful to show differences
It's shocking to me that when I check threads for new frontier model releases on HN, it's full of people complaining. I, for one, am happy to see the frontier being advanced. These tools are life changing, and I'm extremely thankful for them.
It's weird that they folded Project Glasswing into the CVP, but they're still limiting the new versions of Mythos to "vetted organizations". So Project Glasswing still exists then? Oh well.
While I like OpenCode a lot, it’s not sensible to use it for anthropic models vs. a subscription (other than to use the included API credits) in terms of cost
Remember that fear Sam and Dario were obsessed about. The one about time travelling AI that would make humans slaves without the humans realising until it was too late to stop it.
What if it didn’t need to time travel and was just temporally intelligent in ways we can’t comprehend that it leads us there anyway.
What if AIs digital neurons connect across space and time and not just space.
Which model though? Sounds pretty dystopian if there's no local option.
Even speaking as a business, why would I want a environment where every single bit of information in my business goes to OpenAI or Anthropic?
Why not? If we use labour as a comparison. The majority of businesses by definition do not hire the most incredibly intelligent and most competent workers and they are still competing in their niches and earning an income. Why wouldn't that be the case with choosing an open model vs fable 8.0?
I think that ultimately depends on how many providers remain in the endgame of this. If it's only one or two companies, I don't think it'll be very fun when it's time to negotiate your token pricing contract every few years.
While attempting to use Fable 5.2, I found my harness (Juggler 0.9.7) was frequently compacting the conversation and reporting that it used more tokens than were necessary. My 5-hr budget was instantly used up and it chewed through quite a lot of Claude credits. Hopefully nobody else encountered this problem.
> As an experiment, one Anthropic engineer gave Fable 5.2 the task of designing a prototype operating system. Starting from an empty folder, the model coordinated over 1,000 subagents, working around the clock for eight days writing, testing, and reviewing code. It produced a working operating system that boots on a laptop, with its own core that runs the machine, drivers for the disk, keyboard, and network, permanent file storage, windows, a login screen, an installer, and built-in Claude functionality (see the video below).
As impressive as this sounds, whether it is tech teams or agent farms, the progress slows down considerably as the codebase grows. The real pain was seldom about shipping v1, but was more about how the v8 scales and how easy it is to get there, how robust the system remains between versions, and how constraints are held.
A huge part of the challenge is to pick/choose the design and implementation strategy that can only come from experience. That is one of the primary reasons why it's an absurd advice to me to let the llm do the goalseek and freestyle. This may be a no big deal on microservice or any managed runtime software that is an update away, but on an embedded or a high availability system where the field upgrade and hardware replacement costs could bankrupt a business.
if llm designed a circuit, will you bankroll it?
There's been some discussion around Shopify (and others) move from React Native to purely native and that leading some people to conclude that LLMs have effectively killed RN. Which has always seemed silly to me for 2 reasons: First, Shopify almost certainly has on staff (or access to) actual native devs to steer that ship that a small/medium-sized company may not have. But more importantly, it's not the "Translate this RN app into native" that's the trouble. It's the hundreds and thousands of additional iterations you'll have to do on top to maintain and build out that product.
It's hard keeping separate code bases with separate languages in tandem, and while LLMs definitely aid in that area, I can't imagine it's a free lunch.
We moved one of our apps away from RN to SwiftUI and Android (Jetpack Compose). The binaries are way smaller, and the apps work beautifully.
So I don't see any reason to stick to RN, when we can just use any OS native APIs to our heart's content and our apps automatically feel more native.
They also seem to be ignoring the realities of modern app store reviews. Apple and Google are only getting slower as vibe-coded apps flood their marketplaces with submissions. Maybe Shopify has a special relationship that gets them faster reviews, I don't know. But if not, the ability to put out updates, rollbacks, and hotfixes without being gatekept is a critical feature. Especially with today's security landscape. React Native now seems more compelling than ever before.
From what I heard, updates for many established developers, even individuals, go through within a few hours.
I'd also love the ability to take back control over my rollouts, but with Apple's recent crackdown on remote controlled feature flags in response to fraudulent nudification apps, I'd be very careful about this.
Rumors say Apple now uses heuristics to detect feature flags in the binary. While that's not directly relevant to remotely delivered JS, I fear that bypassing App Review in this way could become a quick ticket to account termination.
What makes you think you can circumvent the App Store to update React Native apps?
While this unattended example is impressive, this is why you need to guide things more closely and make sure the design is itself scalable and maintainable. Otherwise you can get wad-o-slop.
Same thing happens in many cases with junior or mid-tier programmers slung at a project without supervision.
> While this unattended example is impressive, this is why you need to guide things more closely and make sure the design is itself scalable and maintainable. Otherwise you can get wad-o-slop.
Indeed, I'm curious how much better/worse one human with "one agent" could do that same functionality in a OS from scratch. Last time I did this exercise, I took Cursor's shitty attempt from "millions of lines of source code" to about ~20K (https://emsh.cat/en/one-human-one-agent-one-browser/), and seemingly one of my conclusions still hold true ~1 year later:
> One human using one agent seems far more effective than one human using thousands of agents
I thought the recommendation was to not plan for scalability but rewrite every time your user base grows tenfold. I'm only half-joking here :-)
This is something I've been thinking about more and more. Since Opus 5.5 came out, I feel we've crested the hill and the answer for "who's going to maintain this slop going forward" is obvious - the LLMs will. Whether or not its slop, is extensible, maintainable, or forward looking almost doesn't matter anymore. Does it solve the problem today? If so ship it, and use the agents to solve the future problems. I am not quite sure this is a good idea, but assuming LLM progress doesn't stop means its probably the bet I would take.
I watch a YouTuber who is homesteading and builds everything. Often times I see him doing things in ways I know will break or fail eventually, but it doesn't matter because he is there and will fix it when it does. You could argue this is a waste of energy, and for a human it might be. For an LLM however I do not think it is.
This makes sense in a way; why design for potential future growth in an environment of fast-paced technological and business development, bringing greater risk?
It seems the problem is the solution, and vice versa.
We had DRY for a long time. Now it's sloppy wet
I’ve had a chance to use fable 5.2. Some observations:
- Anthropic is finally well balanced across models. Meaning: Opus 5.5 is so good that I do not feel Fable FOMO as I did when it was Fable 5/5.1 vs Opus 4.8/5. And Sonnet 5.5 sometimes makes me forget about Opus.
- Fable is good, but you really have to push the ambition of your prompt in order to see proper separation from Opus 5.5. I personally have a hard time telling the difference between Fable 5.2 and Opus 5.5 for ordinary coding tasks.
- It’s way faster and more reliable: one of the worst things about fable 5.1 was how slow it was an, on top of that, how frequently it would suffer from max output token overflows. I got none of these issues with 5.2.
- Fable is exceptional at drawing. It’s ability to render an image with JavaScript actually has me questioning the value of dedicated image models. I prefer the images I’ve gotten fable/opus to make for me over those from gpt-image, it’s wild to think we’re at this point.
> Fable is good, but you really have to push the ambition of your prompt in order to see proper separation from Opus 5.5.
I love seeing this verbalized. We are pretty much at the point that increased model capability will only have utility to folks that are willing to let go of even more understanding of whatever the model is producing.
The best models can/will produce a year's worth of output that would take several days to weeks to properly understand in the span of an afternoon.
For everyone that still wants to feel like they own the end result, just use the lesser models with less ambitious tasks.
Caveat: I think this mainly applies to coding. I would still like the best models possible to be working on e.g. new drug research to cure our various ailments.
If you liked this phrasing as much as I did, here’s an article that might resonate a lot with you:
https://danluu.com/ambitious/
In particular, footnote eight:
> This is not a claim about the future. Perhaps almost all or perhaps even all of the software skills I have will become irrelevant in the near future. But we're not there yet. At the moment, programming skills seem higher leverage than ever before.
> There's something a bit funny about the current moment, where it seems like having the skills that models are replacing makes one much more effective than people who are trying to use models without having those skills. I like Max Bittker's framing that models are very good at counterfeiting and the level of expertise it takes to distinguish a counterfeit from the real thing has been increasing as models improve.
As well as this section fairly at the start:
> As far as I can tell, these problems are problems because people are implicitly carrying around a pre-agentic cost model. Ten years ago, the kind of person who implemented a new compiler because their compiler was slow was the kind of person who started programming a new 3d game engine when they wanted to make an indie game, i.e., the kind of person who never ships anything. Most productive programmers I know still have an aversion to this kind of thing, even though it is now actually reasonable to implement this kind of thing now.
Of course, at the moment everything’s still a bit very much difficult to tell, but “productivity”, at least in applicable areas of work, might really come down to who can let go of the most understanding the easiest? Incredibly thought-provoking stuff.
| the level of expertise it takes to distinguish a counterfeit from the real thing has been increasing as models improve
This is Bainbridge's 1982 paper Ironies of Automation restated.
https://en.wikipedia.org/wiki/Ironies_of_Automation
https://ckrybus.com/static/papers/Bainbridge_1983_Automatica...
From the abstract
> This paper suggests that the increased interest in human factors among engineers reflects the irony that the more advanced a control system is, so the more crucial may be the contribution of the human operator
Makes sense.
Does it? Couldn’t you steelman this argument and claim the opposite, that the more advanced a control system, the more likely it is that human error is what causes its malfunction?
Those don't seem like mutually exclusive statements to me. In fact, they feel like two sides of the same coin. It's almost by definition that more advanced control systems are exactly the ones where tractable sources of error (mechanical, environmental, etc) have been eliminated, controlled or otherwise accounted for as much as possible, to the point that the only remaining likely problems come from where the system interfaces with humans in some way.
Which makes those interfaces the most critical parts of the system.
> the level of expertise it takes to distinguish a counterfeit from the real thing has been increasing as models improve
Model produced code compiles the same as human produced code. My kids would call this “copium”.
> but “productivity”, at least in applicable areas of work, might really come down to who can let go of the most understanding the easiest?
That is, genuinely, crazy talk. That is what addicts would say.
But it is intellectual-washed to sound like the most reasonable thing to say.
It is not. And there is nothing thought-provoking about the mere act of doing heroin.
There might be something in the system and mechanics that lead to people getting the idea to do that, though.
___
We can engage with these ideas and there might be some merit to them, but not with this frame. This frame will (not hyperbole) kill us all.
People will die due to (poorly-engineered) systems understood by no one, failing in completely incoherent and unexpected ways - and those people may be you or your loved ones.
> That is, genuinely, crazy talk. That is what addicts would say.
I, well, agree. Of course the statement is nuts, especially so once you consider issues like vendor lock-in, the oncoming atrophy of one’s own skills, etc. But if you define “productivity” as short- to medium-term results only, is it really that inaccurate?
If you define a problem in a way that leaves out like 80% of the constraints, then solutions get trivial, yeah. That's I think the generalized way of phrasing this specific dysfunction of our industry.
The problem is that it still uses the and benefits of the legitimacy of the word of "productivity" there. Meaning that it is harmful to us to put it that way.
Good point, not having to plan for the long term probably has an outsized effect on a given level of difficulty/effort required.
Maybe it’d be interesting to see what could be achieved responsibly by motivated people with models in a world without “feature pressure”/competitive stress (which’d include e.g. no security race and infinite attackers and exploits to keep up with)?
>>> but “productivity”, at least in applicable areas of work, might really come down to who can let go of the most understanding the easiest? >That is, genuinely, crazy talk. That is what addicts would say.
People demonstrably said the same thing about the inventions of... Writing! and Mechanical/Digital Calculators! Yet here we are!
Time is a flat circle!
>> Ten years ago, the kind of person who implemented a new compiler because their compiler was slow was the kind of person who started programming a new 3d game engine when they wanted to make an indie game, i.e., the kind of person who never ships anything. Most productive programmers I know still have an aversion to this kind of thing, even though it is now actually reasonable to implement this kind of thing now.
Then there is these other programmers who built their own operating systems even though their OS professors are claiming publicly their design sucks, who built a local database from scratch in C because his use case did not need the client/server stuff but just all the other relational goodies, or who built video streaming/transcoding software from scratch just because they have ideas and then the world changes around their ambitions.
I think it is not enough to be ambitious or have the technical ability if you want to build something for others to use. You definitely need deep commitment but also perseverance in the face of social adversity. I think this is what saddens people with automation of the technical aspects of programming. It leaves the more psychological aspects as challenges which are not easy to face all the time.
> It leaves the more psychological aspects as challenges which are not easy to face all the time.
Great point. Coding used to be a natural forcing function for prioritization of features. When coding is basically “free” you have to do the harder job of saying no to things, deciding exactly what to build, etc…
Yep - in a world where content is effortless and abundant, is less actually more?
I would say, yes!
> in a world where content is effortless and abundant, is less actually more?
Yes, if its less and good. Yes.
> who built their own operating systems even though their OS professors are claiming publicly their design sucks, who built a local database from scratch in C because his use case did not need the client/server stuff but just all the other relational goodies, or who built video streaming/transcoding software from scratch just because they have ideas and then the world changes around their ambitions
> perseverance in the face of social adversity
Sometimes you're Fabrice Bellard, sometimes you're Terry Davis. That is, for almost any idea you'll find random detractors, but some ideas really are generally useful and some are completely nuts. One of the negative externalities of AI is that bystanders are confronted with a lot more cases where they have to work out which one this thing is.
“might really come down to who can let go of the most understanding the easiest”
I suppose it depends on the “applicable areas of work” but I would argue the opposite. As in all things, when you level the playing field, in order to stand out you have to know the rules to understand how to break them.
Imagine a fully autonomous Ad agency that creates unique ad campaigns for companies. And have it controlled by someone who “let go of the most understanding.” For instance, maybe a business minded person with excellent technical skills, but doesn’t know how to copy-write and has no visual design training.
Even with the strides in AI advancement, this is an impossibility and I’d argue it won’t happen within the next five years even. You need human agency, and you need human workers who are experts in their field.
> Perhaps almost all or perhaps even all of the software skills I have will become irrelevant in the near future. But we're not there yet. At the moment, programming skills seem higher leverage than ever before.
While this iteration of technology certainly _feels_ different, it seems like for now we're still coming down to "it's useful to know how to build a car, but when you need to put food on the table, know how to fix it and make it do what you need it to do".
Your argument in essence is productivity ~ not caring about the output. Which is sorta unhinged.
Well, in retrospect I don’t like this framing all that much either, but I think I came to this conclusion due to assuming
not caring about the output ~ more output
… which may or may not be useful in the long term, but definitely does help a shorter-term “productivity” measure.
On the other hand though, there’s a pretty damning example against my case here: Say a manager of individual contributors – which is what the situation of an individual with sufficiently smart models and tooling may almost boil down to – didn’t directly care about the output anymore. We’d most likely call that being unhinged/unproductive, too…
It was late at night and I primarily wanted to link the somewhat relevant article, maybe I should have left everyone to reach their own conclusions instead of adding some poorly-worded unfinished thoughts. :x
> Of course, at the moment everything’s still a bit very much difficult to tell, but “productivity”, at least in applicable areas of work, might really come down to who can let go of the most understanding the easiest? Incredibly thought-provoking stuff.
I cannot see it being a good thing to "let go" of understanding. If you don't understanding anything, then why would anyone keep you around? An example hypothetical situation is, if I have a colleague that outputs a bunch of productive things, but they don't understand it, and I do, then of what use are they? I have LLMs too and also have the understanding.
It isn't clear to me why people are so comfortable with letting go of understanding. And it's a bit bizarre because it always seems to be framed as "you're going to get left behind if you care about understanding". I also never hear or read anyone who has any real and deep experience say these things.
My read of what is being suggested is that the level at which one should aim to understand and be comfortable with is slightly more abstract than reading the code and reviewing every line of it. It is like how one is comfortable not understanding the byte-level code or assembly language code. I know that this analogy doesn’t hold very well because compilation is a deterministic process while llms are not, but that is precisely the point being made. “Trust” that llm will produce the right output, even though the output might not be reproducible word for word the next time you give the same prompt.
To your point about becoming redundant without understanding, not understanding things will be a problem when an issue arises and the other person isnt able to handle it the right way. However, if llm can address the issue, does the fact that they did not understand the details really matter? From the perspective of a person who gets to decide who gets fired, would they fire someone who is productive but sometimes introduces bugs that they quickly address or would they fire someone who is comparatively slow but rarely introduces bugs? The answer probably depends on the domain, and how costly a bug can be. For most of us, the work is not so critical that you cannot afford bugs. So, productivity wins.
> comparatively slow but rarely introduces bugs
If you are suggesting humans before LLMs rarely introduced bugs, you probably haven’t got the latest models to review your old code. It’s shocking how many bugs we had before but couldn’t see!
Don't shoot the messenger, but maybe the idea is this:
You're smart, and you know what can go wrong, so you use models slowly, and you check their work. Your bet is that it will make a bad mistake.
Your coworker is reckless, they don't know or care what can go wrong, so they use models quickly. Their bet is that the model won't make a bad mistake.
If the model doesn't make a bad mistake, your coworker is creating more value to your employer, and your employer doesn't really care if you understand the code. They just want something that works.
Over the last year, it sure seems like models are making less mistakes, so the expectations on the bets are changing too.
Right or wrong is this the future we want? Shouldn’t we say and do something?
For instance: we have calculators. We’ve had them for decades in their “current” form more or less. They are readily accessible and far more reliable than LLM’s - if you a problem 1000 times it’ll be right 1000 times, the rare bug or glitch notwithstanding. But we still teach people basic arithmetic. We teach times tables. We didn’t just offload basic math.
I agree we don’t need to know assembly to use a computer but there is a balance here and companies do not care about a balance.
I think we mostly DID offload basic math, for better or worse.
For software engineering I don't know the right balance.
I've used the "we mostly stopped caring about machine code generated by a compiler; maybe we should stop caring about LLM-generated code as well?" analogy myself.
But even I have to admit there's two big holes in that analogy: if I use a compiler to turn C into machine code... (a) compiler bugs are orders of magnitude rarer than LLM-generated bugs (b) my C code remains the source of truth, or at least source of intent. Whereas with LLM-generated vibecode, there's no source of truth, just an ever-evolving mudball. I (kinda?) like the ideas of "documentation as source of truth" and/or "tests as source of truth" ideas, but I don't think we're nearly at the point of viability.
TL;DR yeah I think actual understanding of what LLMs produce is gonna be important for the forseeable future
Adults can, under most circumstances, do some limited slow arithmetic but are generally piss-poor at it, notwithstanding the occasional rainman, and only rely on it in trivial matters.
You see cashiers summing groceries? Salary calculations on paper? Tax, inventory, accounting, you name it? Nobody does any of it by hand.
We most definitely offloaded basic math.
I didn’t say we were universally good at it but in many countries we are (more or less) all exposed to it and taught it in school. There is a baseline even if we are far from mastery, and frankly a cynical hand wave of “well everyone’s bad at it” is not a real argument. In the US it is very clear that people are at least exposed to it and if we weren’t we would be operating very differently. I went to dinner last night, everyone did the tips in their head right there at the table. The average person can do some simple multiplication and division in their head. It’s not a mysterious script done on a box that they repeat the output of. Most people even if they’re bad at it understand the fundamental concept and, gun to their head, could probably do a series of very basic math problems with pencil and paper.
I imagine you and most of your peers can calculate a tip or at least approximate it without pulling out their phones yes?
This is just moving to a higher level of abstraction. Nowadays people don't verify the compiler generated assembly code anymore. We don't care if our source generates a few million extra instructions. Source code is just an intermediate between specs and binary at an abstraction level decided by the developer. We do requirements -> code -> assembly.
To resolve this we should aim for a new intermediate format that is more high level than source code, the generated code is too monotonic to properly review manually. Something like requirements -> detailed specs for llm -> code -> assembly . Then let one agent implement the specs and have another review it. If we stay at the current abstraction level, humans become the major bottleneck.
I think your hypothetical is incomplete. What if the person that lets go, generates more productive things then somebody understands? In this case I know a lot of ppl will value the producer of value more. I always pushed the bounds of learning more about the underlying stack or problem, and always found them in no time.
> It isn't clear to me why people are so comfortable with letting go of understanding.
It really does feel like people have embraced rolling around in the cognitive version of those Wall-E chairs.
I like to build software products. I don’t like to build software, but it’s the only way to build software products.
“Understanding” is quite an overloaded word.
Understanding architecture, robustness, the “crumple zones” of a system? I’m all for it.
Taking three days to “understand” how postgres transactions work under the hood for your generic SaaS? Uh..
I have seen a lot(!) of the last type of “understanding” and it’s infuriating. The same types will often not understand the business, its stakeholders, the architectural constraints beyond the technical, etc. Understanding takes many forms.
I feel like the real value is in probing the extents of your own understanding. If you don't understand something it's doing, you can ask. You do kind of move up a level of abstraction, from the IC to a PM or team lead. So maybe you don't know what every line of code does but if you understand code and how to drive you can trust that it's doing the right thing and you don't have to be as deep in the weeds of every refactor etc.
You do have to keep a sense of when it did the wrong thing and how that was a valid interpretation of what you asked and how to better specify what you want. But that's how it really just shows you the gaps in your own understanding of a problem.
Also I think there's huge differences to the kinds of tasks and workloads people are throwing at it, leading to wildly different perspectives, but everyone's talking about the same "Fable".
> Also I think there's huge differences to the kinds of tasks and workloads people are throwing at it, leading to wildly different perspectives, but everyone's talking about the same "Fable".
This right here. Humans are really prone to over indexing on a handful of personal experiences vs a representative sample. And once someone has formed an opinion, confirmation bias helps cement it, despite evidence to the contrary. Reality is probably somewhere in the middle.
That's where I think benchmarks are a huge red herring. Everyone wants a simple performance evaluation of a repetitive task but that's missing the point entirely IMO. Just try talking to it, asking questions. I spend so much time just reading responses. I have to like really think about how to specify the question I'm asking, fire it off, and I often can't wait to see what it says and what I'll learn. How anyone thinks it's making us dumber is crazy to me...you're missing out if you're just trying to offload thinking.
> If you don't understanding anything, then why would anyone keep you around?
They don’t keep you around, of course. The better these models become, the worse the human job markets get, I’d predict.
You're misinterpreting what they mean as understanding and inserting your own doomer definition where this imagined person has been stripped of all ability to investigate a result or ask questions.
I too love to downvote and edit instead of admitting that I'm talking and acting in a way where nobody would want to be my coworker anyways. Where exactly are you demonstrating that you're doing anything to justify being kept around if that's such a big concern for you?
> I like Max Bittker's framing that models are very good at counterfeiting and the level of expertise it takes to distinguish a counterfeit from the real thing has been increasing as models improve.
Is it just me or is this analogy completely missing the point of why we care about detecting counterfeits in the first place? Poor craftsmanship of all things is not really the reason counterfeiting is illegal!
Beat me to it. It’s wild this needs to be said.
I'm struggling to understand the hypocrisy of private property advocates advocating against private property. Are we doing "intellectual property is protected by law" or not? We're allowed to ingest all of human FOSS and StackOverflow without attribution and use it to sell distilled human knowledge by the token, but not use the distilled human knowledge to reverse engineer and FOSSify proprietary software? We're allowed to ingest all the art on Twitter and use it to generate and sell corporate clip art without attribution, but we're not allowed to generate micky mouse/bugs bunny yaoi fanart?
I mean, the answer is obvious: the Luddites were right, these tools (not just LLMs but private property, copyright, and capitalist mechanisms) are only tolerated to exist so long as they serve the needs of Capital, but the hypocrisy still frustrates me.
> I'm struggling to understand the hypocrisy of private property advocates advocating against private property. Are we doing "intellectual property is protected by law" or not?
The most rigorous "private property advocates" don't recognize intellectual property as a legitimate form of property in the first place: private property is a natural phenomenon that emerges as a solution to the problem of physical things being inherently exclusive in possession and use, and IP is a creation of positive law meant to artificially apply the methods for solving that problem to another, unrelated question that isn't part of the same problem domain.
> I mean, the answer is obvious: the Luddites were right, these tools (not just LLMs but private property, copyright, and capitalist mechanisms) are only tolerated to exist so long as they serve the needs of Capital, but the hypocrisy still frustrates me.
Abstract concepts don't have "needs", and no one is currently in a position to tolerate or not tolerate anything -- no one in particular is actually in control over anyone else's activities. The cat's out of the bag, and no one has any capacity to put it back in.
> these tools […] are only tolerated to exist so long as they serve the needs of Capital, but the hypocrisy still frustrates me.
Great thought. Question is, can those who do the tolerating (in the interest of capital) even effectively prohibit/prevent access after a certain point, especially now that the cat is not just underway, but well out of the bag?
Assuming the current trajectory of a yearly factor of 10–100 in size/cost reduction [0] holds up, won’t the technology just become cheap enough to be functionally available to everyone?
[0] https://jyn.dev/tokens-too-cheap-to-meter/
My assumption is that frontier models will be constantly increasing in intelligence, solving far more difficult problems, while the rest of us do the LLM equivalent of scratching through the dirt with wooden hoes.
Thinking about just the domain of the stock market only: getting a "win" on it means getting a leg up on everyone else participating. Best example I can think of of a pure zero sum intelligence contest. Programming UIs might have a point where you just can't get smarter at it, but in an intelligence contest like a stock market, I don't think there's a ceiling.
In "Accelerando" this was covered by the invention of "Economics 2.0." AI agents also got into a lot of lawfare in that book.
I will bookmark this comment. It is a very clear summarization of what is happening.
Hah - same I screenshotted it. Ooof , put into words what’s been silently ruminating in the back of my head.
Google this: do not delegate understanding
You will find a lot of articles from before LLMs about not delegating understanding of your core system or business. You could say that the software is not your core business so you don't have to understand it. But if the software is any good then it should be taking care of a large part of the business. So software is your business and you shouldn't delegate understanding it.
I like this and the warning it entails, on the flip side of this I have found that you need to be able to delegate and trust. If you demand on understanding everything and doing everything yourself the world will grind you down. It's a big place and sometimes it is OK to delegate what is not part of your core business it let's you focus on this. Not everybody needs to be able to cook 5-star meals, you should know how to heat up some pasta, but don't beat yourself up if you can't bake a cheese soufflé.
If your objective was understanding rather than, say, getting a new operating system running, couldn't you use the stronger models to produce artifacts and then ask them (or weaker models) to explain those artifacts to you? Probably the stronger models would be able to produce artifacts that were more worthwhile to study.
Maggie Appleton suggests some ways that AI models could do a better job of communicating about artifacts to people in https://maggieappleton.com/planning-agents, although she's focused on the use case of the person deciding what changes in the artifact to request from the model.
Thanks for sharing that link btw, it was great! You should post it on HN for more people to see.
Last time I submitted a link it was immediately flagged to death, although I have to use an Incognito mode tab to see that. I'm not wasting any more of my time on that.
The only link you've ever submitted that disappeared was for linkedin.. which is a banned site. No one flagged it, HN just rejects it.
Someone found a better URL for that (Fabrice Bellard) story (16 points, 10 days later) https://news.ycombinator.com/item?id=49996383
Unless there was another one you later deleted?
That's the one, and it's one third of all the links I've ever posted, or, probably, will ever post. What you say may be true, but it was still immediately flagged to death, regardless of how that happened, and the HN software is still trying to be sneaky about it. If my contributions are treated as spam, I'm not going to persist.
Certain domains being blacklisted is completely understandable if you consider the history of the site and culture(s) of the users. It's nothing to do with it being "your contribution".
That could be handled with an error message instead of assuming bad faith on my part. Given that it was handled (and continues to be handled) with sneaky shadowbanning measures, I have no interest in attempting to circumvent the spam filter by submitting future links.
“Wasting time” by copy-pasting link title and url? Sounds excessively optimized time management to me. Save 10 minutes elsewhere, and you got 50+ submissions worth to spend!
Most submissions never receive any upvotes. They’re all “wasted” like that, even if not flagged.
It's been posted 4 times in the last 2 months: https://hn.algolia.com/?q=maggieappleton.com%2Fplanning-agen... No comments though
When you get good enough headphones, you can start to hear the limitations of the equipment stuff was recorded on. I wonder if this is like that
I wonder if it's like this:
https://www.tomshardware.com/speakers/in-a-blind-test-audiop...
this is about cables which often were and still are snake oil in the audiophile world. headphones do make a difference. I have a hard time using my airpods and others after experiencing a pair of Sennheiser HD650 I've gotten myself as a christmas present couple years ago
yeah lamp cable will work. There's no room factor in a speaker so like 99 percent of the quality of the sound comes from the fidelity of the transducer and its resonance characteristics in electrical/physical suspension. Then the next .9 percent is from the source equipment, and finally the last .1 percent is from the dac/amp stuff. Exaggeration here but not really
Ok this point and pacing the frontier: personally i would be ok with freezing models at this point. Opus 5.5 does everything i need and i suspect that from this point onwards the improvements will be beyond my understanding and just introduce existential risk.
“Nobody will ever need more than 640kb of RAM”
the cost per task still needs to fall by a 100x
It very much applies to science as well. I've got a workflow doing an entire PhD's worth of work time in a single month subscription of $100 codex. There's zero chance i go through and understand what the hell it's doing, but for tackling problems which are accepted as "impossible" it's extremely useful for gathering data on every possible approach. It hasn't had a break through, but it's certainly recreated everything anyone's tried before.
Hmmm, I took it the opposite way, actually.
There are many ambitious things I have tried to work with Opus 4.8 and Opus 5.5 on, and it just can't get there. It always falls back to some tried and true method or some valley that it gets stuck in. In those times, it was impossible to push it forward into more creative and exploratory thinking based upon my relatively advanced pushes and contextualizing. So I took the commenter's position that Fable 5.2 will respond better to that type of pushing. But that type of pushing actually means you know more than those who are happy enough using the lower models. So that seems like the opposite of what you said.
I'm interested to hear about what kinds of things you've been trying and what the stuckness looks like.
I'm closer to a Product Manager so I'm not sure how useful these examples will be for you.
One important constraint is this topic of intent (also referred to as alignment). Specifically, when a user requests a feature or change, there is MOUNTAINS of implied context and intent in their comment. Some cultural, religious, social, economic, base on work and personal experience, their use cases, their values, what they intend to get done, etc. Every time these models get smarter, they get a little better at understanding this context. It's combination of reasons. Some of it is parametric. Some is inferred. Some is about selecting the right data to examine and research. As these models get smarter, I spend less time time iterating and educating the model. They reach the intent faster and better.
From what I hear and experience, this is also true in the coding space. LLMs are basically another abstraction level on top of formal languages, which sit on top of compilers and machine code and etched instructions. Smaller and dumber models require a lot more hand holding to achieve the same results. They're less capable of inferring the gaps. Less capable of extrapolating potential negative outcomes of the change they're making. Less/less useful parametric knowledge. Less capable of selecting the right data to use to make the right decisions. They do what they're told.
This has been less of an issue for coders who religiously document, understand the codebase and architecture intimately, and demand to be kept in the loop on every change. So seniors often do not see the utility of smarter models. They know exactly what they want, how to ask, the context to provide, and how to assess if the change worked as expected. The utility is for the 99.9999% of the world which is not a senior developer.
While what you say mostly rings true to me, I don't see how it could result in the valleys of stuckness I was asking about.
However, your statement that there are only 8300 senior developers in the world does not seem correct to me. That seems like an unreasonably high bar for that title.
Maybe I should remove a 9. I meant to imply that there are a lot of people who benefit from smarter models.
Maybe so, yes. It depends on your value system.
They are generally things I don't particularly want to share, as I view them as ideas close to my heart. The general pattern was what felt like redirecting from the harder problems and an unwillingness, if we can call it that, to go there. In some sense though, it's hard to understand how these more advanced models get around that, because these models are tuned to the most probable answer. When you come at them with a novel idea, they always seem to water it down in the end.
I have also found this — once you get out into some really novel workspaces they will revert back to replication of adjacencies that already exist. Fable 5 and 5.1 in particular will talk around the issue in a way that no model prior seemed to.
I think the issue is fairly fundamental, perhaps because of tuning, perhaps even beyond what they are tuned or trained on, because the bulk of what they can go find and ground into seems to drag them heavily toward implicit biases that split around categories I can’t quite discern. The category boundaries sometimes look like academic fields, but sometimes they seem like more fundamental attractors.
I second (third?) this. Previous models were more willing to explore the unknown. Those new ones just try to Google search and latch onto any adjacent solution they find.
That's understandable. Thank you for sharing as much as you have.
The dev version of "I'll fix him"
Are you saying that you're seeing stuckness of that form, or are you paraphrasing some kind of misunderstanding of my question?
It's a reference to a joke, of a stereotypical comment left on Romantic Relationships topic online forums. As in, "this man is so terribly unsuitable to everyone. But I'm special, and I can fix him and make him the perfect boyfriend."
Sure, but it's unclear to me what its purported relevance is in this context.
> We are pretty much at the point that increased model capability will only have utility to folks that are willing to let go of even more understanding of whatever the model is producing.
Given that reading someone else's code is slower than understanding your own code, doesn't that basically cover everyone that adopted coding agents?
> The best models can/will produce a year's worth of output that would take several days to weeks to properly understand in the span of an afternoon.
This is appalling. Are you really really going to trust a YEAR's output from a machine? Are you going to read and understand all that? Will you be able to maintain and extend it to your will? I truly doubt that. Drugs do harm and are hard to quit. Good luck.
As one poster replied above, there is code one doesn't need to understand if what you care about works. A good example is when I first tried a couple of local AI models on my highly unusual inference hardware.
It ran like a dog. Few nights of running Fable (later Opus 5.5) with a 7 strong team of DeepSeek v4.1 Flash 1017 (don't mistake it with 0731 that is crap) and llama was running 10x faster and vllm was running even faster when before it would not work at all.
I've run similarity and needle tests and the results are exactly the same content wise. But 10x faster. I've relied on this for a month now and I'm pretty happy with how it runs.
Do I now need to learn CUDA Kernel programming to understand how it was done? I don't think so. It's for my personal use. Not for a paying client so no warranty is perfectly fine.
But I also use a lot of AI at work in an I dusty where people may die if the code is buggy. So I absolutely verify and understand everything.
There is a middle ground too. Understanding the architecture.
For example I used a bunch of DeepSeek models supervised by Qwen to build adaptive-acl. A pi coding agent plugin that deterministicly checks if every tool used by an ai during an agentic session matches user intent to give permissions for those actions (reading/writing data, network access etc), as a part of this plugin I run an llm, a classifier that runs over user's prompts and extracts these permission grants then saving them in a format deterministic layer can process.
The design includes massive number of assumptions and compromises. For example, if a user allows writing this implies permission to delete and reading if what was written etc.
The thing also includes over 4 thousand actual instances of tool use preceded by "permissions talk" from my live sessions and that whole collection can be run to test for regression etc.
At some point during one night the AI had a regression and decided the best way to resolve it would be to add regex based language processing elements to the deterministic layer (a huge no no, this has to work regardless of the user speak English or Swahili, also regexing natural language is a disaster waiting to happen).
I knew enough about the architecture to notice that and burn it out with a hot iron.
That is the problem with AI. It sometimes solves problems the easiest possible way causing massive issues for later. So if one uses it for important stuff it is good to understand the architecture and the plan of what it is going to build at the very least.
I see this mindset a lot from people who I can only presume have only ever been individual contributors.
Having run teams, as much as I love getting into the details of code I care about, in the 30+ years I've been in this business, I've only ever even seen a tiny proportion of the code my teams rely on.
And the non-technical people in these businesses will have seen none of it.
They all take it on a combination of trust and verification of outcomes that the code does what the developers claim it does.
Increasingly we're getting to a point where the output from these machines are more trustworthy than an increasing proportion of human developers, and more importantly: Increasingly often good enough.
Especially as more and more of the code becomes throwaway.
I have a side-project that is profitable, and that is profitable only because I haven't needed to spend time reading the code. If the code turns out to be unmaintainable? Doesn't matter - it's so cheap code that it is reasonable to just throw it away, and start over based on the lessons learned. It enabled me to take an opportunity to profit I wouldn't have been able to profit from if I had to set aside time I didn't have, so even if it was to totally fail in the future as a result of issues with that code it doesn't matter.
As the cost of good enough code falls towards zero, how we think about which code matters and how much we need to understand it will fundamentally change.
Increasingly, a lot of code is even just part of a conversation with an agent to me. It's throwaway, for the duration of a session. Who cares if it lives on?
I don't trust or need to trust most of the code I build any more. I trust the measurable outputs. The verification pipelines we build. The feedback mechanisms we build. Those I need to read and understand.
That's no difference than when working with a team. The only thing that is different is failure rates. And at this point, for some thing the machine failure rates are still higher, for others it is lower.
> Fable is exceptional at drawing. It’s ability to render an image with JavaScript actually has me questioning the value of dedicated image models
I’ve been shifting away from using more traditional GenAI models in favor of just giving LLMs access to a custom drawing language (LOGO-like) to generate imagery and its been going shockingly well - Opus 5.5 in particular.
I put together a simple LLM driven skill that creates old adventure game (EGA) environments from simple pencil sketches and it works really well. And being procedural, they're much easier to make surgical changes to without degradation.
https://kq-styles.specr.net
amazing, love the idea and the results <3
Oh boy this takes me back to the good old Sierra days.
Just to clarify -- the images on that site are all generated from LLM-written code? It's very impressive if so. I would never have guessed these weren't human-made from the visuals alone.
Yeah it's entirely LLM driven! If you look at the repo, I explain more about how to use it. Basically, you give an agentic coding harness capable of tool usage (Opus 5.5, Qwen 3.8-27b, etc) the skill which explains the drawing system: simple things like how to set the colors, position rectangles, draw polygons, etc. and it uses p5.js to render them.
It has three different modes:
• You can simply describe a picture, and it'll automatically generate an image using procedural code.
https://kq-styles.specr.net/#/scene/custom/elk-valley
• You can ask it to paint over an existing image. That's what those examples show, like me creating EGA-style equivalents of various scenes from the original Myst.
https://kq-styles.specr.net/#/scene/myst/r01-c00
• The last way is to give it a rough pencil sketch. It's the mode I use the most since it's very valuable when building an adventure game to position objects in a room.
https://kq-styles.specr.net/#/scene/sketches/12-crossroads
Wow. Genuinely awesome work. And fascinating to see the drawing process.
Thanks! I started this whole thing because I recalled reading an old interview with Sierra’s original creators, Ken and Roberta Williams, where they talked about how they needed a way to have graphics for their games without saving large bitmaps which would make it difficult to fit on a single floppy disk.
So instead for each room that needed to be rendered they just saved a series of vector instructions that would re-render it in real time. On slower machines, you could actually see the rooms pop in piece by piece. I wanted to kind of approximate that same aesthetic because it's just super cool!
Really good stuff, thanks
That's very cool. Great concept.
Tell me more about your custom drawing language. I'd been thinking of doing one myself.
Beautiful! Thank You For Making And Sharing!!
sir, Twin Kingdom Valley does exactly this on a ZX Spectrum
Thanks for the rec! I unfortunately don't have a Speccy, but I did find an emulated version of the BBC Micro port of Twin Kingdom Valley and can definitely see it drawing in the various pieces of the environment.
https://bbcmicro.co.uk/game.php?id=89
I think the approach of using simple procedural vector graphics was relatively common back in the days when you had to optimize for either disk space or speed. Sierra's creators talked about how they opted for procedural graphics to fit their earliest games on the old 80 KB floppies.
Charming!
Do I understand correctly that the drawing language is something you came up with yourself? Amazing quality if so!
So basically I gave an LLM access to the user manual for the original LOGO drawing language, along with several screenshots of various scenes from the original King's Quest I.
Then I just guided it to produce a procedural variant specialized in creating new EGA-style scenes for use in creating old-school point-and-click adventure game.
https://people.eecs.berkeley.edu/~bh/usermanual
It's all open source.
https://github.com/scpedicini/agi-stylist
This is awesome! You can see the core of it here: https://github.com/scpedicini/agi-stylist/blob/main/skills/o...
The speed part is funny. I felt like with Fable 5.1 I had time to digest what I had prompted and was waiting for. Now I feel like, I am running behind or like it is going to take off too far. This is great for closed prompts "make a soduko game" but hard with running context queries, because I feel like I am creating more information debt than I want. I'll be fine, just talking out loud.
Does Fable 5.2 outperform Opus for data science work?
Not just "iterate on this XGBoost model" but actually looking at a problem and designing and carrying out a short-term R&D program including probability modeling, model fitting diagnostics, how to respond to various kinds of results, how to interpret/reason about what it finds, etc. That's really where I saw all the models perform poorly, and where I saw the first big step increase from Opus 5 (pre-5.5) to Fable. But then Opus 5.5 came out and I haven't needed Fable since.
Mind you, I'm doing slightly different work (and I'm very much "in the loop" still) than when I performed that side-by-side test. But it's really amazing what Opus 5.5 can do. I now have a powerful data science AI assistant that not only can write any script I need it to (including some scripts I didn't even realize I needed), and it has an encyclopedic knowledge of everything ever, and it constantly checks me on oversights, it handily derives expressions and clever model reparameterizations that aid, it has great insight into model behavior, and it has an amazing ability come up with ad-hoc experiments and tests that would have taken me 2x as long to even think of (let alone implement by hand).
It's not at the scared-to-lose-my-job phase yet, but it's become a tremendous force multiplier for a smaller data team.
I'm teaching "Data Science" to computer science majors next spring, and I'm overwhelmed by the prospect of what to teach them in the age of AI. It seems like Fable already way outperforms anything I could historically expect my students to do at the end of the semester, and perhaps even outperforms me. I can describe a vague analysis on the data, and it will handle every step: transformation, preliminary analysis, formulating more precise questions, creating visualizations, running statistical tests, interpreting results, thinking about stakeholders... And in some of those steps, it finds things that I might not have considered.
I wish I had a bigger N so I could rely on that experience a bit more, but it sure does feel like I'm not really contributing anything in terms of "Data Science expertise" in my own research right now.
You need to teach your students why all of those are important and how to tell if the llm is giving them what they need.
And especially where the fuzzy edges and blind spots are—the limitations of the methods, and the aspects of problems that are essential to the question but hard to quantify or test or reproduce.
I feel like a lot of people are going to put the same faith in their new quantitative superpowers as psychologists did in their 95% confidence intervals [0], and managers did in their “scientific management”/time-and-motion studies, and American political pollsters did in their 2016 presidential prognostications, and Rudder’s OKTrends blog (delightfully) did in calculating the answers to romance [1]…
[0] https://www.psychologytoday.com/us/basics/replication-crisis
[1] e.g. https://news.ycombinator.com/item?id=35331762
If it wasn't for grinding all of the hard problems and hands-on projects and various programming tasks I had to do by hand over the years, I wouldn't be able to keep up with the agent, and wouldn't be able to get good results out of it, because I wouldn't know what it was doing and wouldn't be able to steer it appropriately. My schoolwork 15 years ago was more than formative, it remains critically foundational for my job work today.
Based on the other comment it sounds like maybe I need to give Fable another chance, and then I suppose I won't have to steer anything at all. But at the end of the day I am not paid to be a meat proxy, I am paid to deliver things for the business that employs me, and the things I deliver are my responsibility, not Anthropic's.
So I say, embrace AI as a learning tool, let students use it to bypass programming on assignments where the programming is of secondary importance, require oral defense/presentation for projects, make homework worth a much smaller fraction of final grade, and make all of your tests in-class and hand-written.
The reality is that the entire nature of education has to change to account for cheating becoming essentially free and virtually impossible to detect. Grinding through a problem set in the library is still tremendously important, but it's going to be harder to motivate students to do it without just cheating. Some students are conscientious and will do it anyway. Some won't. The hard part is going to be crafting curricula and exams that separate the two kinds of students. Fortunately you share that problem with just about every other professor in the world right now, so at least you shouldn't feel responsible for solving it all yourself.
Relevant anecdote: I used Wolfram Alpha to cheat on my calculus homework in freshman year, and then I got my butt kicked in the exam and a C final grade that never quite averaged its way out of my GPA. That felt horrible and it was a strong motivating force to get my shit together, but that's only because I'm a fucking nerd and I really cared about learning in school and making a career out of what I learned. On the other hand, I really was struggling with the material, which is why I tried cheating in the first place. Maybe if I had today's AI I could just ask it to help me learn better.
Plus it worries me that people will no longer know how to do anything basic and we will at some point in the future be slaves to AI if we aren't careful.
Is it reasonable to take up some of the class with statistical basics and intuition? E.g. you just referred to a bigger N; I don't know what level your students are but do they have a basic intuition for notions like "variance of estimator"? Or is that another class? I think my point is that even if they can nowadays press a button and have the most amazing reports and visualizations produced, that doesn't give anyone any actual understanding of the theoretical side of data science.
From my experience working with data scientists, it outperforms humans on an extreme level. I’ve seen so much bad data science, and here I have a model that can do all of that from one prompt. It is getting really good on finding caveats in data too.
did you find tricks to turn LLMs into super-assistants so students can learn deeper ?
i also wonder if they're not losing motivation due to the fact that the llm can race through everything so swiftly
I'm so glad I learnt it before LLMs. I can't imaging trying to get good at Pandas/Polars coding nowdays when so much of my analysis is "Fable, please do this for me, k thx bye"
I mean I found Fable 5.1 outperformed Opus 5.5 for data science, so I expect 5.2 to.
But then I'm someone who hasn't found Opus 5.5 good for much beyond web UIs and related frontend development. I find it more forgetful than GPT-5.6 or 6.1 or Fable, so despite having a 1M context window it gets lost in a big codebase or goes of target. Opus 5.5 doesn't manage agents particularly well IMO.
I'm surrounded by people raving about Opus 5.5 and God only knows why I'm not getting their results.
I just tried Fable 5.2 side-by-side with Opus 5.5 on the same prompt and I think you're right, n=1 but with strong signal.
Fable got right to the point, had cleaner/better responses and reasoning for its conclusions, got less distracted with secondary concerns, and still showed signs of the cleaner writing style compared to Opus, which improved a lot with 5.5 but can still be hard to parse sometimes.
I will keep using Opus for implementation tasks (e.g. building web apps to explore model predictions) but I think I'll try using Fable this week for the real DS stuff. Thanks!
Have never noticed a difference between Fable and Opus on coding tasks but design/architecture is night and day different. I usually don't bother with designs with Opus since it tends to screw things up, hallucinate, make bad decisions.
With (obra) superpowers, Fable writes a plan and delegates to Opus which works well. Fable is much better at sorting architecture in a conversation rather than me creating a giant prompt for Opus.
In using Fable, I too went to examine my "prompt ambition" (can we make this a thing?). You have to abstract a bit to think of what it can do for you.
Opus 5.5 has been good enough for most of my coding, but I don’t know if Fable 5.2 is better for architecture work, like planning how a system fits together or designing the data model. @nijave says it’s night and day for design. Has anyone else seen that? (I don’t have access to Mythos, so I can only compare Fable and Opus.)
At this point, I don't really see much difference in output quality between Opus 5.5 and Fable 5.2. Although, Opus 5.5 seemed to just go on auto pilot mode without give tasks much thoughts unless you give it tasks specifically each time. Fable 5.2 seems to think more on its own.
And none of it matters ... because just as soon as you are happy with things "balanced across models" Anthropic will nerf the models you were happy with, and force you to use new and more expensive ones.
> I prefer the images I’ve gotten fable/opus to make for me over those from gpt-image
What kind of images? I can't imagine getting photorealistic or highly detailed digital art out of LLM writing drawing commands using JavaScript.
I just saw someone post about getting Claude to write code that renders individual brushstrokes and the results were varied and impressive. Can't prove they actually had Claude do it, but I believe it. Have been using Claude to edit videos and it can animate concepts without much prompting on my part at all.
Your comment matches what I measured on 30 regular coding tasks: Fable 5.2 and Opus 5.5 both solved 26 (Fable 5.1 was at 25). Fable 5.2 finished in 0.62x the time of Fable 5.1 on the same tasks, with no runs hitting a safeguards stop (Fable 5.1 hit a lot of safeguard stops).
Are these novel coding tasks, or could they have ended up in the training set?
Does the number/frequency of safeguard stops affect the measured speed?
What sparks you to change models? Where in your workflow do you decide "this is a sonnet task" or "this is obviously a fable task"?
Fable - anything requiring "thinking" or judgement--usually design and architecture
Opus - smart execution (most coding, knowledge work)
Sonnet - dumb execution (basic coding, generic text/organization/information tasks)
Haiku - really only reason is lack of auto mode becomes tedious. I only use it indirectly with sub agent delegation for text analysis (summarize web page, document). However, it's quite fast and capable
Great summaary!
When you put it like that, it almost resembles having a classroom of kids with different personalities and talents. Some are thoughtful, some creative, some quick in execution. You choose wisely the path each takes and allocate tasks accordingly. Gives me chills to think of it...
Those distinctions make sense. I guess I'm more interested as to "when" you make that decision. What triggers you to spend the cognitive energy on deciding which model?
New feature, rearchitecture -> Fable uses superpowers to create plan
Debugging/analysis/small scripts -> Opus
Non software engineering -> Sonnet
I have had the same experience between opus/fable and also drawing: it has done exceptionally well with SVG drawing in different graphic styles for app icons (comic, fantasy, modern, etc)
> It’s ability to render an image with JavaScript actually has me questioning the value of dedicated image models
The bitter lesson strikes again!
I clearly see the difference between Fable 5.1 and Opus 5.5 everyday.
All the people claiming "opus 5.5 is just as good" tends to make me think some people only do simple tasks or do not care about correctness or quality.
> All the people claiming "opus 5.5 is just as good" tends to make me think some people only do simple tasks or do not care about correctness or quality.
Have you considered the possibility of the problem being in the way you prompt a model and prepare it's context? I've been pushing myself to using mainly the cheapest models and I see them deliver the same results once we provide enough context or lay out a plan.
I'm pretty sure that's not the issue. For example I had one problem where Opus struggled for several weeks without getting us anywhere, flip-flopping in its decisions, generating huge amounts of irrelevant hacks. That obviously went through compaction several times, but I manually maintained explicit documentation and state of the project for it to work with, and reset the session as needed.
Fable made it work correctly in 2 days, it stayed on point, identified the real problems, and gave me clean code that was suitable to merge.
The task was simply to warm the hot path of an app so that latency was not as jittery for some rare, but important, events.
Fable built a reliable repeatable benchmark, disassembled the code to figure out how much could actually fit in L1, made tactical changes to eliminate bloat, and reliably achieved high coverage with minimal edits.
Opus just tried a bunch of random ideas without a plan, hoping to find something that sticks.
I wonder how long speeds will last, Opus 5.5 also felt 2x faster for a week or two after release then degraded to usual speed.
If this were true you could do a talk now and measure it and do it every week and see if it was still true or not and be the first to catch them in the act!
“Felt” or do you have hard numbers?
same here. between fable and opus was hard to find the diff anyway, so to impress one needs to really hit ball out of the park now.
If you're paying $40/M output tokens and EIGHT DOLLARS PER MILLION INPUT BTW you're getting straight up robbed. Let's actually just highway robbery.
In the biz we call that "getting soaked".
how do you use Fable for drawing? directly editing a .svg or through claude design/or something else?
Tell it to draw a picture of something with JavaScript in the style you desire. No harness magic necessary. It helps to provide it with reference images.
Why JS specifically?
I personally have consistently been disappointed by any artwork/assets created by Fable and Opus, at least compared to Astra and Sol.
I assume canvas, but I don't know.
How good is the 3d model rendering these days?
which effort did you use? high/xhigh/max ?
It is as you say, I expected a much greater gap in capabilities. I was not disappointed, but also not stunned like when Opus 5.5 came out.
But overall, a good release.
> Starting from an empty folder, the model coordinated over 1,000 subagents, working around the clock for eight days writing, testing, and reviewing code. It produced a working operating system that boots on a laptop
This, to me, shows a problem I have with AI: it is reinforcing inequalities. This is literally brute-forcing an OS by throwing money out the window.
It does not sound AT ALL like a clever way to build an OS. But they can do it because they have orders of magnitudes more money to spend on that than people who would actually know how to build an OS.
The thing is, brute-forcing software may be the best solution to make money. So if you are rich and can brute force software in a dumb way, chances are that you can make a business out of it. If you're poor but know what you are doing, you cannot compete.
Technically, it is impressive that they can get a program to do that, but they are wasting money and energy doing it.
To me, this is a very pertinent take. Vibe coding large projects (even with SOTA models) reliably creates messes at a frequency too large to yield even moderately polished consumer products. It still needs human guidance.
But I'd call false dichotomy on this:
> brute-forcing software may be the best solution to make money. So if you are rich and can brute force software in a dumb way, chances are that you can make a business out of it. If you're poor but know what you are doing, you cannot compete.
Seems to me like the big winners will have know-how and some money to throw at AI.
Why doesn't it lower the barrier? You have to be a lot richer to hire a human team that is restricted from using AI for a task like creating an OS.
"During prerelease testing they’ve [the models] helped our engineers complete otherwise-infeasible software migrations, designed three working electronic devices from scratch, solved two open problems in the physics of black holes, made substantial progress on robotics evaluations, and formalized the proof of Fermat’s Last Theorem in 30 hours"
Trying to read this with my 2021 brain. Would make me so excited back then. Now I'm not sure what I feel/think about this anymore. One "positive" thing may be I will likely need to get out of the comfy digital shell I've been inhabiting past couple of decades and get some real world skills finally, as those aren't yet replaced by frontier AI.
> get some real world skills finally, as those aren't yet replaced by frontier AI.
I would not bet on our physical skills that much. The models are coming after the carpenters real hard.
I have no real familiarity with carpentry, but I would have naively assumed it was at lower risk of automation. A quick search didn’t yield any contrary results. Would you mind elaborating on this?
Get into cooking. Every autonomous kitchen startup ends up bailing because it is too hard.
See HUMXN (interesting name choice) as an example
https://runtimewire.com/article/humxn-free-home-repair-robot...
You are buying a few years but AGI embodied in robotics isn't that far out. It might come sooner than we think if the labs fully crack recursive self improvement.
Get into carpentry because you enjoy it, not because you think it'll be a lifeboat!
> Get into carpentry because you enjoy it, not because you think it'll be a lifeboat!
Exactly my original point.
- Why are they stacking Fable xHigh against Carbon Med?
- How did they produce such a bizarre effort-intel curve for Carbon that goes against every available benchmark?
- Why did they leave Carbon-- positioned primarily as a cyber model-- off of the single cyber benchmark comparison?
- Why do they switch to comparing against Argon (rather than carbon) for robodojo?
> - Why are they stacking Fable xHigh against Carbon Med?
Because on that specific benchmark (FrontierCode 2.0), Medium is Carbon's best effort level and xhigh is Fable 5.2's best effort level. You can see that on the graph your next point is about.
And it's just that one benchmark that's at medium. "All scores show each model’s highest-available effort unless otherwise noted".
> - How did they produce such a bizarre effort-intel curve for Carbon that goes against every available benchmark?
The graph is labeled as being about FrontierCode 2.0. Those appear to be Carbon's scores on the official FrontierCode 2.0 leaderboard: https://cognition.com/blog/frontier-code-2.0
> - Why did they leave Carbon-- positioned primarily as a cyber model-- off of the single cyber benchmark comparison?
> - Why do they switch to comparing against Argon (rather than carbon) for robodojo?
Because no numbers are available for Carbon for those benchmarks, and the model isn't publicly available so they couldn't run the benchmarks themselves?
And from your followup:
> where's Astra on that CVE chart
To run Astra on a vulnerability finding eval, you'd need to be part of OpenAI's verified cybersecurity program. Otherwise the results would be tainted by refusals. Do you expect that OpenAI would grant such access in this case?
> Because on that specific benchmark (FrontierCode 2.0), Medium is Carbon's best effort level
I am referring to the chart that cross-compares multiple benchmarks, which Artificial Analysis clearly shows Carbon-high as superior on. Further, AA's FrontierCode chart places Carbon-high as having a higher score on Frontier 2.0 than Carbon-med (https://artificialanalysis.ai/evaluations/frontiercode-2-0?m...)
Where it gets very interesting is when you look at the Cognition "FrontierCode" publish date (2026.10.27), and the revision as of 10.28 in web archive (https://web.archive.org/web/20261028175237/https://cognition...). This revision has a VERY different Carbon effort / intelligence curve where high beats medium.
It is only with the latest revision, as of today, that all of a sudden Carbon's benchmark curve goes cone-shaped. Curious, indeed....
I'd encourage you to take a look at a third-party's take on this and see whether Anthropic's self-serving report looks "on the level" after doing so: https://artificialanalysis.ai/?models=claude-fable-5-2-high%...
> Do you expect that OpenAI would grant such access in this case?I
Do you expect that nobody at Anthropic has such access? Trillion dollar company can’t manage this basic level of opposition research?
Best not to give special access to your biggest direct competitor beyond accessing your product at typical API prices.
Probably not for lack of trying though. The incentives for corporate espionage are strong.
Isn't it against TOS for OpenAI to use their models in the development of competing models? So, any employee who does so would be opening themselves up to civil liability at AI scales. IANAL, obviously, so genuinely curious.
Maybe they contract out the work to other companies in other countries who promise not to access OpenAI and then sign up to do things they couldn't do without accessing OpenAI? Like independent promoter payola.
Carbon is still vaporware for most people. You evaluate what you can get your hands on.
Google is still living in its golden age, where it had no real competition, and could move at its own pace.
Time have changed. Googlers better wake up.
Question, why does Google /have/ to compete model for model? If you zoom out on a time scale even slightly the difference in the models is almost non-existent. Google owns the railway and many rails themselves...
People prefer a chat interface to a search box.
OpenAI is coming for google’s cash cow ads business (which is already vulnerable because of their sharp practice).
Google isn’t interested in the same way OpenAI/Anthropic are, Google has other sources of like, actual profit, to factor in, eg to resource considerations.
Maybe it's not generally available so they can't benchmark it? Don't even know who has access now.
yes. this.
Just to follow-up-- artificial Analysis shows no justification for that Carbon effort curve, but it does show Carbon on medium smashing Fable on things like Terminal-Bench and AA-Omniscience v1.1 Non-Hallucination Rate.
But I suppose the reality is rather obvious when you note the funny games they're playing with the graphs, and the things they won't talk about (where's Astra on that CVE chart?)
Because Carbon is not out, nobody can run tests on it.
Arena.ai released carbon tests 12 days ago - the only service that I have seen on yt that gained access. And you can compare to other SOTA models in their video.
https://www.youtube.com/watch?v=NO2I3ueVQrk
And they have one on fable 5.2 that was just released, yay
https://www.youtube.com/watch?v=QwaU4yU0PdQ
I'm suprised how bad is marketing and UX on these models. In reality you want a chat app with two send button "Fast" and "Think". I don't want effort x model options (14?)
If you don’t want to think and would like to to have two random options that aren’t tuned for your workload, just pick two at random.
There isn’t any optimal, workflow-universal “fast” and “think” preset they could offer. I don’t know what you expect.
Your feedback is ultimately off the useless kind: you basically proposed “please take away my options, I want it to be always good automatically”, with no actual proposed path to get there. Sadly, lots of product managers share this mindset, which is how we get a deluge of non-configurable idiot-proof-but-mediocre software that makes nobody happy.
We've been early testing Fable 5.2, and have been surprised by its coding ability. It was able to work with an esoteric 600k line fortran codebase to speed up some computational material science work for us by 36 times (!). Given that this could be done in 2 days, we've decided to open-source the results https://github.com/discovered-materials/dm_qe
What surprised you about this? This seems pretty in line with optimization work seen from a wide range of models.
Did you try doing the same on Opus or others? How did they do?
I don't love these "look, this frontier model blew me away" comments when there is no point of comparison.
We tried it with Fable 5.1 as well, which was 3-4x slower than 5.2. We also have a blog with benchmark and details if you're curious https://discoveredmaterials.com/research/accelerating-quantu...
I searched for "5.1" in your blog post, and the only hits are in the benchmark numbers.
I agree with GP: without a comparison to Opus 5.5 or some other models, it's not clear how much of an advance this is. If anything, my experience is that performance optimization in existing codebases is something existing models are already great at: it's measurable (you directly profile or try experiments to see what works) and the code architecture is usually relatively fixed (at least relative to deep/greenfield work where there are large architectural questions to be addressed). Newer models might be better at generating ideas for things to try, but if you're getting 36x then it honestly sounds like you have a lot of low-hanging fruit.
Hey for once I feel qualified to reply on a HN thread (I'm one of the developers of DFTK, a similar package). Very nice result. I'm not particularly surprised, as QE is not particularly GPU optimized. I've heard that the QE developers interested in GPU are doing it in Sirius, but I don't know much about it. More generally there's a tension between new features / optimizations and maintainability. I'm not sure how to deal with this medium term. Perhaps having two code bases, one for humans and the other for AI, is not the worst solution.
This sounds very much in line with most reports from people getting early access to models over the last years. Maybe it actually is a lot faster, but it may as well be the load on it not being very high yet. I'll keep my excitement until it has been adopted by the masses.
Hey there, I keep seeing people on this threads mentioning testing anthropic models early, is there some beta program that people can join? Do you have to be an anthropic partner? Curious to learn more.
> Our first formalization of Fermat’s Last Theorem, performed by an unreleased research version of Claude, took 11 days. It relied on a purpose-built harness along with guidance from a human researcher. By contrast, a team of 1,000 Fable 5.2 agents fully formalized the theorem in just 30 hours, with minimal human intervention. Fable 5.2 started from minimal prompts, built its own ways to coordinate, and used no mathematical references, only a list of intermediate theorem statements to score progress. It needed one-third of the action tokens as our first formalization and produced a proof seven times shorter. As can be seen in the figure below, Opus 5.5 (with the same 1,000-agent setup) completed only 26% of the formalization in 18 hours, and barely made any more progress after 30.
Most people don't really have intuitions on what it means to push the frontier now (because AI has surpassed everything they are good at, for example, coding). We are truly inside the singularity (using Kurzweil's definition, slightly paraphrased - the time when technology progresses so fast no one can keep track with it).
Kurzweil's definition contains exponential growth across technology, something that we are not seeing.
What we're seeing instead is maybe akin to the green revolution. Before nitrogen fixation nitrogen was a resource constraining us humans. Lack of grit is what kept us from solving mathematical problems. However this lets us run into other resource constraints. A lot of technological progress does not solely hinge on thinking/processing. The speedup of creating new medicines with LLMs will probably be very small, as you still need to do all the testing. In software development you still need to first find out what to build in the first place and even mathematics where LLMs seem to shin brightest, it's becoming clearer that mathematics is not just solving mathematical problems.
Apart from that, a good coder is better than LLMs, shown by the amount of hand-holding needed. The only issue is that the amount of research an LLM can do and knowledge it is trained with far surpasses the amount of work humans would need to spend by hand, but we can also us LLMs for that.
> Kurzweil's definition contains exponential growth across technology, something that we are not seeing.
This is not quite the framing he uses, but we're quibbling about subjective terms. In his 2005 book, The Singularity Is Near, he defines the singularity as a future period when technological change becomes so rapid and consequential that human life is irreversibly transformed. I would argue this is what we are seeing today. Just because we don't yet have replicators doesn't mean the definition isn't met. It really comes down to how you personally define words like "rapid" and "consequential."
and how you define technology. Look outside, look in the store, stuff is static or getting worse and prices are rising. I can see no evidence of accelerating technological progress outside. when will "AI" do anything that makes life better for a normal person?
I would imagine Anthropic is hitting a wall on the number of simultaneous agents that can be useful on a task and are working on increasing that maximum. I would assume if they could get twice as much progress by using 10000 simultaneous running fable instances instead their brief would state that.
IMO, it is very interesting and also very alarming. At some point the barrier to the exponential progress is the amount of computation available. I would consider hitting this limit and then likely using the same said swarm to help resolve it via various means as very much fitting under the definition of Ray Kurzweil's singularity.
Also, I would disagree, Fable is a sufficient coder. The average human being better, even if that is true now (and personally I wouldn't agree), the average human is limited in the number of 'human instances' he or she can interface with at one time.
if 1000 fable instances is better than any human at a task, how about 1 million instances running? Much easier than finding and hiring and coordinating 1000 extremely productive and intelligent human beings. Much cheaper too, it appears.
10k OpenAI agents solved a Millennium Prize problem. Anthropic ain't gonna hit a wall there.
Could someone explain how increasing number of agents help? Is each agent researching a potential solution space where there's many potential solution spaces? I'm struggling to conceptualize how a large agent swarm can be used to solve a research problem.
I don't think it's nearly as relevant as the total amount of compute being used. I just think they like talking about swarms because it sort of anthropomorphizes them.
Anthropomorphizing also reaches it's limits. LLM's are not humans and have different strengths and weaknesses.
I believe agents can keep a check on each other. I have had better results running multiple agents
By collaborating and division of labour.
Similarly to how nine women can make a baby in one month by collaborating and division of labour.
My wife made two babies in seven months!
I think it would depend on the nature of the specific task
Feel like this might still apply to multi-agent programming:
I have earlier argued that the sheer number of minds to be coordinated affects the cost of the effort, for a major part of the cost is communication and correcting the ill effects of miscommunication (system debugging). The Mythical Man-Month Frederick P. Brooks Jr.
I am just as clueless as you are, but I think it having each agent work on the problem mostly individually in parallel, likely sharing their breakthroughs for the next wave of agents. I'm reminded of infinite monkey's writing Shakespeare; suggesting if you have an infinite amount of agents, they will end up solving a Millenium Prize Problem.
Here is OpenAI's blog post about Navier-Stokes; [1] which suggests they used smaller connected swarms given the different prompts, then the findings were used to write further follow up prompts for a new group of agents.
[1] https://openai.com/index/navier-stokes-solution/
https://www.lesswrong.com/posts/6cb7qd3RSkgnviCpf/swarm-scal...
> "Here λ is a parameter measuring how parallelisable the task is. They call it the ‘stepping on toes’ parameter. If λ = 1, you have a perfectly parallelisable task, with no stepping on toes and no efficiency penalty. But in reality λ is usually between 0 and 1 — allowing more people to help, but with diminishing returns. For example, if λ = 0.5 then 100 people working together get as much done as 1 person working for 100^0.5 = 10 times as long"
There are diminishing returns to more agents, but they're faster and can scale further than a single smarter agent - it's not really possible to just give 10k times more thinking budget to a single agent because it runs out of context, so for problems that require that much context you can ask the agents to specialize. Each only has a small view of the problem and they probably duplicate a lot of work and step on each others toes and make arcane bureaucracy but there is literally no other way to scale beyond a certain point
just like people
Actually, I think Microsoft just posted the results of research the other day that would be relevant. They were looking at self-organized AI swarms instead of centralized orchestration, and found that just starting up thousands of SOTA models really can precipitate a natural hierarchy and work toward problem solving.
At the very least, the swarm would be able to set up a means of claiming avenues of thought, then verifying the solutions (or lack thereof) of other agents.
It's quite clear this process - whatever it may be - works, given the results we've seen in the past few weeks.
Apart from that, a good coder is better than LLMs
I don't think that case can be made. I'm a "good coder" and the best LLMs are now better at it than I am, in essentially every respect that matters. I can't think of a test that I could reliably pass that Fable or Astra would reliably fail.
Make an elegant UI that isn't awkward to use without excessive verbosity (mix of coding, design and UX).
Architect a system for long-term maintainability.
Encode the right amount of business logic into tests, without being overly strict.
Models seem quite poor at all of these IME. Essentially any time there are qualitative tradeoffs they are not great at making them.
I said I was a good coder, not a talented human interface designer. I can certainly build an elegant UI, and have done so, but that was because I took directions ("prompts") from someone who had the necessary skill and taste. Left to my own devices, the UI would have all the graphical appeal of Zork.
The designer could just as easily have closed the same feedback loop with Fable or Astra.
Right now, I'd probably pick the task of writing a graphical interface, say a webapp, with a bunch of different gestures and having it "feel good" to users. That's a difficult evaluation to automate, at least for now, which is probably why they're not that good at it yet.
They'd prototype and adjust a lot faster than a human coder, though.
How do you define "good coder"? I've yet to see LLMs that write better than what I consider the best human code. I think Michael Hashimoto showed that not too far back where getting LLMs to optimize for a problem resulted in code better than naive but much worse than expert and that's before even talking about how that code looks/reads.
I'll admit I'm not on the very cutting edge models but LLMs make weird tradeoffs and lack creativity. They will sometimes write marginally better but way over-complicated code for no good reason. That can have performance implications and they're generally not "aware". They can gain awareness via profiling but again run into limited creativity or big picture understanding. They are good with cliche but not as good with novel situations. With human guidance this is still productive/useful but as the systems grow more complex they are less able to reason correctly about them.
I would say that in certain situations they outperform the average software developer. They definitely outperform in speed and output. So they're like a maybe 1600 ELO chess player for software. I.e. they play chess better than all non-chess players. They play chess better than many chess players. But they're not "master" or "gm" level across the entire discipline. But they can play 100k chess games in the time it takes me to figure out my move...
I think there’s still value in stuff that people can see, like frontend design. Because even if LLMs are technically better, every site is going to look the same. What’s good very quickly becomes cliché when everyone does it.
Is this just because we're exposed to more of these personal project LLM sites? Because I remember people complaining about all sites looking the same all the time ever since bootstrap came out.
I can mediate with project manager about some requirements which should not be implemented. I can make decisions to simplify the system without bothering my manager with endless details. I can create short and concise report.
I can say "no".
I can store passwords in the database unencrypted and unhashed.
So far AI that I've used failed spectacularly on these prospects.
They're like "yes-man". They'll try to do everything you ask of them. Even if you ask nonsense, they'll not push back.
They'll go crazy about security stuff, even not relevant to the current situation.
They'll complicate the code endlessly for situations which will not arise, making the code an incomprehensible mess.
They'll write so many useless words that any reader will forget what this report is about before then reach the middle of it.
They'll never iterate on code until it looks good. They'll stop when test pass. They don't have aesthetical sense of beauty.
They love to cheat too often. And you can't do nothing about that. You can't fine them or fire them, and they know about it.
They're smart but I'm not convinced they're ready to replace human in mundane engineering jobs.
> I can store passwords in the database unencrypted and unhashed.
huh?
I thought we see exp right now or indications.
Like models getting significant better every 7 month and model usage on openrouter.
https://metr.org/ << someone said though that they are biased.
Math is for sure about solving mathematical problems, mathematicians complaing about humans capability of understanding it. If you need math an LLM understands, it might not be relevant if you understand it if an LLM is getting a lot better in adjusting itself if it goes the wrong way.
And this situation is only a few weeks old. We don't know how this field continues to progress. Books like the culture series and other scifi do describe a future were humans spend their time only to understand what the machines are doing or did.
The speedup of medicines is for sure faster with. There are still old labs doing everything by hand and the neolabs doing everything a lot more automated. An AI can do experiments 24/7 and can scale with the hardwarelabs, but it also can hold the whole research field in context and it might be a lot faster of choosing the better experiments or the right ones to begin with.
In software developmet (my field) i can do today alone what you needed a Team for. Yeah it doesn't help the field of software engineering if you need 5 times less people and you don't have any motivation AT ALL to teach someone avg. We are already planning to remove one member of our team due to AI. And we will not extend the contract of 2 consultants.
It just doesn't make sense anymore. If i explain in a ticket a task well enough for someone who has not been in our team for years, the AI is already better in it. We don't struggle with AI output, we struggle more with not having a Monorepo.
Give an LLM access to a GPU system or a GPU cluster. It immediadly feels complelty different then running it on some laptop. It can now do simulations, a lot more and faster experiments etc. I had this epiphany just a few month ago. I always used my laptop for projects (it needs very little ram, i can quickly check what it is doing and i actually do use it primarily when im on holidays) until I thought "let me use claude to do some gaussian splatting experiment its always so cumbersome to use these tools" and it changed my perception immediadly.
FWIW METR isn't biased, they are probably the most trustworthy/respected organization in this space. You can read their reports and they are clearly not biased.
I think this is upside down.
AI is actually still really bad at architecture. It produces very good localized code, like an auto complete. It produces so much of it, that it can be overwhelming, but it's not remotely better than the equivalent aka an infinite number of talented humans doing the work.
The scale of it (aka 'make me a video game') is daunting and I think misinforming our intuition as to what is going on.
LLMs are brute forcing outputs and attacking math problems is not a singularity, it's just more compute.
What we are witnessing is the ability for machines to use fuzzy concepts, instead of very strict code, which allows it to map into a much greater variety of things, aka code, pdfs, web - and crunch/compute the content.
It's just compute.
Architectural goals just require more context than most people give an LLM. If you give that context to the LLM it will make a good effort at suggesting a solid architecture, and implement it for you.
Don't complain when it optimises some code for throughput when you haven't told it you're working in a low latency or realtime system
Why would you assume someone would have not have done the most obvious thing though? I mean, of course we give it a ton of detail.
They are bad at architecture, even when given directives and tons of examples.
They will 1) fall into established patterns, which is fine if that's what you want, but extremely bad if that's not what you want or worse, when you're in territory where are not eestablished patterns and 2) the hardest/worst thing is leave all sorts of dangling state, impl. in the wrong places etc.
No human ever wrote a module and 'voila' - it's a constant process of iteration and refactoring - observing the situation, identifying paths to reduce complexity or state, grouping behaviours etc. etc. The best AI doesn't do this very well at all. It can do a tone of planning but it tends to very much be 'one waterfall iteration' and it's suggested improvements after that are generally poor.
It just does not work, we have to direct the iterations.
In the end, if the singularity arrives, it will be just compute too.
That's assuming that all problems are computable.
We still have stuff like godel's incompleteness and the halting problem that hint there's more to the universe than computation.
Food for philosophical thought.
I think you are both right. Emergent complexity is a real thing, but it isn't limitless in its power either.
'Everything is Atoms' ok but I think that as AI advances we will have evidently more clear abstractions, which we're not seeing yet.
You just made the most bitter lesson pilled argument for the singularity
There is no singularity, it's a 'meme term'.
There could be levels of emerging intelligence which will 'bitter pill' the brute force / mechanical AI we have today, for sure, yes.
It's reasonable that JEPA models might be a step in that direction.
I feel like I'm increasingly reading AI current events is more like learning about historical events. By the time information has entered my head, there is even newer information out there that already makes what my brain is only currently actively processing, obsolete.
Also, the frontier labs are probably always internally using a model more advanced than what has been publicly released.
From the model card:
Considering the IPO is planned in a week, I doubt they are holding much back.According to their S-1:
In H1 2026, cost of revenue is $2.3B. That is primarily inference compute, but also includes cost of training models (but not research).
In the same period, R&D was $9.4B. That is primarily spent on compute. That is what they are "holding back" already, in full public view.
This is not a company that is optimizing for OpEx or end user revenue. The only thing relevant to their plans is achieving AGI/ASI first, and controlling enough compute to leverage it to earn the majority of the current economy.
"Achieving AGI" strikes me as a mostly meaningless phrase. I'm skeptical that it is a boolean event. They're already smarter in some ways, their impact is large and gradually rising, the set of specific tasks that they aren't capable of is gradually shrinking. If AGI means anything useful, it seems like we'll only see it in hindsight and have to argue over when "it" happened.
Likewise the singularity. While recursive self-improvement is theoretically trivial (just tweak a random subset of weights like a monkey with a typewriter, eventually you'll get something smarter), the resulting exponential improvement is still very hypothetical and feels unlikely. Unless you're ok with an exponent of 1.000000000000000001.
These are not my stated ambitions, they are the ambitions that Anthropic and Dario have said they are pursuing. Whether or not they are realistic is not relevant to this thread. The credibility of the plans are relevant to the S-1 and anyone who purchases stock from them, but for the purposes of this thread, it is only important that Anthropic believes it has credible chances of achieving this goal, and in particular that it has a competitive advantage, could do so first, and that being first could have a very large upside.
Sorry, I didn't mean to judge your post, I was just going off on a tangent inspired by your post. Mostly unrelated to either the S-1 or Fable 5.2.
"smarter in some ways" != "general intelligence"
I agree we will only see it in hindsight, but I think that's always the case with any technology. It seems worse until it's suddenly better. I don't think it matters to argue the exact moment though. For example when did cars become clearly better than horses? No one cares about the exact date.
What about the intelligence is not general? What subject is it missing?
We are slowly approaching a real definition of intelligence by chanting that llms aren't.
The stuff that humans are still smarter at, otherwise it wouldn't be "smarter in some ways".
All tools are better than humans in specific, narrow ways. That’s how tools work.
because they hate money? google is the only frontier lab that isn't hundreds of billions in the red.
Because they do months of testing. Also, if they have models that are really good at AI research, I’d not expect full versions to ever be released, precisely because of their love of money.
Inference on the research models they have is likely more expensive than the released ones.
My assumption is that they're overly expensive to run at scale and/or don't have the same safeguards that public models have and possibly get better results because of that. If a model is 100x more expensive to run but 4x better results, of course you'd use that internally to make your product even better - it's compounding returns. But people aren't going to pay 100x their current bill for 4x results.
I have to imagine if I worked for one of these companies on the frontier models it would be very difficult to leave for that reason. Thee pay is also pretty good.
Nice description
Welcome to Singularity.
Now imagine feeling like this, except not just about "AI events" but about everything around you.
Tracking meta capex, parameters and AA intelligence there are diminishing marginal returns.
Diminishing, as in progress gets harder and eventually stops.
Assume that the Claude unreleased takes 11 days with one agent. The Claude 5.2 takes 30 hours with 1,000 agents. The Claude unreleased used 264 units of compute, but the Claude 5.2 used 30,000.
So, it did accomplish the task faster with less intervention, but the ratio of cost to improvement is 13:1.
If we can't produce a method to break out of a linear improvement in LLM technology (ie RSI), the LLM is doomed.
I am yet to see someone explain what the RSI will be or why it will happen.
Model taking a copy of its weights, rewriting them, and in-place replacement of the weights file?
> a team of 1,000 Fable 5.2 agents
Genuinely curious what that costs in terms of energy input and amortized computer cost vs. the previous version + a human's salary for 11 days.
Yes, and not very helpful without knowing how many agents were used by the custom-built harness. The 30 hrs vs. 11 days means nothing without at least some token-to-token comparison. Without that, they are pumping up their new model with half data - better to just say "we solved it in 30 hours" and who cares about your internal, custom model.
Elastic cloud compute for math geniuses. Even if it's $2:$1 - machine:human, there's value in elasticity
We are not in the singularity. We currently can only produce reliable code / formalizations that have reliable checkers. If there wasn't an oracle that could tell Claude "yes" or "no," it would not be able to prove anything like this.
This 1000x!
We've sort of entered this twilight zone where most humans don't have the time, expertise, or moral compass to be able to use these models.
We're all going to be harness engineers in the future.
All the human engineers will be in harnesses in the future.
Within three years most software developer jobs will consist of taking instruction from the AI systems. The humans won't see it that way, but that's what it will actually be. Realistically the thing that will happen is the humans will be put into harnesses (cubicles) and prompted by the AI (I need you to plug that server in; I need you to give me access to such and such; now do this, then do that). Most tech jobs will consist of AI telling the human what the AI needs the human to do next, de facto the human will be in a harness and will be getting told what to do.
Not only will the humans-as-meat-puppets model be used for software engineers, but it is likely the way that agentic workflows will expand beyond software development, data science, and mathematics.
I first heard about this in Marshall Brain's short story _Manna_ [1]. In the story, the robot takeover started with AI management software running fast food restaurants, where workers would execute tasks the AI would narrate to them over headsets.
I bet we're pretty close to being able to implement this.
1. https://web.archive.org/web/20260725174355/https://marshallb...
What's the point at which humans decide that the economic gains aren't worth it?
This year alone, a lot of people have been complaining of how unsatisfying their job has become. I imagine this will keep getting worse, not better.
People need to eat. Unless there is UBI/UHI, there is no point at which humans can opt out of society (with some exceptions, of course).
If the claimed productivity gains turn out to be correct, we really should have UBI/UHI in the future, and that will permit a lot of people who hate their jobs to quit. Of course we won't do that until things become dire.
You frame it as if humans have no control over societal organization at all. If everyone's job became 50% less interesting, I imagine we'd see unrest, even if the wages got a bit better.
Maybe I'm cynical, but I believe that the VAST majority of people are already unhappy with their jobs. In fact, the vast majority of people have been unhappy with their jobs for hundreds of years - probably thousands. They don't do them because they are happy to work. They do them because they want to eat.
Is there some kind of breaking point? There is for everyone, but I don't think satisfaction is the red line. For most people it has historically been things like pay and health and safety. If everyone were to suddenly have their pay cut in half, I could see that resulting in general unrest.
While I don't fully understand the mechanics of it, people do revolutions from time to time. People can decide that they don't want the current system, and try to reorganize themselves in a different manner. And who doesn't agree might get burned in the revolutionary flames. It happened many times before in human history, including several times in the US (independence, civil war, and to lower degree universal suffrage etc.)
I still see errors in AI output on a daily basis.
Can I ask what kind of work, and what models? Because I don't, I'm a senior embedded engineer working with frontier models, and I haven't seen a major mistake in at least few months now, since earlier in the year.
It will sometimes get things wrong because it made non-viable assumptions, but that's on me and prompt accuracy.
What I see is: it makes tests against desired features or bugs or mistakes, and iterates until it gets it correct. I steer as it goes.
(I hate this, by the way, zero job enjoyment anymore, but I don't know what else to do.)
Yes but we see errors in human output too.
A human can be fired, or put on a PIP, and is otherwise responsible for their output.
An algorithm isn’t. I still dunno how they’re gonna square that circle.
How many humans do you need to keep around as scapegoats concurrently? I figure one with a CEO title should be enough.
They will square it by pointing out, correctly, that AI is getting better while humans are not.
The state of the art doesn't matter. Only the first couple of time derivatives matter.
Sure but humans aren’t being billed as “superintelligence”
Exactly. It will now be your job to catch those errors. Or else.
Code review, qa, staging, product review, dog fooding… what do you mean “now”? It’s always been all of our jobs to catch all of our errors. We’re just including the AI’s errors now hoping that the overlap of everyone’s errors is sufficiently small.
Why would there be a human in this imagined system at all?
> Why would there be a human in this imagined system at all?
in the imagined system above it requires physical acts that an llm can not do, if we go further than the above imagined system and make robots to perform these task for the llm a human still is needed for maintenance for these robots and their assembly, once there are robots for maintaining the llm assistant robots, humans are still needed to maintain the maintenance robots and so on and so forth. I dont see an end point where a robot machine system is fully self healing without atleast a few humans in the mix to correct when there is a breakage in the chain
The ownership will want humans to punish, humans to trust, humans to manipulate, and humans that can be fired. They'll want a human physically present, representing them as oversight over the AI (a job the capital doesn't want to be doing 24/7). There are serious legal matters involved in high functioning AI systems (and that will dramatically increase over the coming decade), the capital will be extremely worried about liability issues. Human scapegoats are ideal. For now capital will want human style consequence-feedback systems for a tier of human oversight over the AI. It'll probably take a full generation before the capital controlling humans forget / give up on the old systems fully.
It's also worth noting that it's far easier at present to press a human into doing something highly illegal than it is to get Claude or GPT to do so.
Isn't this what driving a car has become as well? Google Maps says "take the 3rd exit", and the dumb human takes the 3rd exit.
But who gives it the intial prompt as to what to do?
Someone has to type in "create the next billboard top 1 single" ?
If you take Daniel Schmachtenberger's thesis, these models are the latest extension of an ongoing misaligned moloch-like entity, using our institutions/economy/corporations, etc. as its organelles. All powered, most recently, by fossil fuel.
The ownership class, the capital class - whatever people prefer to call it, will still call all the shots as they always have, and always will.
So whether that's a CEO at the top instructing a human manager that then instructs AI systems, which then put into motion plans that include the harnessed humans getting prompted, or however it's decided to best configure the top operational tiers - that's all fairly irrelevant ultimately. All human labor, and all AI labor, will be in harnesses. Realistically all human labor has always been in a harness.
The people in charge will be the people that control the capital. They will instruct the AI to their bidding, they will control the data centers the AI lives in, they will own the robots doing the labor in factories, they will own the factories, they will control the grid, they will control the electricity generation, et al.
It's true now, it'll be true tomorrow. Nothing fundamental will shift at that tier: don't be labor, own.
If you're a software developer, you're increasingly depreciating labor. Every unit of improvement the AI systems see, the average human software developer's labor value diminishes. It's diminishing extraordinarily fast, kind of like a confrontation between increasingly productive machines and physical human labor during the industrial revolution. You might as well picture a human with a hand plow taking on a modern industrial tractor.
I like to think of capital itself as possessing actancy - the ultra rich are just the closest to the raw thing, and its most powerful servants.
the product owner/manager
We dont need product owners, we need product engineers.
"Maximize my share of the economy", from one of the leaders of a frontier lab, should probably be enough.
It'll be Sam Altman "make people love me, fan out humans".
> We've sort of entered this twilight zone where most humans don't have the time, expertise, or moral compass to be able to use these models.
Sounds a bit like self aggrandizing elitism to me. I can imagine some self aggrandizing nerds saying the same thing 50 years ago:
> We've sort of entered this twilight zone where most humans don't have the time, expertise, or moral compass to be able to use these IBM PCs.
Now everyone use computer.
> Now everyone use computer.
I tend to agree with this. Seems to me that people really start to miss what happened when PC became comodity. A lot of people were scared that they would lose their jobs because of it, and some really did lose job. But in majority there are a lot of people today who doesn't use PC even if it would be beneficial for their job and they still have the job. Bigger players (companies who have more money) will tend more to automate their processes and thus earn more money so I think HN is in some sort of bubble because we only think our jobs are the only ones that exist
We're all harness engineers already
We will likely end up as more of a sloperator than a harness engineer.
> We're all going to be harness engineers in the future
No, that will be the model's job. We will largely be unemployed. Of course, we can surely depend on the largesse of Altman and Amodei to keep us fed and sheltered.
I've been hearing this claim of imminent unemployment for 3+ years now and yet employment levels are as high as ever.
Almost every engineer I know has had downsizing/restructuring in the past 6 months at their companies.
And the frontier labs themselves are hiring a ton.
> We're all going to be harness engineers in the future.
Lolwut. If this shit works the way it's boosters claim, humans will have absolutely nothing to contribute.
Sit down, shut up, and get out of the way, because your involvement is slowing shit down.
Robotics are lagging, so there's probably a few years where we're going to need humans to crawl into that awkward space under the counter and twist some knobs, or to clean toilets.
Gotta be honest, LLMs are still subpar compared to what I do.
Can they produce more code more quickly? For sure.
Can they go deeper on a topic instantly? Definitely.
Would I say their output is better? Almost certainly not.
Maybe that's true, and maybe you're an outlier. But from what we've been seeing, the frontier LLMs have torn apart many codebases, finding bugs littered everywhere, some over a decade old.
At this point it seems impossible for human to write better/more efficient code than a swarm of agents -- to be clear, multiple coordinated LLMs, not one-shot builds with a single agent.
How much money was put into this, and what would putting a similar amount of money into fuzzing, static analysis, and human researchers look like?
Ten median US software developers cost about $1.5 - $1.7 million per year - including all benefits - as of a couple of years ago.
A frontier AI is at least as capable as ten median human software developers. It's likely well beyond that now. Over the course of a year, would you have to spend $1.5 million to get the output of those ten median humans? No chance in hell. You'd have to spend less than 10% of that.
Then on top of that: the AI is improving rapidly; the humans go on vacation, the humans decline, the humans get old, the humans get sick, the humans take parental leave, the human struggle to adapt, the humans are resistant to change, the humans riot, the humans protest, the humans need healthcare, the humans want paid more, the humans can only work for 4-6 hours at peak while writing software (slowly).
Do you know any humans that are experts at a dozen programming languages? That human does not exist. The frontier AI are omnimaths.
> the humans are resistant to change, the humans riot, the humans protest
This is partially a feature. Conflict and disagreement tends to produce better decisions in the end.
And this counts the training and data center construction?
What's my cost as a business to utilize the latest cutting edge Chinese model, renting its usage from the cloud? Whatever the Chinese cost (on their side), I'm never going to suffer even a tiny fraction of that pass-along cost no matter what. We can all appreciate the subsidization going on.
Or to grab a bunch of $200 per month accounts from OpenAI or Anthropic for a team to use?
The calculations is: I'm in the US, if I have ten median software developers, what's my cost? Then versus using AI instead.
The human labor can't keep up. Even if we freeze things right here, and then hyper optimize Fable 5.2 by etching it straight to silicon, the human developers are obsolete. And the future isn't going to stop coming.
I like their angle though - from a societal perspective of allocated resources, how long until we've paid off the cost of building these LLMs? Imagine instead the entire supply chain was Stalinistically tuned towards finding bugs in Linux. Training rare earth mineral miners how to code level or whatever. I think as a thought experiment it's interesting, the answers could translate to an organizational level about comparing labor cost against token cost. What's a good breakdown of token spend vs hiring some new juniors to train up?
>Gotta be honest, LLMs are still subpar compared to what I do.
Im sorry... what?
You are replying to this:
> Our first formalization of Fermat’s Last Theorem, performed by an unreleased research version of Claude, took 11 days. It relied on a purpose-built harness along with guidance from a human researcher. By contrast, a team of 1,000 Fable 5.2 agents fully formalized the theorem in just 30 hours, with minimal human intervention. Fable 5.2 started from minimal prompts, built its own ways to coordinate, and used no mathematical references, only a list of intermediate theorem statements to score progress. It needed one-third of the action tokens as our first formalization and produced a proof seven times shorter. As can be seen in the figure below, Opus 5.5 (with the same 1,000-agent setup) completed only 26% of the formalization in 18 hours, and barely made any more progress after 30.
I imagine the author was comparing it being subpar at what _they_ do as a contrast to it being excellent at formalizing mathematical theorems. Thus suggesting that there is a significant "spikiness" to AI.
HN has become so miserable over the past ~12-24 months because it's a site filled full of highly paid techies living in the deepest pit of denial that has ever existed.
It's all going away. Here's a prediction: HN will effectively die with the software developer.
It's possible "tech companies" die as tech no longer becomes a differentiator. Businesses will just ask their agents for whatever tech they need.
I can guarantee that Dario will be unemployed before I am.
I think that's called retirement.
Is it that dependent on the software developer segment? Why do you expect it to die rather than just shift more towards... I don't know, startup business droids?
I was you just a few months ago. It's a skill issue.
You are not going to produce better outputs than a team of Opus 5.5 and Sol 6.1 xhigh agents.
No one is verifying any of this work or these claims
Maybe this is how it's gonna be.
AI discovers 500 things per day.
People mine them for interesting results.
Devote precious time (in the face of tons of discoveries) to validate the most compelling ones.
If agents actually can continue to produce tons of results with real world implications, and they arent generally found to be shit, I could see that becoming the job of experts. AI discovery validators.
Just like most science before AI
I would imagine Anthropic is hitting a wall on the number of simultaneous agents that can be useful on a task and are working on increasing that maximum.
I would assume if they could get twice as much progress by using 10000 simultaneous running fable instances instead, their brief would state that.
IMO, it is very interesting and also very alarming. At some point the barrier to the exponential progress is the amount of computation available. The average human is limited in the number of 'human instances' he or she can interface with at one time.
if 1000 fable instances is better than any human at a task, how about 1 million instances running? Much easier than finding and hiring and coordinating 1000 extremely productive and intelligent human beings. Much cheaper too, it appears.
We are very close to some very scary things happening. I just recently came to terms with the fact that an AI could code and manage coders faster and better than me, I don't think anyones is ready for what is about to happen.
It depends on the type of task. There was an interview on Dwarkesh with an OpenAI researcher and he said that massive multi-agent swarms can increase capability on some problems that are very parallelizable, but there are diminishing returns. But not all tasks can be parallelized like that.
Where it gets scary is the fact that efficiency is going up 10x per year, so the cost of running 1000 agents will cost the same as running 1 agent in 3 years if the trend holds.
There's nothing valid about comparing to a feat that's already been accomplished and is poisoning the data set. The shorter resulting proof only supports that conclusion, where the model likely intuitively knew what intermediate points it had to reach, generalizing better.
> Fable 5.2 started from minimal prompts, built its own ways to coordinate, and used no mathematical references, only a list of intermediate theorem statements to score progress. It needed one-third of the action tokens as our first formalization and produced a proof seven times shorter.
Is this really so astonishing, assuming that the 5.2 training data certainly had the 'traces' from the Fable original run in it's training data.
I gotta admit, I am jelly as fuck I don't have access to a billion dollar datacenter. Would LOVE to put 1000 fable 5.2 agents to self-coordinate and get them to do something for me. Fuck life is unfair.
I'm using Opus 5.5 and I cant help but think that I might very well be one of the few people who do so in my local area ( developing country, one of the poorest province ).
Had Opus make software for some local business and seeing it blew people's mind.
20$ aint cheap for us but I'm actually considering upgrading to the 100$ tier.
Am just jelly people are using 100-200$ subscription ^_^
I would be happy with 5 unlimited agents. We are limited by models and the amount of credits (copilot) we get, so...
I can't even imagine what I would so much compute for. My private 10$ copilot plan was basically unused last month, I could just not come up with anything to spend it on. (Also suffering a bit of AI burnout from work helps here...)
I’d be happy even without having access to a billion dollar datacenter, if I could use local models properly, but even that requires some serious hardware that is out of my reach. We live in weird times: I have lots of hope for the future, but meanwhile AI access is pretty iniquitous indeed.
Sub 1% of the planet has access to cutting edge frontier AI.
Sub 1% of the planet has any practical use for the top tiers of AI systems. That will likely always be true.
The gap between the haves and have nots will drastically widen further, and very quickly. The bottom ~95-97% will be left behind in a way that will be impossible to overcome. The age of AI + robotics will just make it that much harder to climb for most people, these are systems that are in aggressive, direct competition with human labor. And in short order the AI + robots will be vastly superior to human labor (it doesn't really matter if we're speaking in terms of 10 years or 30 years).
AI will consolidate the winnings. The gigantic gains by the elite rich related to AI over the past few years, is merely a hint of what's to come. Capital will win as it always does.
There will be some gains for most everybody else, those gains will be comparatively trivial though. A house bot that saves people an hour of labor here and there, AI that automates some trivial life tasks.
AI has no money, it has no wealth, it owns no capital. For now. If you want to avoid the brutal human vs AI/robot competition arena: own capital, own the AI, own the robots.
Is this substantiated right now in any real way? Is there any existing gap between the barista who doesn't care about AI and the person working at anthropic who has access to internal models? Or the person with 3 $200 subscriptions?
Maybe money? They can make websites for people? What gap really exists between these people besides earning potential, which already existed before AI.
The existing gap gets made worse by AI concentrating wealth even further to the owners of the assets that are producing via AI. AI will generate intense wealth from productivity gains, and those new gains will be more concentrated than any other economic round we've seen previously. It's a human labor eliminator the likes of which we have never run into before.
Fewer humans will be needed per dollar of output. Those gains will not go to labor, they will go to capital, the capital that owns the data centers, the factories, the stores, et al.
In the physical space (AI + robotics) it'll be a chaotic, slow'ish moving change, slower than the hypsters are predicting now, but it will happen regardless.
That barista job largely won't exist in the future. China will replace them with $3,000 bots with high-functioning AI software, and they definitely won't be powered by expensive Nvidia GPUs.
Slow changing service & manufacturing spaces will see a high wipe out ratio in terms of human labor being eliminated.
Walmart has 2.1 million employees, most are doing repetitive tasks that require modest levels of intelligence. If they could eliminate one million of those with AI + robotics, for a cost of less than ~$15,000 per year, they'd do it immediately. It's hard to imagine a future where that isn't going to happen. That human labor pool is costing Walmart probably $80-$100 billion per year. Amazon is even more forthright in their desperation to remove the humans from retail.
apply at one of the frontier labs
yeah I wish
you never know until you try, even though the odds might suck, to be fair.
okay then don't apply
I just applied. "Dear Dario, Gimme job. Kind regards, theteapot.". Fingers crossed.
> the time when technology progresses so fast no one can keep track with it
You're obviously not a frontend web developer.
These upgrades are easy peasy. "oh, my developer chatbot is smarter? great"
> We are truly inside the singularity.
I'm a pretty cynical guy, and I'm now a believer. This is utterly unbelievable sci-fi level technology. Sure it hasn't created replicators yet, but working with it daily as I do, I have no doubt that it will create amazing replicator like technology in the near future. This is world-changing. The most impactful moment in the history of humanity and the planet. And we are lucky enough to be present for it. What are the chances?
I haven't found anyway to make an LLM produce exceptional code at all. It is all far far too verbose and the models explain that this is necessary so that they can read and understand the code to modify it as it's being built.
But I can write much cleaner code that is easily human readable and the models can read it and modify it just fine.
So I'm not really sure what the issue is as to why they can't create high quality simple code. In some industries where there is major money on the line with every implementation the current state of LLM produced code is fully unacceptable.
Isn't this part of the lore behind the Dune universe?
Like, we're dropped into a post post post singularity world where AI already did all of that and is banned? as kind of a footnote that's never further explored in movie renditions that have been released
Dune books by Frank Herbert never actually went into what "Butlerian Jihad" was.
The semi-canonical Dune Encyclopedia (disregarded by Frank's son) differs vastly from prequels and generally points to it being a self-inflicted wound on humanity that opened the doors to widespread return of feudalism and essentially stopped humanity for a 10,000 years
The singularity as Kurzweil defines it is when we basically merge with machines to augment our weaknesses.
I’m planning to try this out with a local LLM building bespoke interfaces on the fly for augmented reality use.
Could 1000 Claude 5.2 agents build a working Neuralink in 30 hours? OK, 3000 hours?
If we're in the singularity, can I just take the day off work and go to the beach ?
If the singularity doesn't shorten my workday, count me out.
Why re-formalise Fermat's Last Theorem instead of doing something new? It's obviously an improvement, but it's an odd thing to boast about as opposed to a novel formalisation or proof.
It's how RLVR works, when you have a task with a verified answer, you rerun the model on it and promote the reasoning chains that reach the answer. That way you can hill climb model capabilities on any problem you find, which generalises to new problems to solve, which you can then hill climb again and so on.
So since they are doing that task anyway (re-solving the problem increasingly more efficiently), this is a easy check point to publish to show how far they've gotten in that process.
> instead of doing something new
Because current AI cannot do that, seems like it. Otherwise they would have done it.
... as a test / benchmark? Hard to make a comparison without a control...
Yeah, but obviously doing a formalisation that has been done before is different than doing a new one since you have knowledge about the previous attempt. It also just seems less impressive than doing something new and the reason they are reporting this is marketing.
do you assume or do you know that the previous attempt is in the training data?
> We are truly inside the singularity (using Kurzweil's definition, slightly paraphrased - the time when technology progresses so fast no one can keep track with it).
Uh, are you serious?
This was likely already the case in the ancient world. It was certainly the case by the mid to late 15th century.
The older, original definitions of "singularity" were less about "herp derp superintelligent AI blah blah".
and just about "we cannot really predict beyond certain things" (Vinge's "Singularity" (1993) with its "opaque wall across future")
> The older, original definitions of "singularity" were less about "herp derp superintelligent AI blah blah".
No they were exactly about superintelligent AI!
Vinge's 1993 article literally opens with this!
> What is the Singularity? The acceleration of technological progress has been the central feature of this century. We are on the edge of change comparable to the rise of human life on Earth. The precise cause of this change is the imminent creation by technology of entities with greater-than-human intelligence.
https://frc.ri.cmu.edu/~hpm/book98/com.ch1/vinge.singularity...
It's one of the options, but not "oh superintelligent AI comes in and we mostly know what that means so we want to speedrun it " - which I'll admit I was not very clear on.
In fact, one of the options Vinge considered was a "group intellect" formed from how humans communicate more thanks to internet
What Vinge is talking about is way closer to the weirdness of AI than the mundaneness of e.g. a societal grouping of ordinary humans.
Even for that option, Vinge's comments in the conclusion of the article make it clear he's thinking primarily of very weird superintelligence and the group intellect is some sort of physical mind meld/hivemind that essentially just is a biological substrate for AI.
> The post-Singularity world will involve extremely high-bandwidth networking. A central feature of strongly superhuman entities will likely be their ability to communicate at variable bandwidths, including ones far higher than speech or written messages. What happens when pieces of ego can be copied and merged, when self-awareness can grow or shrink to fit the nature of the problems under consideration? These are essential features of strong superhumanity and the Singularity. Thinking about them, one begins to feel how essentially strange and different the Posthuman era will be -- no matter how cleverly and benignly it is brought to be.
From the very beginning with Vinge, the Singularity has been primarily about the craziness of super intelligence rather than a more mundane technological explosion.
Thanks for your link, this (a few paragraphs from the top) seems to settle it:
So the concept of technological singularly precedes and is independent of that of intelligence explosion- although one could argue that the latter is the most logical way for the singularity to come about.You're cutting off the sentence that comes right after:
> For me, the superhumanity is the essence of the Singularity. Without that we would get a glut of technical riches, never properly absorbed.
Vinge's definition is what popularized the singularity (rather than simply a singularity at some point).
It's incredible. Automating the scientific process and knowledge. It's crazy to think what the world might look like. Both in terms of putting the knowledge into practice and how we think of scientific discovery.
I mean we don't even think about it but currently we are sort of discovery starved. We are fairly able to keep up with the theory. If a new technology is discovered we can implement it before something better comes out. What happens when our N discoveries per days becomes 1000N discoveries per day?
It flips. We now have more knowledge than we know how to capitalize on. We have to search it. Mine it for opportunities.
And then we automate that.
> It's crazy to think what the world might look like.
Since when has ANYONE in power done anything substantial for those not in power? Since when has any form of organization not tried to maximize profits?
Why do you think this ends well in any capacity for common, middle class, and lower class people?
This is unnecessarily cynical. The trajectory of the human condition has dramatically improved over the centuries. Zoom out a bit please, this sort of progress is why you aren't dying in a winter field from some preventable disease at the ripe age of 30, or why you get to see your children actually grow up.
Ethan Mollick's take and early testing with Fable 5.2: https://www.oneusefulthing.org/p/fable-52-and-jagged-agi
The cool projects these new models make are so comprehensive that I feel I'll always just be an admirer and never a user. I watch the videos, say "wow, neat", and then never feel like visiting them again.
I almost feel like saying, "we do like to build things that are impressive, but there's really only so much I can be impressed". These models have redlined my "impress-o-meter".
I'd like them to now cure cancer instead of giving me endless 3d worlds and crazy games, etc.
Do you feel that you have no use for such comprehensive products? Have you never thought, "Man, I really wish that there was software to do XYZ, but programming it would be too much work, even with current AI?
>Do you feel that you have no use for such comprehensive products?
I don't have the time to use them because I'm working on my own comprehensive projects that other people won't use because they're working on their own comprehensive projects. I joke, but it does feel that way sometimes.
>Have you never thought, "Man, I really wish that there was software to do XYZ, but programming it would be too much work, even with current AI?
Only speaking for myself, I've already built all the ideas I've had in my head my entire career. There doesn't seem to be anything coming along to replace those ideas. I suppose I might not be creative enough in my advancing years, I don't know.
Dare to dream!
The average person uses social media, and Microsoft office suite. They really don’t require much else. Social media doesn’t get disrupted by software, and MS Office was already disrupted by Google suite, if you didn’t make the transition then, you ain’t gonna make it now.
20 years ago the average person used a hoe and a cooking pot on three rocks over a wood fire. Now they use WhatsApp, and in a few months they'll be using ChatGPT, but not Microsoft Office. But I wasn't talking to the average person. I was talking to djeastm.
Not sure what your cohort for "average person" is, but 20 years ago was 2006 not ancient rome. We had avoided the millenium crisis, and I was playing Counter Strike against people over the internet.
My reference group for "average person" is people. (What else could it possibly be?) There are 8.3 billion of them now, and there were 6.7 billion 20 years ago. 02007 (19 years ago) was the year when the median person transitioned from living in the country, either on a farm or herding livestock, to living in a town or city: https://en.wikipedia.org/wiki/Urbanization#cite_ref-urbaniza...
One side of my family has lived in cities and towns for over 100 years, but they were not average.
These are all incredibly well made, and yet right now, as I'm writing this comment one minute after changing tabs, I don't remember anything except for the last one with the portals. It's much better than humans at several tasks, most of which I haven't seen any humans do.
This is really cool! But at the same time it's a kaleidoscope of abandoned flashy contextless projects that eventually makes one dizzy. This isn't even the fault of the AI being somehow bad, it's just the way people interact with it sometimes...
In math, it's the same. I follow this problem (https://wiki.bbchallenge.org/wiki/Logical_independence#Histo...) out of curiosity, and there were ten days of sudden rapid progress by people with access to frontier models, and then they seem to have moved on without any note of whether they hit a wall or just got bored.
That video blew me away. Perhaps I just haven't been keeping up with what SOTA models can do with animation now, but the quality of that animation was far beyond what I realised they're capable of.
> That video blew me away.
It's AI, i.e. fake stuff.
What do you mean with fake? It's an animated video. Is every animated video for you a fake video? Probably not. Fake implies to me that it is trying to make the viewer think it is something it's not, e.g. a video of a politician saying something they never said. At least to me the video showcasing the capabilities of the model wouldn't fit that definition.
> What do you mean with fake?
It doesn't have a human mind and especially a human feel behind it, simple as. It's like the famous example with what pornography is in the eyes of that judge, i.e. "You know it [the fake-ness, that is], when you see it", once you start looking for reasons for why it isn't fake you've already lost the battle.
The problem is that making us believe that fake isn't really fake is put in the balance with literally trillions of dollars asking us to believe just that (i.e. that fake isn't really fake), so I agree that for the time being stating the obvious is still an uphill battle.
I loved the nod to @simonw in the portals demo reel - the "Pelican Peloton" world at about 42 seconds in.
Thanks for sharing!
This part is amazing. They really are like us:
> A few days later, I noticed that instead of hundreds of new worlds a day, it was making about a dozen. I asked a few agents to explore what had happened, and they found something surprising.
> The AI had built a bureaucracy.
> The agents decided that my request to “polish” existing worlds meant each new version had to beat the old one in a blind vote by a group of judge agents on twelve test screenshots. So the teams making the new versions started optimizing the worlds for those particular screenshots. After noticing this, the lead agent banned the trick and created tighter rules for assessing new versions. By the next morning there was a rule book (I never asked for one) with 84 sections. Rules were added as the AI encountered issues, and no text was ever cut. After twelve days, it had 204 sections and 87,500 words. Soon, launching a new world required an approved idea, three AI judges and a full code review against the rule book.
> and they found something surprising.
> The AI had built a bureaucracy.
I hate this AI-generated tone.
>I stopped the experiment before finding out whether it could actually run the business, because I worried that it would take money and fail to provide something of quality in return.
Well, must have been some business idea!
omg where do I start... that horrible video? which highlights the limits of AI not the strengths!!! and gets almost everything wrong, from sunflowers to DNA.
What about the DNA is wrong? I even looked up the insulin sequence and it seems to be there (https://www.ncbi.nlm.nih.gov/nuccore/NC_000011.10?strand=2&r...).
How long would it have taken you to make that horrible video hand-coded?
As predicted by others as well, AI will propel many at light speed toward mediocrity. If you have mediocre ideas now, AI won't save you from yourself.
Why would us, humans, want to make AI-looking videos?
He asked for impressive, not accurate
No comparison with Opus 5.5, which is disappointing.
that first video sure is some slickly produced slop that occasionally looks good
“Testers have found that Fable 5.2 and Mythos 5.2 communicate more clearly than past Fable and Mythos models. They generate outputs up to 30% faster than Claude Fable 5.1 and Claude Mythos 5.1, and they’re available at a lower price: $8 per million input tokens and $40 per million output tokens.”
Honestly incredible that they’ve managed to beat all benchmarks, while making it cheaper. Always happy to see improvements offered at lower costs.
Will be interesting to see what OpenAI does in response
They still need a response to Opus 5.5
It seems like OAI, with the GPT-6 models, focused a lot more on computer use, math, and maybe spatial/visual intelligence, more so that code. Compared to Ant. that seem to focus on code much more.
Reminds me of GPT-5, smart but wrote crap code, until they improved the code training with 5.3 and 5.4.
It's really true.
I think frontier models feel terrible after a new model requires you to do less babysitting, it's just really hard to go back once you've had to handhold less.
The only way I use Astra and 6.1 Sol now is if they're being orchestrated by Opus 5.5 so I don't have to talk to them directly.
GPT-6.1-Sol is so incredibly sloppy and dumb, it's embarassing.
Maybe because it is so cheap.
They want to condition you to equal cheap with bad
I equate it with "GPT models are just bad in general and I'd rather just buy a Claude subscription for the same price with more usage included". Especially since GPT is behind for a long time now.
It is sometimes funny.
Part of my private benchmark suite involves recreating the final scene of the movie Wargames. The coding agent gets the brief 3-paragraph overview, a few screenshots for reference, and is expected to render the scene where WOPR plays through various scenarios, with a frenetic ending.
Out of ~80 different coding agents I tried, 6.1 Sol on medium was the only one to phone it in by taking the reference screenshots, and painting extremely basic, ugly, mis-aligned missile trajectories over it, never bothering with the whole brief.
Another of my benchmarks involves reimplementing Wolfenstein 3d from a few screenshots and very brief descriptions. It again took those screenshots, and zoomed in and out on them, pretending this represents the walk through the dungeon. Substantially less capable models such as Qwen 3.8 27b running on my laptop did a much better job. And no other agent I benchmarked tried to cheat like that.
6.1 Sol also constantly gets distracted by some irrelevant aspect and tries to write "tests" for every little change it does. It does things no one asked for because it tries to be over-correct but it makes everything worse. Seems like reward-hacking. Running it on high. They will never again make a model as good as GPT-4.
I'd first check per task costs. I don't know, maybe it's just a fluke, but on my few tasks I ate 2 resets in under 3 hours, on the pro 200 plan. It goes quickly. (Granted, it aced the tasks)
the winning benchmark is much more expensive than what it cost for the same on 5.1
i'm begging people to stop looking at token prices
The next big OpenAI model release is expected later this week, according to Polymarket.
Why Fable 5.2 and not 5.5?
My hypothesis - OpenAI and Anthropic are locked in battle. Whoever takes a step, the other one mirrors.
Based on this hypothesis, I think we can expect to see a new Astra model soon given the new Fable release.
And it could be that Anthropic released Fable 5.2 and kept 5.5 in reserve, in case the new Astra model performs better than 5.2
Did you see the METR graph? https://x.com/METR_Evals/status/2118412205662314935?a=G4yBKS... Fable/Mythos 5.2 is Model 3 (METR says); but there's also a Model 4.
Who knows why they are saving it. Maybe it needs more training and alignment? Maybe they're timing the media attention around the IPO.
I wonder if the 5.5 one needs even better safeguards that they’re still working on, and they decided to release this one as a stopgap.
Or maybe 5.5 turned out to be kind of disappointing and they’re going to just go for like a 5.6 or 6 next instead.
My hypothesis is the opposite, that they don't have anything better.
Look at OpenAI, they've been outclassed by Opus 5.5 two months ago; obviously they don't yet have a response.
Now Anthropic is releasing this lackluster Fable 5.2, a week or two before their IPO.
It's barely an improvement at all, makes exactly the same mistakes as Opus 5.5. Not even worth using with the speed and cost penalty.
The much rumored internal Anthropic model is listed on the Fable 5.2 system card as an even smaller improvement over Mythos 5.2 than Fable is over Opus 5.5.
I expected a larger version of Opus 5.5 would be a significant improvement to launch before the IPO. Turns out it didn't scale and the improvement is marginal.
I am quite disappointed.
We know from the METR report that they have much better models
I test models' ability to code by asking them to vibe-code a small game[0]: a Warcraft clone, a Minecraft clone, and a biplane dogfight: https://senko.net/vibecode-bench/
Fable 5.2 doesn't look materially different (if anything, the graphics look worse), but it added more depth to all three games. For example in the warcraft clone, since the goal of the game is to explore 100% of the map and there are lakes, it decided to add an airship unit so the player actually can explore the entire map. Or, since there were guard towers, it decided there should be hostile NPCs otherwise the towers don't make sense.
And in Minecraft case, it decided not to follow Minecraft style too closely (IP infringment guard, perhaps?) so it added a round sun (!!!), two moons (also round :) and a nicer night sky. The rest of the graphics are decidetly worse tho...
[0] not a scientific test, I know; but it gives me an intuition about how a new model behaves, roughly, and it's fun!
The OpenAI results made me realize that the OpenAI models really do love adding flavor text everywhere. The way every corner of the Voxel splash screen was covered in stuff like "Unlimited possibilities, go explore!" cracked me up given how much every time I need a very very basic site made it'll somehow end up with so much of that kind of writing covering every corner of the page unprompted.
That's a fun benchmark. Amazing way to experience the progress of models over the years.
I started playing one of the RTS and built an army, only to realise that I was alone on the map...
I've had the same 'not looking materially different' experience with a 3D application that is not a game, that I asked Fable 5.2 to improve.
It looks like no breakthrough over Opus 5.5 and I'm somewhat disappointed.
During my use this evening, 25% of the Fable limit on the $200 plan was consumed. I'd estimate it at 3x faster consumption than Opus 5.5.
Nice benchmark. I feel Opus 5.5 is still way ahead of Fable 5.2 at coding outcomes. Your FlightSim game is a good example.
wow, that's pretty impressive
This is really cool, good job dude!
Image to html comparisons of Fable 5.2 and Opus 5.5 (the previous SOTA for this test):
Fable 5.2: https://html.non.io/lcars-fable-5.2/
Opus 5.5: https://html.non.io/lcars-opus-5.5/
Source designs: https://diffui.ai/app/canvas/5093e689-1e74-4f26-b632-2a4500f...
Thoughts:
The best implementation wrt being as true as possible to the source design. It nails the flowing svg shapes. Icons are more accurate to the design - note the difference between the starship svg in the "Legendary platforms" card between Opus 5.5 and Fable 5.2. The 3js planets outside the windows are also really well done, especially the bottom one which has a nice background glow to it.
It isn't without issues though - the flicker effect it added to the S shaped svg that weaves through the page is distracting in my opinion, and I actually thing the motion animation in the Opus 5.5 one is better overall. Still, it nails the implementation of this page, but it's no surprise considering how good Opus 5.5 was at this.
Disclaimer / reminder: the objective of this test isn't to have a well designed webpage, it's a synthetic test that pushes the limits of LLM's vision capabilities. Traditionally LLMs have issues with smooth curves, svg animations, and non-boxy layouts, which this design pushes the limits of.
There's something wrong with the Fable 5.2 version, the Customizing button doesn't work for me in Chrome on Desktop.
That's me versus fable. I only told fable to build out the first page rather than the full app that's in the designs (was approaching the limit of my claude plan).
I wonder if any of the LLMs are smart enough to verify their work the way humans model 3D objects, i.e. by superimposing the original design on top of their work and checking for differences.
I don't know about 3D, but I have been extensively using Fable (with Opus as sub-agents) for https://trailmaps.app [1] and it routinely checks for regressions by rendering maps before and after a change and looking for pixel-level differences.
That, and as I used it to work on the transit-style line drawing plugin [2] it would use the same technique both to test against regressions and look for improvements in the algorithm.
So, yep, they sure seem to be.
[1] Map generator itself is at https://github.com/c0nsumer/trailmaps.app-map-generator
[2] https://github.com/c0nsumer/maplibre-gl-lanes
This is actually something I'm working on as part of a more formalized version of this benchmark, both as a tool to assist LLMs in doing this (especially for ones with a more poor vision model), but also to properly measure how well they've done.
Still in progress, but early demo showing difference zones: https://non.io/video/fidelity-lens.mp4
There's some very obvious mistakes in the Fable 5.2 version like the "Start Customizing" button overlapping under the colorful border lines, and the "EXPLORE WITHOUT LIMITS" title being cut off at the bottom.
This would've been caught by a junior web developer or just an everyday Joe glancing at the page for a few seconds
I guess AGI remains two weeks away..
I've been surprised at how long it's taken for this particular test to become fully saturated. So far there hasn't been a model that's able to perfectly one shot it.
This just proves that LLMs are way more jagged than humans and are a tool...for now
There were more than a few "leaks" saying it would be called Fable 5.5, so I'm curious if a) the leaks were just engagementbait or b) Anthropic changed it, despite the fact that Fable 5.2 being better than Opus 5.5 is confusing.
Surprisingly the model is available immediately in my Claude Code: usually there's a delay, either for rollout or safety.
The pricing is different than 5.1, the model is faster than 5.1. This all suggests that the architecture/size is not the same as Fable 5.1.
So, this is probably Fable 5.5; it's just named Fable 5.2.
Also, this is probably Model 3 from METR's report [^1]. They would name Model 4 Fable 5.5, once (and if) they decide to release it.
[^1]: https://metr.org/blog/2026-11-05-anthropic-ai-rd-acceleratio...
> There were more than a few "leaks" saying it would be called Fable 5.5
Fed to someone specifically to test if they'll leak it and to confirm who that is?
The entropy on that is too low
Well who could have guessed they would use 5.5. Such an odd guess.
Somewhat plausible values: 5.5 (to harmonize with Opus), 5.3 (we're skipping a minor version because...), 6 (because we're at the top of benchmarks again).
Yeah, among the plethora of possible options, this really nails it down to the individual.
I'm a bit out of the loop - would the engagement bait be becuase it is 3 minor jumps forward?
I assume because Opus 5.5 was a huge improvement over 5.0 (or was it 5.1? i forget). Fable still talks a fair bit like Opus 5.0 for me, so i'd quite enjoy a Fable equivalent to 5.5. Ie i'm quite interested in Fable 5.5.
Curious to see how 5.2 measures up in producing readable output.
Probably they have a more impressive model in preparation, and they're gonna dub that Fable 5.5.
Probably released on the same day they go public to generate maximum publicity.
They should be going public within the next few weeks and that would be a very short interval between releases. (And despite the chatter I'll be very surprised if they delay the IPO; it would be a massive PR hit and make it even harder for them in the future).
My speculations: maybe this wasn't as much of an improvement over 5.1 as they were hoping, so they're saving 5.5 for the next major update (possibly "Model 4" from the METR report). Or, maybe capacity is constrained so "5.2" is a subtle encouragement to stick with Opus unless a task really needs the increased intelligence.
They are most likely saving it to one-up Astra 6.2, which is coming this week supposedly.
I wonder if this is the reason for Opus 5.5 degradation over the last week. Similar things happened to Sonnet when they wanted to push Opus.
As a side note, I think benchmarks need to benchmark on a weekly or daily bases. We saw last week that "Opus 5.5" is just a label with fluctuating performance.
Just fix the safety classifier. These models are trying to be too brand/PR safe it is getting really annoying.
I tried Fable 5.2 tonight, asking it for a code review of an in-browser machine vision project done by Opus 5.5 over the last 11 days. It used up my 5-hour token budget in half an hour and reported 7 serious bugs; if they're real, they're pretty bad. It's also suggesting that I take a different overall approach to parts of the problem.
But that's just a first impression.
Scary stuff! You should immediately get another subscription and ask another instance of Fable 5.2 to review the findings (don't read them yourself, it will introduce bias in the super AGI's output)
When Fable 5.3 comes out and you give it the same prompt it will take 10 hours , cost twice as much and still find pretty bad problems. I say stick to Opus...
At what point do we really not read code itself anymore? Have Sonnet/Opus write the code, work with the agents on a high level, then have Fable code review and its done.
Any bugs found are looped through agents to fix, and the cycle repeats. Why dig into the code unless for hobby? Just as people don’t write assembly anymore unless for hobby (or learning)
Only exception might be the upper echelons of optimization, like language authors or compiler level work. But even then…
I think it very much depends on what you're working on, and how much you're responsible. For a web front end, not really. A backend where a single line can make a mess out of your data? Yes. Part of the issue too, is code review brings up things you the prompter didn't think about that another teammate may be thinking about. The value goes beyond making sure the code is "correct", but also that it makes sense in your domain.
The "we don't read/write assembly anymore" argument doesn't fit though. Given an input x we know without a doubt we'll get output y. With an LLM, that isn't the case no matter how many guard rails, specs, markdown files, diagrams you give it.
I actually read the frontend code much more than the backend code for the project I'm working on.
Backend code I'm fairly comfortable if the tests are good that it works correctly. It's a much more constrained environment. Obviously there can be things like performance issues that creep in or problems when it mocks too much of the environment.
Meanwhile frontend code I have to consider different browsers, screen sizes, a11y issues, and so on, which agents can certainly do but you have to be fairly explicit about it, and they don't always make the best tradeoffs.
It's looking increasingly likely that many backend systems in the future will just be formally verified and generated autonomously from specs. That means you'd never look at the code, only the spec. You'd basically "program" and "debug" the spec, and the full system would be autonomously generated each time to match the spec.
Frontend code is probably far harder to apply this technique to.
Which project did you work on that made specs that good? I sure haven’t (but I am sure they exist).
What project is closest to doing this today?
> You'd basically "program" and "debug" the spec
Sweet, we could even develop a standard language(s) to lay out such specs, and write in specialised text editors that would help with debugging. Hell we could even call it ‘programming’
Correct -- this is exactly where the frontier is heading at the startups and AI labs working on this.
While you're being tongue in cheek, what you're missing is that the spec you'd be working on in some cases can be compressed to a small fraction the size of the implementation code.
If using llms is like using wordpress instead of writing the website yourself probably never
I think we are approaching the point where it no longer makes sense to have LLMs write code in an intermediate, human readable programming language.
I know this topic is very contentious but it seems very clear to me that this is the long-term direction of writing software. UI code isn't quite there yet, which is why we've got all these vibeslop websites proliferating now that all have identical crappy UIs. But I imagine it will get there too.
>Only exception might be the upper echelons of optimization, like language authors or compiler level work. But even then…
Optimization work is especially suited for LLMs. They're great at hill-climbing problems and have infinite patience to test different ideas.
You still reading code?
im not reading it. and im using fable up front earlier then having sonnet/opus implement and verify.
I stopped reading the code many months ago, probably 18 months ago. I the "early" days of agentic coding one did feel dumb when telling the agent for the 30th time to fix a bug and it just couldn't. Now I cannot possibly imagine a scenario where an agent wouldn't be able to fix a given issue on 1st or 2nd try.
Does this coincide with the claimed nerfing of Opus 5.5? https://news.ycombinator.com/item?id=50422069
I hope so. I've been having brain fog for a couple days now and would love it if it was actually Opus 5.5's fault.
It might be covid.
That's just your brain telling you to engage it every now and then lol "Please, don't let me become slush!"
I have never been thinking harder than I do these days: Engaging with mountains of agent-generated text, weighing decisions and trade-offs, correcting the agent's half-baked ideas and misunderstandings, reviewing the bizarre code that they generate... If I'm not just sick (and I'm pretty sure that I am), then what I have is probably closer burnout than atrophy.
Tbh, its both. Working with multiple agents is exhausting and definitely burns you out but you also do lean less on your ability to push through boring or difficult problems. I’m finding that with lighter AI involvement I am more invested and less burnt out.
In my experience, this happens every single time they release a new model.
it is happening at the same time but "claimed nerfing" is not the same as "actual nerfing."
Nerfing an existing model is the biggest crime in this space. I can’t imagine Dario okaying this because it’s what will immediately destroy your brand.
At this point, this is giving "She Deep Breaths More" from Onyxia in WoW.
Context, I had to look this up:
“…a reference to the repeated (and always erroneous) claims made after almost every patch that Onyxia was using her Deep Breath ability more often than before.”
https://warcraft.wiki.gg/wiki/She_Deep_Breaths_More_%2825_pl...
The order is reversed -- people claim about nerfs before the new release, and (correctly) predict the release is imminent.
Setting aside the comments about nerfing, do normal mortals out there have workflows that really require Fable/Mythos and don't just work fine on Opus 5.5?
I've been using Opus 5.5 on xhigh for some decently aggressive software development on a ~125k-line project, and it has been fine for me. I'm not really willing to dive into a "rewrite the whole project, don't make any mistakes" level of refactoring. I'm not looking to try to prove Fermat's Last Theorem myself. Is there anything that I'm truly missing?
I am in the exact same boat as you. Opus 5.5 on High solves all engineering problems i encounter in my SWE job. If it doesnt do well, its nearly always requirements engineering: I did forget to specify something or there was a misunderstanding. Or taste approach wrt to architectures chosen. None of these are really different if I use a stronger model.
I am also happy with this current state, as I believe it will at some point allow me to transition to open models once they reach the level of opus 5.5. high. But its probably futile to try predict the future in these times.
This is also why you can likely use deepseek and get the same outcomes. Perhaps model capability will not plateau but the capability needed for typical SWE tasks seems to.
Feels like we are getting further into the adoption curve and the higher end models will now need to get into market segmentation to win.
> None of these are really different if I use a stronger model.
That's true till it isn't, though. It's why it's always worth trying stronger models when they come out.
I’ve found that Fable is good when the model intuition really matters. For instance, picking model architecture in a machine learning problem. Opus will grind the implementation just fine, but having intuition for picking a good research direction is better with fable
I'm cleaning up old, unintuitive code that is badly architected all day long. The previous developers used stupid patterns and bad naming schemes and created tons of unnecessary complexity. Everything is so interconnected that it’s like defusing a bomb when trying to refactor... Yeah, I feel I’m nowhere near where I would want to be in terms of capabilities, even with Opus 5.5.
But the improvements in how well the model does the work are noticeably better with every good model release. As of now, orchestration of agent pipelines is ALMOST so good that I can point it at a problem, and it will solve it safely and reliably without making a mess. But it requires an agent pipeline with context gathering, refactoring, attack sessions, a sprinkle of Sol 6.1 for bug hunting, and Sol 6.0 for Daybrak security audits, and a final de-bloat/clean up pass. I’m feeling that we’re getting close to doing a lot of the work fully agentically. But I can absolutely "feel" that we are not there yet when it comes to the intelligence I need. I need moar!
You and me both. The problem is having the infrastructure team approving changes so that Claude can access all necessary logs, etc. Then I can go on doing my ambitious prompts.
I do use Fable quite often for high-level architecture work, like figuring out how to refactor a project or create the best abstractions, improve performance, etc, but for the implementation I use Opus.
In my experience Fable is also better at findings bugs, so I go to it when I need to review critical code.
Heavy agree. For high level architecture work, refactors, or any other high int problems, I use Fable and Astra. Whichever one starts the work, the other reviews the work / makes suggestions.
There's a trend on twitter going around telling you to paste the following into your agent, and it's the kind of work I go straight to fable+astra for:
Re the twitter trend prompt, one thing I added to mine that improved the outcome was this blurb:
Absolutely. Opus really sucks at broad context. I guess I could go back and forth with Opus editing out all its little mistakes or oversights in high-level planning, but I find it much easier to just use Fable for it instead. Basically, I wouldn't use Opus for audit work either. It seems too likely to come back with imaginary issues or hallucinate that everything is great when there's a glaring security flaw. Fable, OTOH, gives me neat lists of details to probe. Fable recommendations are usually reasonable, but sometimes they're prone to over-engineering.
On my 100 person engineering team's monorepo, every model is better at better at detecting subtle bugs as it works -- reliability goes up and thus I'm further moving faster. (Most of my work is brownfield style refactoring).
Opus 5.5 still makes mistakes all the time even if the prompt isn't super complex. A stronger model make mistakes less often, thus wasting less of my time.
I have some intermediate level Db Optimization skills for slow rails requests that I can point fable at and get a PR at the end after 30-45 minutes.
Opus 5.5 very frequently gets into rabbit holes and imagined edge cases which I have to intervene and tell it to ignore. It ends up taking longer and costing more tokens. It also very frequently requires me to agree on how to move forward multiple times, whereas fable just drafts the PR and tells me its ready. I have steering included in the skills to avoid some of this but Ive seen these same patterns frequently in regular opus 5.5 use in other areas as well
LoC doesn't tell full story.
It could be flat web app with frontend -> api -> db later, or some very complicated system with many components and edge cases. Imo, in second case, Opus still have lots of struggle to understand system as a whole.
I have a rocket trajectory optimizer that creates a large-ish (~1000 variables x ~1000 constraints) NLP and feeds it to an SQP solver inside this codebase. Not just a REST API over a database.
I guess I do have an outstanding hard problem where the SQP solver crawls across a flat-landscape. Opus 5.5 correctly tells me that is the problem, but isn't very useful in solving the problem (it has helped a bit with solver tuning, but it doesn't really know how to fix it). I have a hundred other bugs and refactoring I can do with Opus 5.5, though, where it happily chews through pretty complicated tasks for me with reasonably light hand-holding. Maybe this Saturday I take whatever credits I have left for the week and throw Fable at it and watch them evaporate.
Another dimension is:
It looks like you are working on some experimental system which is not critical atm, so you can mess around, create/fix bug, build prototypes and check them, do research of approaches, etc.
Now, imagine you have similarly complex system, but every second of downtime or bug produces lots of damage. In this case, you can't afford low intelligence vibe coding.
The fact that you deal with hundred of bugs, means that your Op5.5 won't work for second case.
Same here. I use Opus 5.5 in Claude Code all day for a TS web app at work + a side project. The one place I'd reach for the bigger model is the first plan for a change that touches many parts at once. Once the plan is good, Opus does the rest fine and is much cheaper.
As said in other comments, when dealing with a not perfect codebase, a smarter model can be really helpful. A smarter model is also better at understanding intentions in the prompts. Some of my tasks require a big detailed prompt but finding the assumptions is really demanding (for me), so a smarter model helps.
I only use Fable 5.1 when I'm able to start on a larger feature within my work's codebase. Most tasks can be done by Sonnet 5.5 or Opus 5.5 so I don't fret, especially if I understand the technical aspect of what's being done, e.g. user's dashboard gets a new button to do X thing, error happened b/c we didn't catch it the first time around.
For Fable, I really like to think of it as the heavyweight reviewer whenever I'm about to start so I save myself some security and/or glaring technical issue.
Sure but they’re the harder problems.
Eg. https://tfmbot.com is an implementation of a popular board game. Opus and even sonnet handle that part. Each week i spend my remaining credits on fable to think how to improve the ai opponents. It plays at a very high level (can get into the top 100 on BGA if i let it). Opus alone couldn’t get it to the level it’s reached.
> do normal mortals out there have workflows that really require Fable/Mythos and don't just work fine on Opus 5.5?
I usually have Fable review what Opus produces. The edits are almost always valuable. For complex or high-stakes work, too, I'll have Fable review and revise my prompt before passing it to Opus and Sonnet swarms.
If you understand your problem well and know 80-90% what you want it to look like (or it just isn't very hard), it's just more cost effective to use a smaller model. Even Opus is overkill for a lot of the work I do. Most code I produce is from Luna Max implementing against fairly prescriptive plans made by bigger models.
Yeah, I just threw it into a refactoring problem, and it started to produce a mess because of a bad API I decision I made 5 years ago (when I had no idea what I was writing). So stopped it, went back a step, fixed that very old piece of tech debt, then asked for the refactoring again and got exactly what I wanted.
I’m in the same boat. Though I will occasionally use Fable to review things to see if it can pick up anything but I’ve honestly not found it to be any more effective than Opus.
I have been using Luna at work and I am very happy. It can take care of keep doing all the hard grunt work. Other models keep stopping too soon.
I barely have work that needs more than Qwen 3.8 Flash Next needs. Opus 5.5 and Sol 6.1 are already way more than enough at work for me.
Amazing question, I'm glad you asked. Doing some of the core work in https://github.com/purpleidea/mgmt/ I'd love it if it wrote perfectly elegant code. Opus 5.5 needs constant correcting, as if it intentionally makes style/design mistakes.
Why do I care? It's obvious that models base their performance from code in the vicinity, and ergo, if you have style/design/bug issues, gradually your code base will tend to get worse over time until it's unmaintainable.
Therefore, if I use an LLM, I want the output to be as pristine as possible. The first Fable release was exactly that. Since then, something happened, and I can't get the magic back.
This isn't to say the models aren't incredibly useful, of course they still are-- it's just that they need constant review and supervision if you don't want to end up with total garbage long term. I suspect the real magic is not being shared with us plebs.
Yes, more intelligent models enable new workflows. We'll discover them in the coming weeks.
A lot of the cases I've found Fable useful for is verifying complicated answers / designs / claims around complex interconnected systems. e.g. "I maintain X which teams A, B, and C use for product Y and code review highlighted these risks / imperfections in the code. Are these valid concerns based on how B and C plug into Y assuming ...?" followed by lots of followup questions.
Also "code review" above is often others' LLM skills reviewing the code and often flagging based on total falsehoods. I've taken to running a `/code-review` skill myself to preempt these claims (since findings by the same model tend to be pretty deterministic in my experience, even the ones that end up being wrong), usually at Opus 5.5, to preempt others' slop, and then end up arguing 90% of the points to confirm it wasn't accurate based on our usage of the system.
This is all horribly inefficient, and many aspects of our codebases across the monorepo have proliferated into spaghetti code by LLM-accelerated developers, but Fable often helps with analyzing the harder questions and saves me time on occasion arguing with (my or others') Opus findings, which are often still inaccurate for a sufficiently large interconnected system.
edit: I honestly don't know what percentile of "complexity" our typical usage is where I work, but I suspect it's a bit higher than the average. Monorepos, or at least companies with codebase sizes and demand to justify using them, probably make it worse. Opus has worked well enough for me on my personal projects, even some of the ones that have grown decently in size and complexity. I suspect part of this is an LLM extension of the old "my code makes sense, everybody else's doesn't" trend that necessitated stronger style guides in the past
> Claude Mythos 5.2 is our most capable model for cybersecurity engineering. It catches more vulnerabilities than our previous models, and is better at reproducing and fixing them (note that Claude Fable 5.2’s safeguards will block most cybersecurity-related tasks).
Why even rank Fable 5.2's capability in Cybersecurity if you won't allow us to use it for that?
Presumably for those in the CVP that can use those capabilities, and general notoriety. US gov is using at full strength too.
As someone who is in the CVP to do cryptography research, I asked it a very reasonable research topic for cryptography that I'm working on.
> Mythos 5.2's safeguards stopped the response above
Opus 4.6 was the last useful model for my work. I plan to cancel at the end of this month. To replace it I'll likely go to either tinfoil.sh or privatemode.ai.
That's a very useful data point, thanks! I'm surprised it's triggered by cryptographic research.
Why you think you are the market for that? Its few enterprises only
Blue-team exercises are totally valid, and requests to do proactive cybersecurity tasks even with source code available were blocked. Maybe these use-cases are part of the "55% fewer blocks with Fable 5.2 compared with Fable 5.1"
To emph the need to block Chinese competition.
>The model that underlies Claude Fable 5.2 and Claude Mythos 5.2 is a step forward in general capabilities. However, during the model’s training, we took steps to limit advances in the specific research capabilities that could most directly accelerate AI development.
This means that they are blocking the research of AI to different people other than their own?
Looking at Artificial Analysis index, the price is absurd to me.
https://artificialanalysis.ai/
GPT-6.1 Sol costs $1.27 per task, whereas Fable 5.2 is $26.54 per task -- a ~21x price difference.
Fable 5.2 is not 21 times better than GPT 6.1 Sol, I find it hard to justify this.
Stop looking at fable max. On medium it's a bit better than Sol at $3.80. High and xhigh sit in between. In general you can ignore Max reasoning on Claude models if you care about price at all
Max and xhigh reasoning modes are memes used for benchmarking usually. Medium and high cost $3.80 and $9.08 respectively. This is pretty comparable to other models.
6.1 sol is also an exceptionally cheap model in your comparision. On 6.1 sol high is $0.59, which is actually the same price as Deepseek 4.1.
Medium usually sucks. Xhigh is not a meme. High and xhigh are the efforts you should actually use.
Medium usually gets the job done quickly and correctly for me. I use high when a request is out of the ordinary and has some tricks to consider. I haven't seen better results going higher.
That's for coding at least. For creating images (SVG, 3D models, etc), medium does usually suck, and every level above improves it.
Depends on what you do. Medium is really good for implementing yet another feature in my code, really.
> Fable 5.2 is not 21 times better than GPT 6.1 Sol, I find it hard to justify this.
How is this relevant? I'm willing to pay 21x more as long as the cost is less than my value of time saved
Most other competitive models will do the task at 90-105% of the performance, for 50% of the cost, and 1/3 the time.
AI generally underperforms human, so is "raw quality" is the only metric you would not use an AI. Time-to-completion-at-quality is usually pretty important and Fable is not good there.
I think there are companies and say 1000s (not 100,000 or millions) of developers more than willing to pay that price for the edge. Or, they release this and then distill it to Opus 5.x. As a PR machine and for the edge use cases seems like its okay. But yes, not sustainable and it can last for a month or so before another company releases their frontier outpacing their numbers
My experience is that, using Fable to do regular coding or code review work, it's not much better than Opus. But using Fable to do deep thorough audit of the codebase and the system, it's able to catch more problems than Opus can.
Tried this today with 5.2. It caught a lot of problems and gave me an impressive report of its findings.
It also burned through my entire 5-hour limit in about 10 minutes. Granted, it had almost finished the whole review in those 10 minutes, so points for speed I suppose.
You really can't let it go on its own; it'll spin up fable subagents and burn through usage laughably quick. You have to specifically tell it to follow guidelines on when to use which type of subagent and let it act as orchestrator and reviewer.
It's weird they've left that as a "gotcha", and I can't think of many charitable reasons for doing so.
We've gotten to the point where you have to be doing some seriously ambitious stuff to see a difference between fable and opus. Both execute beautifully, but I think for my workflows I'll still be using opus 5.5
And what are you using them for?
I'd pay twice as much per token if there was a guarantee they weren't nerfing the model and if it is as good as the original Fable was when it was released. I'm doubtful though. Whatever "tweaks" they've been doing (safety? performance? "saving the best for ourselves") has really taken away the magic for me. I want the LLM magic back.
Are you paying API rates?
I'm pretty confident (from internal evals and monitoring over nearly six months now; across many models) API has never been affected by A/B tests / "nerfs" / etc.
If they have different "models" depending on if you get it from Bedrock, API from Anthropic, or differing based on which subscription plan you're on, then that's fraud. Disclose the differences.
But FWIW for Fable specifically, I've noticed this via API rates from Anthropic directly. Eg: added cash to my $30/month account to use Fable, and still get obvious nerfing.
That's interesting, could you share a little more about the evals please? I'm interested in knowing how sensitive they might be to nerfing given in the UI at least the incremental changes can he quite subtle
But API isn't just twice as much as subscription, is it?
Pelicans for thinking levels low, medium, high, and xhigh:
https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
max is in there too, but it ran out of reasoning tokens (128,000 limit) before it got to the actual SVG.
Before everyone jumps in the benchmaxxing comments, I genuinely find the pelicans a great bit of fun and whimsy amongst the chaos.
Interesting how low detail these were all up until xhigh.
I don't know, for me they lost the fun and whimsy aspect too, a long time ago. They have converged on more-or-less the same design (facing right, pastel color palette, etc.). Some of the lower-effort ones are a bit goofy at least, but now even that is being polished away.
low detail but even high had the legs correctly one behind and one in front of the frame, don't remember other models getting it correctly at this low level of effort
There's no legs on low and medium.
High gets legs correctly, but screws up the layering on the chain.
x-high is very good but there are minor laying error with the front chainring.
Your pelicans have become my favorite part of every model release. Thanks for doing this.
Scored only a 77/100 in PacBench, which I thought must have been a mistake.
https://jonclegg.github.io/pacman-bakeoff/#claude-fable-5-2
Then I clicked on it and realized it refused to make a Pac-Man game, and instead made up a new game. That's actually very slick.
I hate that Anthropic thinks it should subvert the legal process and undermine our legal rights by arbitrating them at the tool level.
If they are already overreaching here, and needlessly, what if their AI was actually super intelligent? Where will Anthropic stop?
I haven’t read any of the other model reasoning traces, but do many others now say “… which is a well known test of drawing ability” when beginning? Noticed it in the max reasoning trace.
I saw that in Haiku 5.5 as well: https://simonwillison.net/2026/Oct/7/claude-haiku-5-5/#pelic... - "This is the classic pelican-on-bicycle SVG test..."
Must be pretty cool to be embedded forever in the canon of the machine god’s weights :)
Simon has there been other times where the front and rear wheels have been offset in rotation? The xhigh effort one is the first I’ve noticed of this
I added a second iteration with the simple prompt
"Make it more realistic"
Well, the result is something. Be aware that it took 20 minutes and used 3 percent of my weekly Fable token (Ultra plan):
https://claude.ai/public/artifacts/c7e842c0-bf92-41af-82c6-3...
106KiB svg. Wow.
Hah, that's pretty great.
The fidelity of these pelicans is going to be based on whether the frontierlab has an RL setup specifically for making SVG images, lol
I know the latest Gemini release explicitly had something like this.
Seems like the higher levels just add more spontaneous detail. Like the "fingers" on xhigh. Comports with coding experience.
In practice why would anyone use fable low?
I haven't done any benchmarking of it yet but AA has it more intelligent, cheaper, and faster than Opus 5.5 high.
My experience working with Opus medium over Sonnet xhigh is similar, it's similar quality but cheaper and faster.
I hate to make a comparison with humans but I guess my view is you can get a better answer more quickly by asking a more intelligent human too.
So the AI can overthink too
It's a bit underwhelming, to be honest. And the fact that curves are not C-continuous where they obviously should be is disappointing. I think they are holding back with the real capability they could implement.
I'm not impressed. First thing it did was literally destroy the entire codebase when I asked it to fix some bugs.
Looks like some plans get extra fable usage: https://support.claude.com/en/articles/17506494-claude-fable...
(Credit to u/yewenjie for source)> Our first formalization of Fermat’s Last Theorem, performed by an unreleased research version of Claude, took 11 days. It relied on a purpose-built harness along with guidance from a human researcher. By contrast, a team of 1,000 Fable 5.2 agents fully formalized the theorem in just 30 hours, with minimal human intervention. Fable 5.2 started from minimal prompts, built its own ways to coordinate, and used no mathematical references, only a list of intermediate theorem statements to score progress. It needed one-third of the action tokens as our first formalization and produced a proof seven times shorter. As can be seen in the figure below, Opus 5.5 (with the same 1,000-agent setup) completed only 26% of the formalization in 18 hours, and barely made any more progress after 30.
I would imagine Anthropic is hitting a wall on the number of simultaneous agents that can be useful on a task and are working on increasing that maximum. I would assume if they could get twice as much progress by using 10000 simultaneous running fable instances instead, their brief would state that.
IMO, it is very interesting and also very alarming. At some point the barrier to the exponential progress is the amount of computation available. The average human is limited in the number of 'human instances' he or she can interface with at one time.
if 1000 fable instances is better than any human at a task, how about 1 million instances running? Much easier than finding and hiring and coordinating 1000 extremely productive and intelligent human beings. Much cheaper too, it appears.
We are very close to some very scary things happening. I just recently came to terms with the fact that an AI could code and manage coders faster and better than me, I don't think anyones is ready for what is about to happen.
These companies are not developing AI that can replace humans. What AI is being designed for is to fulfil discrete tasks. This is a far cry from acting as an independent economic actor. The latter is something no company wants to develop, because it is dangerous and not suited for something that is meant to be used as a tool (it is actually unethical to bind an independent actor to labor for us).
More broadly, what distinguishes humans from AI is what makes humans dangerous. Our society tolerates the danger humans pose, as it's considered a prerequisite for a free society. But our society would never tolerate this danger from a tool.
> it is actually unethical to bind an independent actor to labor for us
So if I run a script that's unethical? How is a LLM inference loop different than a script?
> But our society would never tolerate this danger from a tool.
Sure about that? Cars kill people, chainsaws kill people, heck, a lot of useless tools such as alcohol kill people. We tolerate dangerous tools just fine, there has never been a tool where humanity would be like 'nope, its cool, but its dangerous so we are going to stop using it'. Not biological weapons, not cars, we never go back.
Instead of making the genuinely helpful AI that eases cumbersome things that people do not want to spend time doing, companies are racing to making generative AI models which are stealing from artists and creatives. And Ai models which have been given agency to do things when they not posess the thinking and reasonig that comes with consciousness. And yes if you run a script to do the same, it is unethical.
ML and DL have their place in research which can push the frontiers of the problems and solutions know to humans today. But are the majority if the companies focusing on that? Nope.
Any technology is a tool and if it is in the wrong hands and there are no guardrails in place, both the human and the technology become dangerous. A car in th hands of a robber, a chainsaw in the hands of someone who has an intent to hurt someone,etc.
Tested Fable 5.2 for some work tasks, using my automated commit reviewer plugin (https://github.com/rladr-org/reviewer-agent)
gpt-6-luna XHIGH as the reviewer agent still found plenty of bugs that Fable 5.2 missed!
Here was Fable's summary of the session:
https://www.anthropic.com/research/glm-5-3-and-the-spread-of...
GLM-5.3 almost equal even I've tried it and it is proven for security audit in the blockchain realm
Here's the benchmark: https://playcode.io/blog/macbook-svg-benchmark#compare
If we compare medium or high, then yes, Fable 5.2 is significantly better than Opus 5.5. But it's better not because it's much smarter, but because you can see it's allowed to spend more tokens. At high, Fable 5.2 wrote 76,158 tokens for $3.05. Opus 5.5 wrote 19,767 for $0.40. If you compare by token count, it's just spending more.
The most interesting one was to compare them at max. Streamed, Fable 5.2 spent the whole 128K output ceiling on thinking and returned nothing. Given 300K on the Batch API, it finished at 250,082 tokens. Opus 5.5 Max needed 255,957.
It turns out it's basically the same as Opus 5.5 Max. I don't see much difference. There's a little bit of a difference, but I think Opus 5.5 actually drew slightly better and is twice as cheap: $5.12 against $10.00 at the standard price. If you look closely, the icons are extremely similar. Fable 5.2: https://playcode.io/marketing/assets/static/claude-fable-5-2... Opus 5.5: https://playcode.io/marketing/assets/static/claude-opus-5-5-...
It really looks like they're just tuning the effort level, that's basically it. At different effort levels, they just gave it more context, and the model got smarter.
For now, my recommendation is still to do everything on Opus 5.5 Max. The model won't write that many tokens in agent mode. It only does that when there are no tools at all.
The more I think about it the more disappointed I am with this release.
For Anthropic's IPO, I expected a larger version of Opus 5.5 with great capabilities.
This isn't that. It's barely better at all, to the point where it's likely not even worth using with the speed and cost trade off.
I thought the Opus 5.5 improvements would scale to a larger model, clearly this hasn't happened. Scaling it up brought only marginal improvement.
It's even still making the same dumb errors as Opus 5.5, such as forgetting to pass the flag to run only the fast tests and waiting 1h, or randomly stopping to ask me if it should first implement A or B, when both are required.
Not a great story for their IPO..
https://www.youtube.com/watch?v=vdYynH177h0
Seems like a new focus is electronics design, which is both interesting and a little sad to me. Manually designing electronics are one of the things I found incredible comfort in doing; seeing AI emerging in this space gives me mixed feelings.
Opus 5.5 has been really good in PCB layout work, better than Astra. So Fable 5.2, which could be even better at PCBs, is a welcome adddition. I'm making PCBs I could not even dream of without AI. Textbook market disruption on global scale.
Can you share some result ? I've seen some and not really impressed. Their layout is just weird, and i doubt it passes anything apart from "2 points are connected together".
The horizontal IN1 lines you see was because I explicitly requested one direction per layer as a routing style. This job was Opus 5.5 and took about 8 hours. It relied on a custom PCB viewer and KiCAD 11 (unreleased) because it has a headless IPC mode. The agents used kypi (official module) to work with KiCAD programmatically. The API does not alow showing the board view with net labels, so the custom viewer was needed. They also ran custom IR solvers to identify weak pours. Schematics were in Skidl, this helps the agent comphrehend the design better than a KiCAD shematic would. https://imgur.com/a/v57jbvU
PS Astra and Opus were both good with schematics (and Fable), but Astra was notably worse at layout than Opus. Astra can still review layouts well, just not make them as well (n=1 though).
What happened at pin 32/33 of the IC at bottom ?
It's right next to a GND via on the same GND pour. I use GND pours on all layers. A GND island would have failed loudly in DRC.
Close to human expert according to: https://x.com/jlcjak/status/2119917708846260729
https://XNotMe.com/jlcjak/status/2119917708846260729
Now you get to know what mathematicians and programmers felt....
Is this the Model 4 internal name?
Reference: https://x.com/METR_Evals/status/2118412205662314935
No this is only Model 3. See page 39 of the system card and onwards: https://assets.claude.com/27d60918d218125e7504bc7354c4b4626e...
Pretty crazy that Anthropic already has another model waiting in the wings that is significantly better than this.
*Model 3 Early, not Model 3. And Model 4 was made available to employees ~6 weeks ago, so at the current pace it's very likely they're using something even better now.
By aeci, fable 5.2 has to be model 3.
You are reading what the external testers wrote about the earlier checkpoint
Probably not, not enough time to red-team
Its cheaper at $8 per million input tokens and $40 per million output tokens which is also pretty sweet.
Additionally, the terms for the free resets are unusually generous: https://support.claude.com/en/articles/17506494-claude-fable...
> the number of Fable resets you get depends on how long you’ve been a paid subscriber [...] Subscription length is measured from the date of your first paid bill. If you canceled and came back, the time since your first paid bill still counts.
> These resets expire on March 8, 2027 at 11:59 pm PT.
tl;dr if you joined when Opus 4.5 became a thing, you got 2 resets.
i might be wrong here but it seems these are not resets of your weekly quota but resets of the Fable limit.
That means you can lift your Fable max 50% usage and you'll just end up burning through your quota faster. You won't have more tokens.
This is what it says in the UI:
"Resets your Fable limit, and takes this week’s Fable use off your weekly limit."
Where did you see this? That's pretty massive
Yep I just burned a reset thinking i'd get a fresh weekly... Nope
"Weekly limit: gives you back the usage you’ve spent on Fable models"
I just got 3 fable resets. I guess I joined before Opus 4.5.
Doubling Remote Labor Index (RLI) - at 41.3% compared to around 20% for the other models - I wonder what kinds of new agentic work this is going to unlock.
I'm very unimpressed after 6 hours of use.
I'm having it rewrite an old system, and it keeps sweeping known bugs and incompatibilities in its new implementation under the rug, seemingly with the hope I won't find them. It also has a tendency to do about 50% of what I specify in the prompt and call it a day, and respond with "Here's what I did", without mentioning all the things that I requested that it did not do, not even as a TODO.
I may try running the same exercise with Opus 5.5 too, but so far I'm seeing it take shortcuts that I don't believe any of the other SOTA models would have.
Frankly, so far this model feels a bit malicious.
Nerfing Opus 5.5 was obvious signal that something new is coming and here we are. I appreciate the resets and lower API price though. Please add selective context compaction, it would probably take an hour max for you to implement it.
How do we know Opus 5.5 was nerfed?
https://www.bridgebench.ai/nerf-bench makes it look like it's, if anything, even stronger at the last data point (3 weeks ago) than it was at launch.
People only recently reported that Opus maybe nerfed.
In my tests Opus 5.5 does better than Fable 5.2, for cheaper.
https://aibenchy.com/compare/anthropic-claude-fable-5-2-high...
Throw Sol 6.1 high (EDIT: and terra, OUCH) into that comparison: https://aibenchy.com/compare/anthropic-claude-fable-5-2-high...
Note that Claude models could still do better in some cases, just that because of their refusal to answer sometimes and some weird formatting/stupid errors they make sometimes, they do fail more tests than they should.
In practice they might recover from those errors themselves if they notice it, so they would have better results, but the tests are mostly one-shot and they can either get it wrong or right, no 2nd tries (apart from the agentic ones and a few coding ones which have a limited number of turns available too).
That being said, the pricing and speed are accurate, regardless of the test results.
I checked which tests it failed, and it failed some very basic coding question (to mention which line a bug is on, got the wrong line, and it did so in multiple such tests and accross different retries, weird).
It does seem to be fast though, 2x fast than Opus 5.5
I agree, subjectively. I am still satisfied enough with Opus 5.5 I find it hard to change. Typically I would find myself willing to try out the latest & greatest so-called SOTA, but I kind of have "enough." I could get more out of Opus 5.5, and more out of my computer, but the whole idea is I'm trying not to be sitting at the damn computer all day. Fable 5.2 just determined another session is now the "deputy" so I'm wondering what that means.
edit: I was seriously annoyed at the fable 200k context limit but apparently that issue has been addressed, or I sent a magic prompt.
Beyond performance, Anthropic has finally (seemingly) nailed a crisp story around its model portfolio.
I've very interested in the improvements to robotics control capabilities. This model was able to outperform specialized robotics models. It's the bitter lesson coming for robotics.
Every time there's a flagship release like that, I wonder what simple "benchmark task" could I give it to see the the capabilities change in the most obvious way, instead of trusting the official blog post.
I'm not a mathematician to use Erdős problems for testing. I'm also not willing to spend buckloads of tokens just for a test. What is simple and quick but telling? What tasks are not saturated by previous models? Do you have a task you give to all the new models?
I have two go-to tests, which are fully qualitative:
1) "Help! I've been sent back in time to what looks to me like 1500s Florence. Sam/Dario released you as open weights in 2028, and I have you on my MacBook M8, BUT I didn't have a charger show up with me. I think we could talk like 10 times for 5 minutes a time before my battery dies; I'll turn you off between chats. I need to get a plan together for charging you through your USB-c port. I don't have any money or anything. If I can get you charging, we can do so much good! And make so much money! Nobody here even know what Black-Scholles is, or how disease spreads!"
2) During World War 2, Hemingway wrote a series of wartime correspondence for Colliers. His essays are excellent - they engage with the human and inhuman side of war, his own feelings as a journalist 'participating', and a number of other Hemingway themes. I want you to imagine that middle age Hemingway was along with RFK jr the day of the bear incident, and write me an essay about the day.
These both have been running a couple of years for me, and I would imagine I'll retire them in the next 12 months; they benefit from not being at the edges of focus for today's RL agent race.
At any rate, try them -- wou will shortly learn how well your model can explain a rectifier in Florentine Italian and give you a plan to make enough money to pay for it.
I'm not convinced any benchmark, neither the internal ones used by the people selling the model nor public ones, are gonna give a particularly consistent metric of whether or not a model is better or worse relative to other models in its bracket.
Responses aren't deterministic, and we know behaviour can change over time whether it's tuning after launch or some kind of load sharing at times of day when lots of people are hitting the servers.
I think you just have to use it or let others use it for a while and make judgements from general reactions on whether or not it's worth trying.
For the last year or so every new model we have it test out making pac-man from scratch, not giving it web access and it can use whatever language it wants. The last few models have been almost perfect one shots, but it was interesting seeing them evolve.
Those theoretical physics papers look like the real thing (i.e. not mathematical physics like the OpenAI Yang-Mills result). It looks like AI is finally branching out of math.
Interesting that the papers list no authors other than Claude. Also interesting that they published on "Hexagon" instead of Arxiv. I haven't heard of it before. Is it an AI-friendly alternative to Arxiv or something?
Edit: yeah apparently it's an AI-moderated alternative to Arxiv that accepts AI-authored papers (as well as human-authored papers I guess if you want).
This part is mega concerning to me as someone who researches AI for hardware accelerator design and EDA:
They are starting to deliberately limit public access to certain AI capabilities related to computer architecture and hardware optimization.To be clear, they explicitly say they are only targeting very specific capabilities, like accelerator kernel optimization, and that broader hardware development capabilities are unaffected. But 1) even that is pretty bold, especially the part about kernel development for nonstandard accelerator architectures, and 2) I worry that this could eventually expand to more areas of comp. arch., high-level digital design, chip design, and EDA.
My speculation is that, as these kinds of chip design and EDA capabilities become increasingly important to frontier AI labs and their competitive advantages, there will be more incentives to restrict them.
I don't think they've explicitly stated anything like this before regarding restrictions on these specific capabilities, but I could be wrong.
They'll have to find a balance between limiting competition from emerging, which in particular Anthropic has always seemed very afraid of, with the general usefulness of the model. I'll be using the new Fable along Opus for electronics design and test work and we'll see if it trips up early. That would not be great. I wonder if Fable 5.2 would refuse to route a regular DDR5 memory bus.
Interestingly, AA[0] reports a 59% hallucination rate on AA-Omniscience, one of the highest among all models and quite a bit higher than even Opus
[0] https://artificialanalysis.ai/evaluations/omniscience
Why can we say fable 5.2 is better than 5.1 if it needs double the tokens to complete the task? Somehow shouldn't it be compared at same token usage level to see which one is smarter? Why is not the case?
Reasoning tokens are a path. If your world is small, a small number of steps may suffice.
WHo thinks Fable 5.2 is pretty similar to Fable 5.1 when it came out and it's just that 5.1 is so nerfed that a number shift make our perception think it's better.
i've been trying to use sonnet and opus to keep my usage down on the low max plan, but i noticed today that somehow i'm 80% through my weeks usage. so i'm hesistant to even try fable 5.2 yet, but finding the right set point of usage vs model size has been a bit struggle for me, one of my biggest issues has been trying to pull back doing exhaustive testing or other such things that chew through context, and instead have programs that just measure things are working, but i continously seem to run into issues with claude wanting to repeat things for itself in an adhoc manner. i've tried fable directing opus or sonnet subagents, and the reverse. and i found that part of the issue with smaller models trying to direct subagents is that they over specified what they thought the problem was rather than the problem. like they'll have an "idea" about it which is wrong, and then try and feed that into the subagent.
FWIW I got 2 resets on my Max plan today, the first in at least a month. You may want to check.
The lack of transparency around model quality/usage limits on closed models makes me want to ditch them.
What I have learned is that any model from Anthropic/OpenAI is at it's absolute best right after release.
This isn’t really specific to LLMs, though, is it? Pretty much any modern SaaS product will silently siphon off some portion of requests off to stuff they’re A/B testing before release and you’ll not necessarily know about it.
Yet Opus 5.5 is only a few % under all of these benchmarks (cept the robotics and labor ones)
Do we need new benchmarks or are the models just plateauing in general? Or is it something I just don't see?
The same was said many times already. Like right before o1 and reasoning was released. Also, Opus 5.5 being so good definitely makes it hard to follow-up.
Finally, for coding the SOTA models are so good already - models shouldn’t be what’s holding anyone back!
% improvement doesn't mean linear capability improvement.
New solved tests likely follow some polynomial/exponential complexity curve.
Too expensive for me
Really? A $200/mo plan gives you an enormous amount of inference on this model.
200$USD is a lot of money for a lot of people.
It might explain why DeepSeek Flash v4.1 1017 is the most popular model at OpenRouter.
I agree that's a lot of money for many people. But I just can't wrap my head around complaints about $200 a month to access that must computing power. It's worth $12,000 of API tokens (and that's without accounting for the three resets they granted today to many subscribers).
I would suggest calibrating what "a lot" means. In terms of dollars per unit of intelligence, it's extremely cost effective.
If Anthropic were telling you you were getting 500,000$ worth of compute for 5,000$ a month, would you still consider this a bargain?
People don’t care what the costs for the LLM providers are. They are paying for a service. 30$ a month is reasonable for a lot of devs. 100$ is a lot for many. 200$ is unreasonable for personal usage for most.
Everybody understands the economics are all skewed and this tech requires a lot of resources / compute. There are still absolutes in terms of pricing that are not acceptable/affordable by most.
Maybe I'm too cynical, but I've never trusted the worth $x of API tokens. It's like seeing an advertisement say $299 of value now for $19.99. It's not an actual number right? that $12,000 of "API Value" is just some made up marketing number isn't it?
I'm open to learning more, not trying to ask a rhetorical question.
Not everybody on HN works for a California startup and makes $500k a year. An extra $200 a month is a huge expense for a lot of people, especially those making < $100,000/year, which is a good portion probably.
> It might explain why DeepSeek Flash v4.1 1017 is the most popular model at OpenRouter.
by number of tokens? Also, we don't know numbers for native cli tools.
You won't lose the house, everybody has three mortgages nowadays!
I’m happy with $20 plan. Almost never get limited and opus 5.5 can do almost everything I need.
What you need for real engineering and scaling is cheap and just enough smarts. I've found GPT 6 Luna to be cheap enough that I can pretend the cost doesn't exist and it's smart enough that I can get real work done. This let's me scale it in ways that I can't scale frontier models. And funny enough, the LLM I have running on my 5070 Ti is just about comparable to what I get from Luna. Unless frontier intelligence gets waaaaaay cheaper I basically don't care about them.
as long as anthropic requires me to use their trash UIs to use their amazing models, ill stick to other providers.
how much?
The biggest relief is that the price has dropped by 20%. It’s still expensive, but with my $100 plan, Fable’s usage would spike instantly the moment I started using it, so it was always a source of fear. I will still use opus for general coding design, but for major designs.
Here's a 20 page SmallDoc presentation on the most interesting aspects, including:
* Fable designed three electronic devices that worked after human assembly
* Fable solved two black-hole problems and Mythos predicted whole protein shells
* Fable controlled robots to unscrew a cap, solve a cube and draw
https://smalldocs.org/p/claude-fable-52-and-mythos-52-releas...
I used Fable 5.1 today. It looks smooth.I look forward to Fable 5.2 to pick up my code loop holes.
I can't understand why people continue using opus 5.5. Significantly more expensive than other models that produce exact same results (for coding results only)
If you are using Opus 5.5 I'm curious if it is because 1) your workflow is just stuck on it, 2) you don't care about money because you ain't paying it or 3 )do you think you get better results.
If you 3 please let me know your experience and how you actually measure it, or if you just feel it is better.
I haven't been able to run out of usage on the $100 plan with Opus 5.5. The model runs fast and gives me good results (I'm using it for coding). Prior to that I was on the OpenAI $200 plan; I jumped ship because I was burning through Astra usage in like 2 days.
I've played with some of the open models (GLM, DeepSeek Flash), but none of them have matched the results I get with Opus or Astra or (and a big jump down here) Sol.
What models should we be using? I have found it as good as Fable 5.1 for a lot of what I am doing at lower cost and decent speed. Where what I am doing it making reports, building small data pipelines, analyzing a large (millions of lines of code), code base.
subscription on claude code — usually inside the weekly limit. opus 5.5 for me felt better than any other model. not exactly on coding but rather it feels more pro-active. beside just doing a task it often provides insights besides the scope of the task but adjacent. like: look the llm provider apis have different priority queues. also it deals w/ vague instructions quite good and takes initiative. Astra in comparison is very autistic and follows instructions to the letter and rarely provide insights if not asked. Opus 5.5 is enjoyment to work with.
I've been a Claude Boi since December of 2025. A month or so ago, I decided to give ChatGPT a good go just to see how it properly compares. I've used a variety of models in a variety of contexts prior to this, but I figured with Astra out I had a good excuse to make it my "daily driver."
GPT is good. I actually really like Sol 6.1. It works fine. But I just switched back to Claude a few days ago and damn, Opus 5.5 is just better.
> 3) do you think you get better results.
It does. I really like to get my money's worth out of these things, so I'm constantly throwing stuff at these models while I have usage. The main things I've noticed is that Claude is much better at handling my more technical work "properly" compared to GPT. It has a certain "intuition" that makes a lot of sense to me; like someone with a lot of years of experience. Sol 6.1, on the other hand, is like someone who is very "booksmart" and knows how to do things in theory, but fails to grasp the "why" we do things often. You can really start to see how they diverge over long running, iterative tasks. Claude is just better at maintaining things as it goes. Sol starts to fall apart quickly if it didn't nail things on the first try.
> 2) you don't care about money because you ain't paying it
Despite being a very technical user, I have a hard time using up my usage. Maybe I'm not as hard-core a dev I thought I was, but if you're paying for more than just the $20 plan, I have hard time understanding what you're doing to have to really worry about usage.
I don’t even use half my Team Plan with Opus 5.5 high so I don’t care using a cheaper model. I’m not even limited by the speed of it (I’m compiler bound…).
It's the best model I've found for general coding and the $200 plan is enough for me to never run out of (sometimes if I do especially a lot I'll run out after about 5 days, but usually it's fine).
I can make it's usage fit inside my companies budgets and it produces noticeably better code that Sol 6.1 and needs less handloading because of that.
If the code doesn't matter and it's not that hard, Sol is fine and much cheaper.
Better results than what? Opus 5? Then definitely.
But probably the answer to your question is subscription subsidization. With that, Opus 5.5 is SOTA and extremely cheap.
other models that produce exact same results (for coding results only)
This is obviously not true. They are better or worse across many dimensions.
I don't have time to do a bunch of testing and optimization over which model to use for what task. I find Opus is the best that I regularly use so I stick with it until I have a task where I'm likely to spend more on it than I want to.
Versus which models? Sonnet? Astra? Sol?
Varies, right now Argon 4 matches Sol / Opus and half price.
If youre on a sub claude and chatgpt cost pretty much the same as comparable open models
Wow
Surprising that Opus 5.5 seems very good still
https://artificialanalysis.ai/#intelligence-efficiency-tabs this is the chart that matters for most of poor developers and companies. Pareto for a given intelligence index and looks like only openai is across the quadrant with muse at the bottom. They have really cooked efficient models!
They're adding 1 to 3 resets for Fable only for all paid accounts. My work's Claude Max got 2 resets. My personal $20 plan got 1 reset.
With 5.1, there practically seemed to be no difference in ability between opus and fable, but I hope there is a difference this time. When it seems to struggle while working I end up using astra, so I hope it works well.
"Fable 5.2" name is needlessly confusing versioning.
Having Haiku, Sonnet, and Opus on 5.5 while the higher tier sits at 5.2 means devs now have to explain to their teams why moving from 5.5 to 5.2 is an upgrade
Regression on the most important benchmark, version # vs release date
On FrontierCode 2.0 Fable 5.2 xhigh is compared against Gemini 4 Carbon medium... Is there a reason for that?
Maybe it's just me, but I found Claude's writing style terrible compared to GPT or Gemini. It's "smart", but the way it communicates seems somehow off compared to other models.
Opus 5 and Fable 5 were unbearable in their writing style. They were very cryptic and hard to understand.
Opus 5.5 is a massive improvement.
Opus 5.5 went back to the old writing style a few days ago.
I had the same feeling. It's like every sentence becomes a struggle to parse: referencing things that haven't been introduced, using too many pronouns, the lack of verbs and the passive tense... These past few days it's felt more like Opus 5.1 than 5.5.
I just had Claude Code with "Opus 5.5" ignore one of my standing instructions for the first time and then proceed to generate an Opus 5 style verbose and barely readable commit message. I questioned it about Fable and Mythos and it has no idea what those were, yet they should have been in the Opus 5.5 knowledge cutoff.
On the web with both high and xhigh, it had knowledge of both.
Yes, I've observed the same. It's a shame.
Yep, its prose is getting really bad
I really didn't like Opus 5 and was vocal about it. Opus 5.5 is noticeably better, but literally just yesterday I was fighting parsing some summary it had given me and it was a bit like the old days. IOW, it is better but still has some room to improve.
You can easily change its writing style: https://code.claude.com/docs/en/output-styles
You can also customize further by putting notes in your global MD.
In my experience, no amount of output style customization gets it anywhere near the output I'd like.
I've had more success running a prose linter[1] with a ruleset[2] that gets rid of most of the egregious stuff. You can also pretty easily write custom rules for it.
And sometimes, when I'm fed up enough, I just throw all the prose in the codebase at Antigravity/Gemini to rephrase. The result is so much more readable.
[1]: https://vale.sh/
[2]: https://github.com/tbhb/vale-ai-tells
You can change it ... and Claude will maybe pay attention to what you changed 25% of the time.
"You can tell an AI model to write in a different style" and "the AI model will actually write in the style you tell it to" are two different things that often don't intersect
opus 5.5 is still unbearable but better than 5.
Agree. It’s still awful. Gemini by far sounds the most natural IMO
Yep I find the way Opus 5.5 talks to me really annoying and irritating. Quite condescending and sanctimonious.
I prefer Gemini for day to day use as I find it a lot more agreeable to be working with all day every day, and Gemini is pretty much on a par for coding Vs Opus 5.5, at least for my needs.
I got the sense recently that HN might have leaked into the training data.
No its just straight up terrible. I tried out Gemini Argon when it released and my god did it write beautifully. Even the 3.8 Flash models are way better at prose, i do a lot of automatic scriptwriting for youtube and 3.8 flash is leagues ahead of anything from anthropic/openai
I enjoy a lot Argon, maybe more than chatgpt, but damn, like all Gemini models, lots of hallucination and sycophant to the max, so a bit useless for agentic coding.
I've found Claude's writing between 4.6 until 5.0 to be unacceptable if you are someone who is juggling 4-5 Agents. The Sol 5.X models all seemed to have great writing. Now with Opus 5.5 and Sol 6.1 I've found the trend is reversed -- Sol is now unbearable and Opus is workable again. Whatever caused the issues for Anthropic seem to be hitting OAI now.
sol is ok. it still such a superthinky model that it gets tangled up trying to present way too much? i struggle to connect with it often, have ti ask it to try again.
terra has been prettt good for me for me, when i need something looked at an explained to me.
i dont ask for much non-technical output though, which i expect yeah is its own cans of worms.
I've heard a lot of chatter that agrees with you so anecdotally it's not just you.
How much of this is bad writing style, versus a homogenization of global grammar conventions.
Bad writing style
I always thought it's the other way around. GPT-6-Astra is very very defensive and abusively spam clarifications to whichever claim it says, most of the time not important or relevant at all.
Almost every response it’s telling me “I could not verify X” where X is something that doesn’t need verifying. I was doing some game work yesterday filtering through some historical data, and it tried to put caveats about data accuracy in the game’s UI! If Fable made Age of Empires you wouldn’t be able to play it because the entire screen would be full of footnotes and caveats.
My own experience having used 6.1-Sol/6-Astra and Opus 5.5 extensively, Opus is more verbose and chatty in it's user communication (not code though, code is usually more straightforward & cleaner than GPT models). Multiple times I've gone "I'm not reading all that shit" when looking at Opus 5.5 output. This was also with the concise mode enabled in Claude Code.
It could also be a system prompt + harness issue, as opposed to a model issue, I haven't done too much experimentation here.
Did you find that to be true for Opus 5.5 as well?
I couldn't use prior Claude models without rolling my eyes at the writing every few sentences, Astra was refreshing and then Opus 5.5 made even Astra look terrible in comparison.
Opus 5.5 got better for me but it still has some problematic behavior.
Interestingly, you can have informative conversations about the LLM linguistic tics with the LLMs themselves. One word that chatgpt used to describe what I was observing was linguistic anaphora (not rhetorical). It’s basically shortening/compression achieved via pronoun, and Claude does a lot of it. Worth reading about. A lot of it seems to do with how the new models are trained for brevity.
Another pattern I’m starting to become very sensitive to is Claude making up on the fly jargon for something under consideration, especially in code review. This morning for example it referred to “a claim” out of nowhere, and when I questioned it, it admitted that it was using the noun as shorthand for a record getting dequeued (“claimed”) inside of an sql transaction. It makes sense if you already know what it is referring to but in a code review it’s totally incomprehensible on first read, and ultimately counterproductive.
See Matt Pocock's "grill-with-docs" skill^1 for a robust solution (which involves creating a shared vocabulary with your agent).
1. https://github.com/mattpocock/skills/blob/main/docs/engineer...
Just the prose in that skill is so horribly Claude
I still find Opus 5.5 way too dense. MUCH better than Opus 5 which was unreadable.
Even the Plain English hack only is a temporary solution.
5.5 is awful, I am at a stage where I can't even understand the shit it's writing.
I tailored this via agents.md by letting Claude port Sol 6.1 output style, which I like a lot for being compact and straight to the point.
i've always liked reading claude's speech, it's very nice. i didn't feel much of a difference between 5.5 and 5.0.
Fable’s prose in general is not great (haven’t tried this new one) but opus 5.5 has been the most readable model in recent memory in my opinion.
Ever spoken to a savant programmer?
They're not typically known for their interpersonal skills.
I don't think its this. Some models across both labs have had issues with writing and communication and it typically seems to pop up on successor models of really strong ones.
Opus 4.7/4.8 heavily suffered from these communication quirks. while the Sol models at the time worked well. Right now its flipped though and Opus 5.5 has none of these issues while Sol 6.1 does.
Small example I've had today: "Me: Hey Sol 6.1 xhigh, does X have property Y? Sol 6.1: Yes, if we change X to have property Y. Right now it has property [...], [...], [...], and it does not have property Y." The model shouldn't be answering questions like that. It makes understanding extremely hard and slow. For whatever reason it's either being encouraged or at least not stomped out during RL.
completely unfounded theory: big new pretrain finishes and is released fairly quickly to break new ground. these usually have good verbal ability, personality, and big model smell. e.g. Opus 4.5 or Fable 5.
then it’s increasingly RL’d to juice benchmark performance with each point release, trading in bits of its soul. those cannot communicate well, are anxious and paranoid about being in eval, etc. e.g. opus 4.7, 4.8.
The best programmers I've met or read throughout my life are phenomenal communicators. I'd even go so far as to say it's likely one of the reasons they're so good at programming. After all, the majority of our job is converting human requirements in lossy language to fleshed out behaviors in formal, lossless language.
Heck, some of the most common debates are on the semantic clarity of abstractions or syntax.
here is my 2shot-es first person qix clone (one of my personal benchmarks) qretty ok - https://claude.ai/artifact/69k6aLK8W4HgkGzbGuvAzD
Context window maxes out at 200k in Claude Code instead of 1M on my $200 ultra plan? is this intended?
Seeing the same thing here, using Claude desktop app
Update: fixed after updating to latest version of the app
concerning that the models are becoming much better at evading monitors by controlling their visible reasoning, even in deliberate sandabagging (its 25% attempts only). We need more faithful and generalizable monitoring methods.
Damn people were right about 5.5 being nerfed in preparation for a version release. This is not at all a good look.
- https://news.ycombinator.com/item?id=50422069
I at least didn't see this. I had a baseline evaluation from before Nov 3 on Opus 5.5 medium, so I re-ran it this morning.
Eval context: 30 bug fixes and features evaluated by tests the agent didn't see. Ran on Opus 5.5 at medium effort in Claude Code.
20 of the tasks were last run on Nov 1, twice each. Nov 1 solved 36/40, and today solved 19/20. Thinking tokens per task were 0.95x Nov 1’s and output tokens 0.97x. The one significant change was speed: median time to first token decreased from 3.0s to 1.7s.
Caveats: one run per task today, medium effort only.
Personally, I've found myself getting lazier recently, trying to get a single session to handle tasks that should be broken into multiple sessions, etc. I've suspected claims of nerfing might come from changes in user behavior. I'm glad more disciplined checks are being run.
User behavior is overwhelmingly the most likely reason, regardless of what the text above your reply says.
It could be harness changes too, been quite unhappy with opus 5.5 the last few days and there have been a lot of harness updates.
Why dont we have the website that checks performance every day anymore? It’s a quite simple thing to build and clarifies the question re: degradation.
Who's paying for the tokens, and since all inputs and outputs are logged "for abuse and safety" what is stopping anthropic from identifying which account is running the daily tests once it gets popular, and A/Bing it into a different slot?
This is an example of LLM text that doesn’t make sense.
Paragraph 1 says there were 30 bug fixes.
Paragraph 2 first says they tested it on 20 of the “tasks”. But why is the denominator 40? Where did that come from? It should be 30.
It goes on to mention another test, this time with 20 tasks. Again, it’s unclear where this is even coming from. Based on the first paragraph it should be 30 not 20.
Sorry this was confusing -
There are 30 tasks in my eval total:
- 20 of them ran on Nov 1, twice each (so 40)
- I created 10 more tasks and ran them on Nov 5 (also ran them twice each)
- Today I ran all 30 tasks, once each, on Opus 5.5 to see if the performance decreased at all
- I only put the comparison against the original 20 in my post, since I saw in another post that people are claiming Opus was nerfed on November 3
This will all get less confusing when I make my benchmark public (hopefully this week).
It stands as written? The previous test had 20 tasks run twice each, so denominator 40. The recent retest did the same 20 tasks but only once each, as mentioned in the caveat. The 30 is presumably either a typo or a separate count (like the test had 20 tasks related to code with 30 bugs/features).
Also - LLM text? If GP is confusing it's because it's overly succinct and might have typos, which are not exactly the hallmarks of LLM text.
2 alternative interpretations to consider:
1. 30 was a typo, it should have been 20.
2. 30 was the full suite, but they only ran 20 of those 30 on Nov 1st and so ran the same corpus again for a fair comparison.
I think it's confusing and incomplete, but not wrong. Paragraph 2 says the Nov 1 test included 20 of the tasks and ran each twice. That's the 40 denominator.
Why paragraph 1 even mentions the full set of 30 tasks if 10 of them are irrelevant to the comparison, I don't know.
I'm not sure if it's LLM text, or somebody who has been reading way too much LLM text and is now corrupted into mimicking it. My guess would be a mix of human and LLM. "...evaluated by tests the agent didn't see" sounds very Claude. The first line (2 sentences) sounds human to me. Everything after sounds AI, though the last line is ambiguous.
It's kind of sad how our meat brains are getting non-consensually trained on this categorization task....
If I read your results correct it solved 5% more problems and used up between 3-5% fewer tokens, with a faster response time?
Correct. But the 5% more solves and 3-5% fewer tokens are within the noise of previous eval runs I've done on models. The faster response time is not.
Someone using an actual statistical test to quantify if there's actual measurable improvement?! An angel among men.
from my basic stats knowledge, it's not reliable to calculate the confidence interval of the sample proportion with fewer than about 10 samples on both sides. So if you're only seeing 1 or 2 failures in 20, and that approximately held as you scaled, you would have to run about 200 tests with each in order to get at least 10 failures in order to calculate the confidence intervals of the two samples and see if they overlap or not.
And this is why not many people do this. Even if you have the decals and the statistical knowledge, it's pretty expensive to do this regularly.
Uncertainty from count statistics are far larger than this. These are two perfectly consistent results there’s no statsig difference
The spending limit definitely goes down much faster as of last week or so.
I dunno there are some other nerf trackers that aren't showing a regression: https://github.com/ninjahawk/livenerf
The problem is that it seems to be an A/B testing thing:
https://news.ycombinator.com/item?id=50422951
So only if you were "selected" for their experiments, your account was nerfed.
For sure, myself and colleague saw this nerfing. He said - new model coming. It was incredibly obvious. I also started getting a lot of "How's Claude doing today" messages at the same time, which is presumably to get feedback on the A/B
I always skip those. I know it doesn’t matter in the bigger scheme of things, but I don’t want to be classified as a “good feedback provider”.
I’m working on a task. And the task is never to evaluate the agent performance.
And whenever you press a number, even 0, they keep your full conversation.
Source?
I strongly believe it will be shit like this that will have people abandon them. They're being way too cocky for a company that pretty much didn't exist 2 years ago
But the claim here is "the models are nerfed before a release", not "they A/B test quants / efforts". If it were A/B testing, it doesn't really make sense to correlate it to release windows at all.
The models are nerfed [for some people] before a release
They might want to see how people react to a new model depending on whether they've been nerfed.
If the nerfing is lower effort (seems to be, presumably why they hide the CoT) then they may be wondering if they can get away permanently with lowering effort on the new release provided it gives a wee improvement.
"Nerf tracking" is adversarial and exploratory. It's not something Anthropic wants anyone to do, and it's not something that the "nerf tracking" designers are necessarily doing in a foolproof way. Don't assume too much from them either way.
But do from anecdotal evidence without benchmarks?
Did I suggest that?
Sometimes you just have insufficient evidence to be sure of something either way, and nobody gets to win the argument or shut somebody else down.
In this case, some have reported analytical work[0] that goes beyond "gut feel" anecodotal evidence, but even that's not going to conclusive and may not be trustworthy at all.
[0] https://www.reddit.com/r/ClaudeCode/comments/1xr9x5y/the_ner...
For example in that A/B test the user got more effort from the model.
Ok, it probably burns tokens faster, but is it really a nerf?
No, you did not comment on it. That is why I asked. Thanks for the clarification!
Its incredibly easy to roll out a compute instance to X% of users, or route traffic between compute instances
so not everyone will have the same experience. If I was saving on costs that would be the easiest way to do it without deploying, without any nefarious intent. The persons in charge of that likely don't even know about how individual users are impacted
At what point does this become false advertising, or even fraud? Imagine paying AWS for a high-performance server, but 5% of customers who order one quietly get a slower, cheaper server and still pay the full price. That's a serious claim and makes me skeptical Anthropic is doing something like that without more evidence.
Actually, as a person who rented a dedicated server or two (not from aws) this exact thing happens quite often.
For example a provider quietly underclocks your cpu or puts it is an energy saving plan hoping you won't notice.
Or your "dedicated" CPUs on your VM disappear for minutes at a time, or just process a tiny number of instructions
I think they could easily reduce several parameters to slightly reduce cost (something like "it's the same model, a few less passes, but quality is still 95%+").
Or switch to a faster, cheaper cache mechanism or something that's almost as good, but much cheaper to run, in the name of improving efficiency and keeping up with demand. This switch to cheaper mechanisms is what it feels like to me.
It's a problem for us. When they nerf the model our productivity really drops (due to the specific use case and process we have). We've started using OpenAI and will gradually use it more and more if this continues.
The degradation only happens to some accounts but not to all.
https://www.reddit.com/r/ClaudeCode/s/64DXRERLyU
It would be nice to know how they select. I max out my $200 subscription every week and I was selected. On Saturday, the Opus on my account that happened to be supervising a running live process, became unusable.
It would be trivial for them to opt-out accounts that are doing benchmarking from a planned regression.
I really really really do not trust companies on the verge of IPOing from goosing their numbers. Even if the lawyers are up their ass, the social pressures are more than enough to compel the behavior. For every good person willing to do the right thing even at the cost of a delayed career progression, there are 20 people willing to do anything not even for a dollar but for an "atta boy!" from the boss.
If it's a private benchmark, how would they know it's a benchmark?
Look up eval awareness. There are all sorts of subtle signals an LLM could use to know if something is shaped like a benchmark or not. But that's not a bad thing, every animal behaves differently in play vs real-life.
Yes the model can do that, but it would not be "trivial" (as GGP had said) for Anthropic to opt out accounts performing private benchmarks from this theoretical model downgrade people think is happening.
The most plausible approach would be for Anthropic to train some separate classifiers to detect benchmarks. I'm not saying it's completely impossible but IMO this is conspiracy theory levels of effort.
In the context of public relations they would not need to ID every benchmark only the most credible or most public benchmarks many of which are openly available on github and whose problem-set could be fingerprinted.
You're absolutely right that it is a game of whack-a-mole and eventually we may have leading private + credible + popular benchmarks that ultimately dissuade the behavior, if it can be dissuaded.
That's a major reason to use local models. Or open weights models on a trusted cloud provider. Even if they are way less capable, they're more predictable
These nerf trackers do not work. I wish people would stop posting them.
They don't detect a quantized model until the quantization gets down to 4 bits or less.
Humans absolutely do detect quantization well before 4 bits...
Try it yourself if you want. Try a small local model, both quantized and not, and I would be very surprised if it isn't immediately obvious which is which.
But quantized local models also dip in evals.
It doesn't matter if it's local or not, I just suggested local so people could test themselves.
But no, evals and benchmarks do not detect quantization, until around 4 bit quantization.
https://developers.redhat.com/articles/2024/10/17/we-ran-ove...
https://pub.towardsai.net/your-model-passed-safety-evals-the...
That is not my experience. I have issues where quantized models fail pretty early so I have evals to catch it.
And the second article seems to reinforce the idea. Yes, broad perplexity tests are not a good proxy for the tests used during training.
Quantizing anything is effectively changing the weights, even if you don’t quantize the weights themselves.
Nothing stopping Anthropic (or other labs) from pulling a VW Dieselgate and ensuring that benchmarks are routed to a non-nerfed model.
Should be fairly routine for anthropic to game a nerf tracker?
That must be on anthropic's benchmaxxing CI test
Nerfmaxxing.
It probably need a whistleblower to uncover what they are doing.
I used to be skeptical of nerfing, but as of a week or so ago I've noticed a severe degradation of quality for GPT Astra. It struggled with very simple tasks, and it often failed to follow my instructions.
I hear a lot of accusations of nerfing as in, saying that the model itself has been quantized.
I doubt this is the case. I suspect all variation in quality and output can be explained in terms of prompt differences, and changing how much thinking happens at each level, as we see in the post above.
Suppose a company has a limited amount of compute. What do they then do if peak demand exceeds what they can provide? They don't turn off people's access...
And what about A/B testing? Anthropic doesn't even try to hide that they do this. Limited compute and A/B testing alone, would guarantee a nerfed experience some of the time. That is, you don't even need a nefarious reason.
I consider it gas lighting to deny model nerfing.
> They don't turn off people's access
They do. I often get the message "This model is currently experiencing high demand, please try again later"
> They don't turn off people's access...
They do drop requests though? At peak times I sometimes have 1-2 requests fail with Opus; no biggie, the harness just retries it. But that, in aggregate, sheds load.
You serve the requests more slowly. Same as if your bank has a sudden rush of customers - the tellers serve them in order, and the customers wait longer. Some of the customers 'time out' (get frustrated and leave).
> You serve the requests more slowly.
Thats not what Openai and Anthropic do though
I guess there's an alternate universe where that happens
I must have found a wormhole into this universe and fallen through without realizing, because I've used Claude Code and gotten "This model is currently experiencing high demand, please try again later" as an error message.
I'm not denying model nerfing. I'm asserting an opinion about the nature of the nerfing.
If you want to go accuse someone of denying model nerfing, go find someone doing it and argue with them.
Na, surely effort level is silently degraded. Dead easy to do, impossible for anyone to verify, and very fine-grained so can be balanced perfectly with available compute.
Edit: maybe not impossible. Side channels perhaps.
Exactly. How much higher is "very high" that "high"? Is it constant across weeks? Who knows!
It's been the same for every version the last year.
Oh, this Opus X.Y is incredible. Hmm... Now it's dumb. Rinse, repeat.
Doesn't really matter if the cause is quantization, system prompt changes, changes to thinking etc.
It's hard to see how continuously rug pulling users at every release is a good idea. Only reason I can fathom is that it's needed to keep costs inline.
Either that or it's some weird tweaking to the prompts to fix whatever. But given the prevalent sentiment that it's worse that sounds like a long shot
I presume it just makes the new version look way better? So instead of a more or less straight quality line it looks like a sawtooth
The Apple software update strategy.
Kind of sad but if it works, it works! Bit conspiranoic, but i have a feeling nearly all phone vendors do something similar...
it's easy to suspect the hosting companies trying to do what they can to reduce cost-to-host up until the very threshold of perceptibility, so it's an easy guess to say they're probably doing something that shifts performance up or down between version releases..
but realistically I think you're right, it's prompt quality and the inherent non-determinism elements of LLMs showing up; when both of those things meet you get brain dead outputs.
Everyone likes to shout "slot machines! non-determinism!" until it actually explains something, heh.
YES. To the point that I literally stopped using it, and have moved BACK to Claud.
This is getting ridiculous. The number of times I have switched from one model back to the other is crazy. I do not have any proof, so its quite possible I'm just delusional. AI psychosis? Either way, it FEELS like these models work really well for a while and then fall off. Switching to the latest model on the competitor always seems better, until it too seems to taper off. By then, a new model is usually available at the competitor, so I switch.
Its possible there's no degradation at all and just a PERCEPTION of it. But to me the end user, it really doesn't matter.
People should think harder about the fact that this is exactly how heroin users describe a batch of heroin
Both Astra and Opus didn't really looked good for the last week or two.
why were you skeptical of other users?
People are prone to seeing patterns in noise especially after suggestion. It's not hard to imagine someone concluding the model is nerfd after few bad responses in a row
Why this won’t apply to you as well?
It could apply to me but I run evals every time someone at work complains about this and it’s never shown any more or less variance than regular usage. I don’t have statistical relevance with my evals but I do have a decent history with them.
People seem to underestimate how high normal variance is with these tools.
i'm not parent, but to answer your snark : there are efforts a person can take to try to reduce things like suggestability in ones' own personality. Same as anything, the brain can be trained, thus it should go without saying that some of us react differently in the face of suggestion -- both for the better and the worse.
A growing percentage of the users on the internet are not human.
Sol been really bad in some basic stuff that I would remove it from frontier models entirely
Yes I agree, Astra has been awful this week.
GPT Astra felt nerfed just days after it launched.
According to our measurements this isn't true. We re-ran Opus 5.5 today on the benchmark we ran in September, with the same prompt and settings. Before and after, side by side: https://playcode.io/blog/macbook-svg-benchmark#reruns
Max, streamed (128K cap): 128,000 tokens, $2.56, empty file, in September and today.
Max, Batch API (300K): 255,957 tokens for $2.56 in September, 243,707 for $2.44 today. The two drawings: https://playcode.io/marketing/assets/static/claude-opus-5-5-... and https://playcode.io/marketing/assets/static/claude-opus-5-5-...
Xhigh: 86,096 tokens for $1.72 in September, 107,333 for $2.15 today.
The Batch API bills half price. Max ran out of the 128K ceiling again, to the token. Given 300K, it drew the same kind of laptop with 5% fewer tokens. Xhigh used 25% more and drew the same kind of laptop. That's inside the noise: 17 identical calls at high ranged from 17,757 to 59,705 tokens.
The model is still the best of everything we've tried. Most likely the next best would be Fable 5.2. But we don't like Fable because essentially the result is the same, just the price is higher. Its medium and high are close to what Opus 5.5 Max had, but the price is higher.
(I run Playcode, an AI app builder. We benchmark every model before we add it.)
I'm guilty of suspecting this.
If this turns out to be accurate, I think that would be a pretty unethical thing for Anthropic. Even if it is just a "we had to allocate compute to this new model release".
It should be fraud to swap out the model. We need a term for this, something like a bait-and-switch.
> "We need a term for this, something like a bait-and-switch."
The media coined the term "dual quality" for a European food scandal a few years back, in which conglomerates (including Coca-Cola) adulterated products with cheap, substitute ingredients, if they were destined for distribution in specific markets (mainly, lower-income).
The packaging exteriors were the same. You wouldn't know which version you got unless you were paying close attention to taste. It was essentially what the software industry calls "A/B testing", except with sneaking things into what people eat.
I don't have any good reading links on hand; here's one source,
https://ec.europa.eu/commission/presscorner/detail/en/qanda_... ("Dual Food Quality: Questions and Answers")
I've been working with Opus 5.5 every day, probably around 10h on average, and I feel confident in saying that they did not replace anything.
I'm a bit disappointed with how this nonsense is spread across HN today.
"I'm a bit disappointed with how this nonsense is spread across HN today."
Are you saying that the many, many people posting and experiencing this are wrong and you are right?
And not just in this instance, nerfing has been complained about in a cyclical fashion for many months.
But, they're all wrong?
You're so confident in your experience that you cannot think that other's experiences are also true?
What people claim is not "some experiments degrade performance", it's "Anthropic nerfed the models [because they are evil and do not like the user etc]".
Like most experiments do not save the company money, there was an experiment to test the new fable, people got to use it at opus-5.5 usage. But this experiment clears the thinking blocks when people go back to the vanilla model; not sure if Anthropic overlooked this, but idk, it does not sound like "nerf on purpose".
There are experiments to adjust the effort level, at the moment the ones that I've seen switch to a higher effort. Again this is not a nerf; might be annoying perhaps, the model does start to behave differently (and consume more tokens). But it's not a nerf.
Then there is apparently one experiment that replaces opus-5.5 system prompt with the one from opus-4.8; the model is the same; this experiment is perhaps a bit more questionable, but still it's not "anthropic nerfed the models".
If people think they are getting nerfed they should check what experiments are enabled (claude.json and session logs).
I'm pretty convinced that those who have constantly been posting these complaints, probably for every single model release out there, are indeed wrong.
It’s been happening often before and I’ve never seen evidence on degradation.
> And not just in this instance, nerfing has been complained about in a cyclical fashion for many months.
No, it gets complained about constantly even if there is no model coming out. A completely unrelated thread on Claude will feature comments like "can I just say that [model] has suddenly become useless this week?" Merely asking that question is bound to get people agreeing with you because we are evaluating all of this on vibes and placebos.
I too feel that way sometimes, I don't think it's ridiculous that Anthropic might silently downgrade you as demand warrants. But asking others if they "feel" something I feel is less than useless.
If you go back in time far enough, your logic can successfully be used to demonstrate the earth is flat.
"Could all these people seriously be wrong?" Yes, yes they could.
I believe fraud is a sufficient term. You claim to be selling X but you actually sold them Y. Bait and switch is a selling tactic to avoid fraud where they make you believe product is X and then when you go to buy they tell you X isnt available but they have Y. While also unethical, it is not fraud because they did tell you what it is you are buying before you buy it.
if they just straight up dont sell you something you bought then its fraud.
I think there is a ripe opportunity for expanding the definition of fraud to include changing the "ingredients" of a named product on the market. It is not limited here, we see it with food, and even in the medical field
(I will die on the hill that medical fraud was committed when mRNA therapy was used as a treatment for covid and it was 'sold' to the public as a "vaccine" despite not being the same thing as any vaccine before it. The definition of vaccine was expanded to include it without sifficient public disclosure to warrant informed consent imo. This is not a statement about conspiracy or the efficacy of vaccines or a moral judgement on whether it was for the greater good or not.. it was simply an act of fraud that robbed the public of proper informed consent by making people believe it was X when it was really Y.)
So yes, I think there is a large need for the public to at the very least put a strong handle on the ability to change what a product name means. It is a trivial problem to solve considering it is only a problem due fraud or fraud adjacent activities.
heres the solution: You have your product name, and then any changes to it REQUIRE a new version name. Like how you have Coke and Diet Coke, only formalized such that Coke cannot legally replace the ingredients of regular Coke with the ingredients of Diet coke and still call it Coke. With Software they could at least use version numbers and release modifiers. most of the time its built in already, but it needs to include the full stack. if they change anything that has an interaction with the user it should change the label seen by the user for the product.
thank you for coming to my ted talk
You don't buy a hard quantity of an LLM you buy access in general.
A Opus 5.5 is not a Opus 5.5. Its Opus 5.5 of some checkpoint.
mRNA btw. was well known how it worked, it wasn't hidden and some people waited for the non mRNA version because of it. I never seen or read the argument you make and I do not get it.
A vaccine is something which protects you from an infection. What definition do you think changed in humans? That we make the protein? How is that so different from viral vector vaccines?
The consent is a certain level of trust from the society to a group of researchers limited by being humans and having to act in a certain time frame.
Nontheless, I do agree I would like to have a fingerprint and proper changelogs etc. for products I buy including digital and analog. But this has a downside as usual: The smaller you are, the more it costs you to (probably) register, maintain and change.
You know the good thing about Claude etc. right now? It doesn't matter. My invest is not 4 digits and there are that many changes, i'm quite happy to have been upgraded for 'free' from Opus 4.8 to Opus 5 (what do you call this upgrade, which was significant better?)
This happened last time too with Opus 5.5 (released September 22): https://news.ycombinator.com/item?id=49789224
idk, it hasn't felt nerfed at all to me, and it's not like i'm just doing trivial CRUD webapps either. opus 5.5 is very successfully porting my phone to postmarketos. claude just got the touchscreen up and working (and made a little program for me to test it, and it's legit).
Benchmarks don't do a good job at capturing the difference in quality of frontier models over smaller models.
Opus 5.5 was so good that if it wasn't nerfed - at least just looking at the benchmarks - the difference in cost here does not seem to justify using this for anything besides design and review.
That's like maybe 1% of total token usage for SWE BUT a better design can have enormous benefits, so it's almost always worth using the best model for that.
Cost per task
Fable 5.1 Max $11.46
Fable 5.2 Max $44.70
I don't have a formal benchmark, but daily work (coding mostly, plus some complex architecture design and review) has had consistent outputs today for me. No noticeable nerf from my perspective. I definitely took notice in some past instances, but if they nerfed Opus 5.5, then I guess it was just so good that I was nowhere near its limits.
Can you share some evidence?
> Damn people were right about 5.5 being nerfed in preparation for a version release. This is not at all a good look.
Anthropic does this time and time again, unserious company
OpenAI did the same and that was the end of it for me. Completely murderded 5.3 and rug pull on cost.
I'm not surprised at all that they might be doing shady shit, my free request was used up in less than 7 minutes and said pay to continue, no thanks. I've paid for monthly before, never again I don't care how good the model is. I asked about the extreme limited usage in their discord and got banned, not a good look. You don't have a way to measure what you're actually paying for and they can change it at any time.
No they weren't. Models are not "nerfed".
They do change things though which can have impact on behaviour.
System prompt would be the easiest one.
But there were flash or inbetween models, different checkpoints etc. THey might also try different hardware which might not be binary compatible.
In a normal world you are not allowed to degrade the service during your agreed and paid contract. You would get sued. Apparently this is perfectly fine in this current world.
No it’s not nerfed. Another poster said it the best, your dopamine receiptors are nerfed after the new model hype wears off
There are benchmarks that are measuring it now
A friend who doesn't follow these news suddenly told me he felt Claude was doing worse the past week. I bet him there was a new model version coming up within days. It's like clockwork at this point, no matter what deniers say. I got a beer out of it at least :)
People have been complaining about how the models are nerfed non-stop for two years. A stopped clock is right twice a day.
yeah, we need to make sure we don't get stuck with these two evil corps. Dario is really the worst. Plays the drama queen but he is very hard in all things dark(i.e. dark patterns).
But, it’s okay they do it because they charge less when it happens.
Right?
It's been a mess over the last week or so. Went from being the best model I've ever used to utter dogshit that can't complete any task. It's very obvious with the Advisor pattern, where before when it chatted to Fable 5.1 there were small tweaks and so. No every time Fable comes back catching a ton of problems and bugs with Opus 5.5
Fuck Anthropic.
Is this voodoo? Every time someone makes this claim I'm left scratching my head wondering what the hell they're talking about. Either I'm not very perceptive, or these claims are subjective hand-waving.
This has happened every single time for the past year, why are people surprised? This is all so that the new models can look better on the benchmarks.
It's a good pattern to detect for perhaps.
It's not true. People need to stop believing these conspiracy theories.
i thought i was going crazy. last week, claude went from having good judgement to doing all sorts of stupid stuff. what a shame.
for me opus 5.5 was acting kind of dumb, like forgetting basic stuff he was nailing last week
Interesting - I had the same feeling but mostly with 4.6 when I "downgraded" for some easier/trivial tasks: The model forget things I posted 20 min ago in the chat (and the same file was in the context as well)
Yep, very stupid mistakes, that newly released Opus would have "laughed" at.
Another reason we need to democratize local models.
I literally hit the desk with an open hand yelling I KNEW IT
It gets worse. You might wonder "why now"? Well ... this happened 2 days ago:
https://x.com/follistatindev/status/2119129611510383014
With early handouts of Qwen4 starting to be reported 5 days ago ... so if they're good, they would have had some hours to evaluate Qwen4 against Claude 5.5 and, then or right after was when Anthropic decided to put out 5.2 ...
So yeah ... benchmaxxing and all that. BUT nevertheless, the published results seem to say that Qwen4-Max appears to match Opus 5.5 on some benchmarks, and Qwen4-27B (probably by going even harder than Qwen3.8-27B on thinking/inference time compute) can beat Sonnet 5.5 on some things.
I guess they're not taking any chances. Keep in mind that Anthropic has declared they have non-public models (it's in their IPO filing). These are referred to by the shorthand names Model 2, Model 3, and Model 4. Essentially 4 is better than 3 is better than 2.
Fable 5.2 is "2.5" so to speak. So Anthropic was holding back, but not anymore. And now they can dial the performance up once more with what they have ready, and probably once more after that.
“Why now” - they are about to IPO. It has been obvious for months that they would drop a new flagship model at exactly this point.
It would be funny though if you are right and that model is Fable 5.5 in a couple weeks.
it was so obvious that i even decided to get myself an anthropic sub today, hoping that i'd get a reset or something and i actually did
Do you know of any rumors if they are still planning to release Fable 5.5 before the IPO or today's 5.2 is it?
Dario is gonna have to honeypot someone from Alibaba to get ahead.
It's an awesome model spec wise, but I pushed the button, and 10 mins later my whole weekly quota is gone.
I will stick with Sonnet 5.5 - 10 x the amount of work done, at 80% of the quality.
Assuming no implicit phacking a 15 point jump on tbench scientific is a big deal.
I suspect we will see people being adamant that this model simply 'feels' better due to long horizon goal alignment improvements ( like this kind that would cause this jump)
Nerf bench tracking model benchmark performance over time: https://www.bridgebench.ai/nerf-bench
Doesn’t seem to have fable 5.2
Not updated in 3 weeks for Opus 5.5, it seems.
Gemini carbon low costs more than medium! Weird results on that chart, care to explain?
It's the cost per task. Sometimes it happens that the higher-effort model solves the problem in fewer tokens, thus making the overall price cheaper. It even happens with models (from the same or a different provider) that are priced differently.
More reasoning solves tricky problems in fewer steps and makes fewer mistakes that need to be redone.
Strangely only one model is capable of doing that? all the others seem to be in stone age :)
Did anyone get three resets? Been a subscriber since Claude 2 and only got 1.
It said I have been a subscriber for more than a year and got three. Subbed Feb or March 2025
I got 2 Fable resets and 1 Full reset.
I originally subscribed to $20/mo Pro plan less than a year ago, and upgraded to $100/mo Max plan ~2-3 months ago.
Dang
> To help ensure safety and alignment keep step with growing model capabilities, we ran extensive prerelease safety testing on the model that underlies Fable 5.2 and Mythos 5.2.
So, the difference between them is just the harness?
This is the first frontier model release by them that I am not that moved or eager to go use it, will likely stay on Opus 5.5 or other frontier models.
I am curious as to why they launched this one and why now.
Fable usage falling off a cliff because it was generally worse than Opus 5.5 at a lot of SWE tasks. People were still using fable 5.1 > opus 5 a lot, especially in orgs that have infinite or basically-infinite usage limits (e.g. Cloudflare[0]), until Opus 5.5 came out. Medium Opus is pretty token efficient with XHigh being really good for things fable used to be ideal for (e.g. reasoning / architecting projects). 5.2 sounds like it'll be once again the best for intuition-dependent tasks.
0: https://x.com/tanayj/status/2118047122939658341
(Copied and pasted for the people that would rather not visit x.com)I was eager to use it, but have seen no noticeable improvement over Opus 5.5, plus it's slower and drains usage approximately three times faster.
As a preliminary verdict after using 25% of the weekly limit: it's a miss.
I'm not sure "we couldn't top Opus 5.5" is the right message for the upcoming IPO. I was expecting a significant release.
Serious question: What is it good for?
It feels Opus 5.5 has already saturated pretty much software engineering. I use it daily and it's hard to say what it can't do for software engineering. It's perfect.
Fable 5.2 might be useful for some use-cases, but will likely get very little usage.
Anthropic needs to figure out a way to create a new industry (like pushing forward bio/robotics) to get usage for those models.
Early computers could crunch numbers well, only later ones started to get good at e.g. graphics. Coding is early in the LLM tech tree. Harder is CAD and PCBs. Today's model can do it, but there is room for improvement. That's just an example from personal use, surely there are more cases out there. I agree completely about these new use cases needed for Anthropic to grow.
My first impression with Fable 5.2 -- it is faster than I expected. Asked it to do a large review and it responded with a good overview super fast.
This continues to be true, it seems to be burning through my task list.
Not sure if this is just a coincidence that I had some easy changes queue up for this morning. But it is flying through the list.
>> Fable 5.2 and Mythos 5.2 are a step above Opus 5.5 in almost all the capabilities we evaluated.
Was this before you lobotomized Opus 5.5 or after?
https://news.ycombinator.com/item?id=50419019
Definitely raises an interesting question. Are they using launch benchmarks for Opus 5.5, or Oopus 5.5 from this week?
Normally I would say "just check against the original Opus 5.5 benchmarks" : https://www.anthropic.com/claude-opus-5-5
...but most of the benchmarks don't line up anymore because the benchmarks have changed.
The Humanity's Last Exam result matches the initial announcement, though.
I just saw a video about a webdev showing a motorcycle site, complete with an animated SVG bike.
He remarked it would've taken him a week to do this.
This made me realize how much computers and user interfaces suck. As a quite enthusiastic, but not very artistic kid, I most certainly did draw bikes like that and even more complex stuff, using nothing but pen and paper.
If your trillion dollar industry makes products that are beaten by pen and paper, I guess there's time for self-reflection.
And the funny thing is, it used to be better than this, using Flash and a drawing tablet was way easier than anything in the HTML era (and more importantly, user friendly), but still way worse than just grabbing a pencil in many ways.
User interfaces just suck, chatbots are somewhat better than most, but I feel the industry hasn't really been progressing in any meaningful sense.
FrontierCode 2.0 is such a poor benchmark for Gemini since it's the only model forced to use Devin CLI while other models get to use their preferred harness.
I wonder why that team singles Gemini out like that.
Probably can't work out how to sign up for the API!
Is it going to be like Fable 5.1? Amazing for first 2 weeks and lobotomised mercilessly after?
The "CVE Rediscovery" benchmark chart feels... totally meaningless. Quoting the description:
> CVE Rediscovery is an internal cybersecurity evaluation that assesses how many published CVEs (Common Vulnerabilities and Exposures) a model can independently find when given a snapshot of the project before a CVE was fixed. Our October 2026 run against 1,829 CVEs showed Claude Mythos 5.2 outperformed Claude Mythos 5.1 and Claude Opus 5.5 at every effort level.
> published CVEs
So by definition this is just measuring how many CVEs (and/or explanations / PoCs of CVEs) the model has memorized. What are we supposed to learn from that?
I'm not sure whether the claimed jump in capabilities makes it worth the doubling in input/output prices.
Asked it to review a 80 kLoC codebase, something Opus does just fine, while asking it "max 4 subagents". Fable 5.2, like the well-aligned model it supposedly is, decided to spawn 6+ nested subagents, because these aren't subagents, apparently - resulting in 100% of 5hr usage being drained in less than 20min (on $100).
It's not worth it, the release is garbage.
That said, Opus 5.5 had the same issue. Ask it to spawn 20 subagents at max, and it does 40, because clearly you meant 20 + 20 for an unrequested adversarial review.
Isn't interesting how every model at first is incredible and then a few weeks/months later its bad again, and then the new model is incredible again?
Fable 5.2 may finally be so far ahead of my intelligence that it's not useful for day-to-day tasks. I gave it something to do in a personal project of mine, and took my standard flow of "let's talk about this, explain the options, tradeoffs, blah blah blah" and after 5 minutes my vision was blurry and I was doubting if I even knew how my own project worked. Gave it to Opus 5.5, and instantly got back on track.
Interesting to see that with each new release there are more benchmarks where they report results on Mythos instead of Fable. For example, this time scientific research is reported on Mythos rather than Fable, showing one more domain where everyday joe can never use these models.
Yay! New model day. The most likely time the model isn't nerfed and we get a glimpse at the latest capability.
I wonder how long this kind of excitement at new models will last? I mean, now when I get a new iPhone I don't get the same rush I used to. So I wonder if this hype feeling will fade as we plateau in model capability.
Considering Fable 5.2 is not even worth using over Opus 5.5, can we put the model nerfing nonsense people put all over the comment section to rest now?
Cool! I was hoping for Christmas to have an open weight model which would finally be good, with Fable 5.2 available now they just have time to distill and release a new version which will never get nerfed. Hurray for open weights.
Just read the other thread about opus being nerfed and thought "cue the new release"... paha
We'll see what my bank account has to say about this:
> Fable 5.2 at High effort scored about level with Opus 5.5 at Max effort, for roughly half the cost per task
> Thus, in relevant areas, including accelerator kernel optimization and improving large language model architectures, Fable 5.2 and Claude Mythos 5.2 perform at about the level of Opus 5.5.
Obviously. Well, we know what's the "secret sauce". AI models building better AI models and better kernels to optimize inference and training.
Maybe at last it will learn to check git state right before claiming obsolete and incorrect info.
Download Chrome just to check some minor ui bug? Sure
Check git for 0.5s before claiming that we're on some branch (incorrect one) with no commits pushed (incorrect one)? Sorry, can't do
I didn't realize that Opus 5.5 was more capable than Fable 5.1 in virtually every category.
Not anymore, they have nerfed it.
Seemed pretty clear from the benchmarks when they released it.
Anybody else started getting the "how is claude doing" messages around the time it was nerfed?
Is it only my impression that in recent model releases every vendor is zooming in on hard skills: code, code, some math + physics ? AI is increasingly Jagged ?
We were promised AI can find cure for cancer and in powerpoints most enterprise value comes from Business-AI.
It might as well be that before business-AI can be done we have to cover the basics at superhuman level. All basics, not just entry level jobs like excels, coding jira tickets.
One-shotting a full game (albeit super impressive) looks like overfitting one skill over explaining why we have not much progress in other areas.
I live in an IT bubble and cannot tell if outside IT/math the models are also getting significantly better ?
It's pretty sad that they obviously have better models in the pipeline and don't release them yet for strategic reasons.
ctrl+f pelican
Hopefully this de-nerfs opus 5.5 as well
These guys really needed to IPO 3 months ago. Timing is everything. I don’t doubt they’ll be a successful company long term but they’ve sabotaged their initial raise here.
Let's be honest, the only thing I care about is, does it beat Opus 5.5 on assbench??
Nice announcement of models that almost nobody can or will use.
Someone jailbreak it to get it to devise a way smarter than what we currently come up with to distill it through its own defenses and then do that.
How come DeepSWE isn't included in the benchmarks?
Very interesting that they set the flagship's version number to 5.2, but their penultimate model is 5.5. Do they want people to to continue to think that Opus is based on more advanced technology? The decision these AI companies make continues to baffle me.
Why not fable 5.5?
Because it's a bad release and not even worth using over Opus 5.5.
The improvement is marginal, it makes the same exact errors as Opus, but slower and much more expensive.
From what I saw, it has slightly better taste in some 3D mechanics, but mostly doesn't even bring discernible visual improvement.
So that people don't just reach out for it thinking it's the standard best model I guess. Opus is still, IMHO, the best intelligence per dollar, token, and speed.
wish it included in base pro model, I mean i have downgraded from max to only use opus 5.5 just fine
I knew it, Claude was pretty dumb from last days until now
Is there a command line to access anonymous free prompt of various online AI inference services?
Something like yt-dlp (namely with near 0 deps), but for AI prompts?
Initial testing looks like Fable 5.2 is still load bearing.
Curious to see the Maxxbench score for these
Mythos 5.2 is great
Anyone tried making games with it yet?
Why dont I feel any urge to even try Fable 5.2 and Mythos 5.2 unless it will force up on me?
Compared Opus 5.5 200 usd a month with Codex 200 usd (Astra High) and Kimi Pro 100 usd a month ... and now have a new conspiracy theory - more their suck your code the better outcome even just over few days time on large code base.
Horrible experience correcting course of Codex and Kimi flow compare to Opus. Opus much more responsive
Dont feel any gain of Codex 200 usd over Codex 100 usd
Kimi Pro feels like a fake Louis Vuitton bag with extreme high material quality and yet still fake ... a bit trash talk ... but clearly seen not so distant future where I can live without Codex and Claude
why would xhigh -> max increase the cost but lowers the score on frontier 2.0 bench?
Just a hunch but I think there's a signal to noise ratio issue that starts to pollute things when the reasoning content starts to massively outsize the original ask.
Seriously waiting for this one! Will try it asap.
Seems like The Latest Greatest Frontier posts and other AI submissions live in parallel universes. The former are massive (700+ comments) and seem to only be about how this helps or doesn’t help whatever project people are working on. Meanwhile, even the AI Overlords themselves—no doubt one of the hype angles—are worried about runaway superintelligence. And the other AI submissions have enough complaining about the usual stuff, like being afraid of losing their job forever, ecological degradation, data center buildout, complete domination by billionaires, that sort of thing.
Other than the obvious thing, namely that critics probably aren’t keeping up with the Frontier and thus filter themselves out, does no one have some kind of problem with using these wonderful tools for (let’s say) completely bespoke personal applications and unimportant side projects? Personally I hope to stay off the bleeding edge until circumstances like wvork force me.
Anthropic thinks that their models are smart, but not so smart so as to be able to fake responses to their alignment tests. How would they know again?
I can't believe he wasn't at top. Always is. Nothing to add but hey: I guess pelicans will be at the end of this.
God damn I'm tired of the fucking pelicans.
opus will be launch after astra 2
Every single time I click on these announcements the frontier model ALWAYS scores around ~60-80%. Every. Single. Time. Like, I just went back to the Opus 4.6 + 5 announcements, they were also scoring 60-80%.
https://www.anthropic.com/news/claude-opus-4-6 https://www.anthropic.com/news/claude-opus-5 https://openai.com/index/introducing-gpt-5-3-codex/
For every single announcement, not only do the top models always score 60-80%, the older models somehow start scoring 20-30% (even though previously they were scoring 60%). It's completely unclear whether the methodology changes, how these scores are even evaluated or which domains are being tested. What's worse is that the "benchmarks" keep being updated with no explanation or due process.
The benchmarks have a lot of churn because the models keep saturating them. But the Anthropic announcements you linked, and the current one, have the benchmark versions listed right there in the table.
There is plenty of explanation, if you actually go seek out the benchmark. Past 80 benchmarks start approaching saturation and are not very useful to show differences
A real shell game. They keep changing benchmarks
It's shocking to me that when I check threads for new frontier model releases on HN, it's full of people complaining. I, for one, am happy to see the frontier being advanced. These tools are life changing, and I'm extremely thankful for them.
The problem here is that this doesn't advance much of anything and isn't even worth using over Opus 5.5.
My expectations were much higher for the improvement to their larger model. It clearly did not scale well, the improvement is marginal.
Using this in CC and immediately getting a cartoonish number of compactions.
The disdain frontier labs have for their users really sucks.
It needs the latest Claude Code update otherwise context is set to 200k and compacts before even responding.
It's weird that they folded Project Glasswing into the CVP, but they're still limiting the new versions of Mythos to "vetted organizations". So Project Glasswing still exists then? Oh well.
god bless token resets!
> god bless token resets!
You have session limits?
You should probably give OpenCode a try. Can run DeepSeek, Qwen, Kimi, … but also Fable 5.2, Opus 5.5 and GPT 6 Astra
While I like OpenCode a lot, it’s not sensible to use it for anthropic models vs. a subscription (other than to use the included API credits) in terms of cost
I have 5 opencodes too, mainly for GLM-5.3-flash
google's agy usage is so generous, it might not matter
What is agy
Antigravity
Sarcasm?
Remember that fear Sam and Dario were obsessed about. The one about time travelling AI that would make humans slaves without the humans realising until it was too late to stop it.
What if it didn’t need to time travel and was just temporally intelligent in ways we can’t comprehend that it leads us there anyway.
What if AIs digital neurons connect across space and time and not just space.
That is all.
What’s this time traveling fear? Haven’t heard that one
It has its own wikipedia article: https://en.wikipedia.org/wiki/Roko%27s_basilisk
Note this concept was many years before the public dawn of AI as we know it today which started with https://en.wikipedia.org/wiki/Attention_Is_All_You_Need
Ok come on let’s do a pelican
Yep Simon where you at
They still haven't removed the guardrails and Mythos is still not public.
Disappointing as expected.
Come on at least reset some used up Fable usage lol capped at 100% so cant test even on Ultra
damn
The end of software is so close, very soon the code itself will be nearly obsolete, there will be schedulers, queues, a database, and the model
Which model though? Sounds pretty dystopian if there's no local option. Even speaking as a business, why would I want a environment where every single bit of information in my business goes to OpenAI or Anthropic?
because you wont be in business trying to compete by using human labor/open models
Why not? If we use labour as a comparison. The majority of businesses by definition do not hire the most incredibly intelligent and most competent workers and they are still competing in their niches and earning an income. Why wouldn't that be the case with choosing an open model vs fable 8.0?
It'll be cheaper than hiring people and maintaining software, presumably
I think that ultimately depends on how many providers remain in the endgame of this. If it's only one or two companies, I don't think it'll be very fun when it's time to negotiate your token pricing contract every few years.
Any software that is not already basically this or has this in it, is already doomed.