Wait a minute - clearly I missed something here. Last I read, Mythos was only available to a handpicked list of megacorps under project glasswing. Did the hourly changing AI soap opera air yet another plot twist that I missed amidst my quest better known as “trying to find a job”?
If not, how’d a small time outfit get access to something the rest of us can’t have because we’re (apparently) not trustworthy enough?
No shade on these guys - I’m thinking it’s just another plot twist in “Hours of AI’s Lives”.
They list Anthropic as one of their customers and appear to have conducted penetration testing for them previously, so they're likely one of those trusted organizations.
Well, this was fun read. Discovering such a high-ranking critical exploit within a week by coupling experts with frontier models is an amazing new journey we're about to embark on.
Is it? It's an arms race between the "good guys / defenders" and "bad guys / attackers". Assuming both sides have access to the same tools, how is this going to make any difference? Their relative strength will stay the same.
What is actually different is that 1. anybody without tool access is out of the game, which includes security professionals from poorer backgrounds (for them it's not too amazing of a journey) and 2. the AI vendors get a constant stream of what's essentially an AI tax from everybody - so yeah, for them it's gonna be an amazing ride.
Meh. A world where the defenders and attackers are both omniscient is a world without exploits. Steps towards that world are steps towards more security. The reason the exploits are being found so rapidly is that they are mining all the bugs left in these projects from decades of coding. Eventually they will run out.
Pretty soon people will just airgap their stuff and that will eliminate most of the attack surface short of a sort of Mission Impossible style operation.
I mean really, given how AI is being marketed, what is the point of the internet going forward when all its contents are going to be ai slop anyhow? Just disconnect and run local models if all you are getting is slop anyhow. The original purpose of the internet is now dead. Real people communicating and sharing information with eachother? Ha! That is no longer valued.
In fact actual innovation might no longer be valued anymore. What is innovation but opening Pandora's box and potentially seeing a disruptive competitor take your slice of the federal reserves money print? Better to nip that in the bud and control all the devils we already know, from a pure sociopathic profitmaking standpoint, which seems to be a very dominant viewpoint among the people in charge of the worlds power structures right now.
So like ... I thought Mythos was just a bunch of hype? Or maybe the researchers are having their skills boosted due to using a model with such a cool name?
I jest, but I did notice having more confidence to take on more ambitious work lately. We're all centaurs now.
My opinion is that it is over-hyped because like any LLM, it requires a suitable human in the loop to keep the LLM on the straight and narrow, and then to weed through the inevitable false-positives and hallucinations.
Nicholas Carlini, for example, whose name is on many of the recent high-profile Mythos findings is not just some random dude with a Claude sub on his credit card .... he's an experienced security researcher.
Random inexperienced people thinking Mythos can replace the need for experienced pen-testers, auditors etc. are likely to be sorely disappointed if/when they get their hands on Mythos.
> Nicholas Carlini, for example, whose name is on many of the recent high-profile Mythos findings is not just some random dude with a Claude sub on his credit card .... he's an experienced security researcher.
I don’t think Mythos is hype for all kinds of reasons.
Anthropic is a young company but their track record is solid; they don’t seem to hype things just for the sake of hyping things. Sam Altman at OpenAI? We already know his track record…
I’m going Occam’s razor here: the simplest explanation is usually the correct one.
Anthropic had an “oh shit” moment when they realized what Mythos can do. They decided to do the responsible thing: give the industry a heads-up and an opportunity to use the preview to identify and fix the most dangerous zero-day vulnerabilities.
Since the FAANG companies have billions of users, it makes sense to start with them.
There’s still going to major issues for users of systems too old to get patches or updates. Or for IT organizations who think Mythos is a replay of Y2K, where, compared to the warnings, not lot happened.
The bottom line is someone with Mythos won’t need to be an experienced security expert to cause real problems. That’s kind of the point.
Over time that will change. Technology has proven time and time again that as we add a new layer of abstraction over the fundamental functionality, knowing the previous layer quickly becomes vestigial knowledge. It is true not just in software but absolutely all technology there is, going back to the first fire made or atl atl or rock sling.
> it is over-hyped because like any LLM, it requires a suitable human in the loop to keep the LLM on the straight and narrow, and then to weed through the inevitable false-positives and hallucinations.
"Suitable human" is a dry phrase indeed. ^_^
The hype is "gosh look at all the bad things this brilliant almost conscious tool found!"
The reality: an insecure toolchain for an insecure language with an insecure compiler produced a runnable but insecure binary for an insecure OS. We couldn't be arsed to address any of this before, but now we're being billed the full price of our laziness.
> likely to be sorely disappointed if/when they get their hands on Mythos.
At first they will be delighted. So much money and time saved. When their adversaries get their hands on their system (with or without Mythos), then they'll be sorely disappointed.
Anthropic just doesn't (didn't) have compute, so they make up a bunch of PR to delay it
They got all that compute with the SpaceX partnership but now the PR has taken a life of its own, so might as well keep hyping up Mythos and artificial scarcity if they have an asset people want now
Just roll with it
Open AI has a history of doing the same thing and it's the same people. GPT 5 was supposed to be AGI at one point, remember.
I wouldn’t know whether that’s true, but this is partly hardware. That tends to make things more expensive, as, at some point, design bugs get really expensive both in time and in dollars, and chances are there were design bugs late in the process (e.g. because somebody published a new way of attacking a system that the latest design didn’t anticipate)
Also, Apple claims it was an effort spanning half a decade (https://security.apple.com/blog/memory-integrity-enforcement...), so depending on what you consider part of this (for example, do you include time spent on their secure memory allocator, on designing/implementing the ARM Memory Tagging Extension or Extended Memory Tagging Extension in the costs of this feature?),
from what they demonstrated, this seems to only be a $100,000 exploit in Apple's bug bounty platform, but if they package it right, it could be a $1.5 million exploit
They simply have to show it against a beta version of MacOS, and frame it as unauthorized access, and maybe from locked mode if possible
They don’t seem to state lpe as one of the bugs. Maybe 100k? There’s alot of factors that go into it so I’m really not able to say. I could see it going for lots more or lots less
The world is so not ready for the impact of LLMs on security issues. If true, congrats to the Calif team. It’s likely too technical for me to understand in details but looking forward to reading the 55 pages report
> The world is so not ready for the impact of LLMs on security issues.
I agree, but it's the people I'm worried about.
I'm hearing anecdotes from all over about devs pushing LLM-generated code changes into production without retaining any knowledge of what it is they're pushing. The changes compound, their understanding of the codebase diminishes, and so the actions become risker.
What's worse is a lot of this behavior is being driven by leaders, whether directly (e.g. unrealistic velocity goals, promoting people based on hand-wavy "use AI" initiatives, etc) or indirectly (e.g. layoffs overloading remaining devs, putting inexperienced devs in senior rolls, etc).
The world's gone mad and large swaths of the industry seem hellbent on rediscovering the security basics the hard way.
The gamble is that you can cruise on the senior engineer’s diminishing understanding for a few years until models become good enough that you don’t need any humans in the loop and you can fire all those expensive seniors.
The tragedy is having a bunch of those senior engineers writing blog posts and what not of how productive they are, without realising that it means business now needs less of them.
I suppose that if you don’t believe that models will be good enough to work completely without senior engineer help, positioning yourself as a master prompter is a good move to improve your chances of not getting fired.
If all you have are being good at prompting you are gone. Business is going to prefer new grads who have taken some class in ai prompting (which many schools now offer). Doesn't matter if the class in ai prompting isn't any good. What matters is the idea that they had a formal training in this thing, and that they are willing to work for far less pay than any senior.
>I'm hearing anecdotes from all over about devs pushing LLM-generated code changes into production without retaining any knowledge of what it is they're pushing. The changes compound, their understanding of the codebase diminishes, and so the actions become risker.
The difference is twofold. First, junior devs who ask for code reviews on massive, 2000+ line diffs get coached, and eventually fired if they persist at it. And second, even the most prolific junior engineer would take years to write what Claude is capable of generating in an afternoon.
When Sundar Pichai announces that 75% of all new code at Google is AI-generated, their stock price goes up. If he were to announce that 75% of all new code at Google is now written by junior engineers, this would trigger a massive sell-off and a lot of employees would resign.
The dangers of technical debt and the importance of mitigating it have been known for a long time. Unfortunately a lot of entities now ignore all experience and best practices as soon as you say the "AI" buzzword.
> I'm hearing anecdotes from all over about devs pushing LLM-generated code changes into production without retaining any knowledge of what it is they're pushing. The changes compound, their understanding of the codebase diminishes, and so the actions become riskier.
I don’t think so.
An LLM can produce higher-quality documentation than most humans. If it's not already happening, when a new developer joins a team, they're going to have an LLM produce any documentation a new developer needs, including why certain decisions were made.
It could also summarize years of email threads and code reviews that, let's face it, a new person wouldn’t be able to ingest anyway; it's not like a new developer gets to take a week off to get caught up on everything that happened before they got there. English not their first language? Well, the LLM can present the information in virtually any language required.
As the models continue to improve, they'll spot patterns in the code that a human wouldn’t be able to see.
> An LLM can produce higher-quality documentation than most humans.
Can bears some heavy weight.
LLM generated documentation has so low level of information density, that it’s useless. Yes, it writes nice sentences… or even writes. But it contains so much noise that currently, reading code is a better documentation than what I’ve seen from every single LLM generated documentation.
The same with LLM generated articles. I close them after the second sentence because at least about 90% of it is useless filler.
I almost closed it when I read the first few sentences because these kinds of articles are useless time wasting nonsenses. But this was different. This was old. Most sentences contained something new. Something worthy. (Of course, people also write unnecessary long articles… looking at you Atlantic)
You can throw out almost everything by volume from LLM generated documentation without loosing any information.
Currently, if I smell (and it’s very easy to smell) LLM generated documentation or article, then I close it immediately, because it’s good for only one thing: wasting my time, for no good reason.
> LLM generated documentation has so low level of information density, that it’s useless. Yes, it writes nice sentences… or even writes. But it contains so much noise that currently, reading code is a better documentation than what I’ve seen from every single LLM generated documentation.
I should clarify: the documentation I’m talking about is not generated using a generic LLM prompt, which would mostly suck.
With the proper context and additions (skills, plugins, MCPs) LLMs can produce high-quality documentation. You'd also have subagents doing QA of the documentation.
If stuff really goes wrong, you need people who deeply understand the codebase so that they know where to look and how to diagnose the issue. It might be the case in the future that LLMs become so powerful they'll diagnose any issue (I doubt it), but until then, we need people in the loop.
That is actually unfair. Most companys spend enormous amounts on security with vast armys of security employees. Not that it is effective, but it is not for lack of resources or trying.
I mean we are literally in a thread about how the 4 trillion dollar company, literally the 3rd most valuable company in the world, with a core competency in software has, yet again, released a core product riddled with security defects for the 50th year in a row.
Commercial IT security is a industry that is incapable to a fault and has, so far, faced basically zero consequences for it.
For every Apple, there are 100 mom-and-pop companies who have nothing.
Even more so in the future when a software company can be launched by a farm of AI Agents with a founder at helm with no clue about computing or security.
What's debateable is how many of those companies actually need irontight security, because they are never realistically going to be targets of criminals and/or they have nothing valuable to steal/corrupt in the first place (other than the owner's pride).
I was pointing out how even Apple, a entity who by all rights should have top-notch security, is still absolutely hopeless in the face of commonplace commercial, profit-motivated attackers.
Massive, extremely well-resourced divisions supported by management in a technically competent organization that is actually trying to solve the problem struggle to produce at best middling security that is inadequate against commonplace threats. This is not a prioritization problem; even if you do “everything right” you are still vulnerable to run-of-the-mill commercial attackers. This is a fundamental capability problem, like how we can not make a net positive fusion reactor right now.
It is actually unfair to blame these companies for not having a fusion reactor because they “were not trying hard enough”. Actual security is not a easy problem, and it is a great disservice to portray it as one that is only unsolved due to dunderheads being in charge since it leads to underestimating what actually needs to be done.
That is not to say that you can not do dramatically worse than the “gold standard” and also that most organizations are actually incompetent; but the “gold standard” is still objectively grossly inadequate. You need to be dramatically better than the 4 trillion dollar software company to reach adequate against prevailing threats.
They have a website that can be used to host malware and/or seo link farms.
I still have nightmares about the contact form on my low-stakes personal website getting hijacked to use as a spam sender (because I used unsanitized input in mail headers).
Hey now, when Apple products get a serious Kernel level vulnerability that is able to be executed just by browsing a website. It's a "jailbreak" not an "exploit".
While maybe true, it is better to back that up with data and the data I know of and read yearly is mostly not great. Between Splunk and SANS surveys of 2025 maybe ~2000 companies have a SOC. [1] [2]
Then you have the many companies in the UK, US, Canada, EU that have compliance and regulatory laws that require them to exist in some capacity in house. Though that is changing with MDR services, but someone still has to interface with the MDR.
Not at all. I’m considering that the amount of vulnerable software in the wild is very, very large, with most organizations not managing their systems properly. Imagine all the small to medium size companies that do not have budgets for a dedicated, talented security team. And all the software that will never be patched. We are at the beginning of the exponential
> I’m considering that the amount of vulnerable software in the wild is very, very large
I'd imagine this set is very similar to just "the set of software on the world". Even before the AI stuff, it was a pretty good bet at any given software had some vulnerability; it was just a question of how easy to was to find it.
Yes, that’s my point. Look at how fast the Calif team tackled that macOS issue. Against the top company in the world. One week from bug to exploit. In 2-5 years things will be really wild for everybody out there. We released a technology that make it possible to design extremely complex exploits at a scale we never had to face before. What does that mean if you’re not the top company? Things will be really bad
It makes you think will everything need to be rewritten from the ground up - potentially by AI itself, or AI having a very heavy hand in validating all of it.
There's so much much lower hanging fruit. Every job I've had has had basically everything massively out of date. Just keeping packages and framework versions up to date is a full time job and none of these companies have someone assigned to doing it.
So much out of date software with known exploits left running for years. The only reason there hasn't been total disaster is no one has tried to hack it yet.
Yes, exactly, that’s the main change. And not just in a script kiddy way. What we see now is LLM + experts can develop extremely complex exploit chains in no time. It’s one thing to exploit a known vulnerability that you can patch by upgrading your Wordpress, it’s something else when the attacker is able to completely take over your systems in ways you didn’t even consider was possible and adapt in 1 day to your attempts at patching
Arm published the Memory Tagging Extension (MTE) specification in 2019 as a tool for hardware to help find memory corruption bugs. MTE is a memory tagging and tag-checking system, where every memory allocation is tagged with a secret. The hardware guarantees that later requests to access memory are granted only if the request contains the correct secret. If the secrets don’t match, the app crashes, and the event is logged. This allows developers to identify memory corruption bugs immediately as they occur.
Hm that leaves more questions for me. Why does this path not have bounds checking, is think perhaps a limit of the clang flag or is it more simply a mistake of omission on apples part. Either way it seems like a bad look. I wish we’d get a post mortem
Well it’s memory corruption so I think it’s pretty safe to assume it’s a bounds issue. I’m not sure if it’s possible to get this with something like type confusion tho I could be wrong here.
IIRC, the GPU is behind a memory controller, so I doubt corrupting GPU memory alone could lead to an LPE. But I suppose it would give you someplace to store stuff if you can make something else read from it.
I had the same question and if this is a data-only attack, the lesson may be that MIE reduces many attack paths but does not remove every useful corruption primitive
LLMs are going to produce amazing Rube Goldberg style vulnerabilities for years to come. It's already starting, this instance isn't the case, but it's happening.
Maybe it's physically impossible to build a theoretically secure system, just as it's (presumably) impossible to have a cell that isn't susceptible to any virus. Maybe this whole time we've been getting away with a type of security by obscurity, where the obscurity is just no one having the time and focus to actually analyze the code.
1. Any given system has a finite number of findable vulnerabilities.
2. All findable vulnerabilities are fixable (if not in software then with a new hardware revision).
3. Fixing a vulnerability while keeping the same intended functionality introduces on average less than 1 other findable vulnerability.
4. It is possible to cease adding new features to a system and from that point forward only focus on fixing vulnerabilities.
If all 4 are true, then perfect security seems possible, in some sense. I think some vulnerabilities might not be fixable, if you include things like the idea that users can be tricked into revealing their passwords. If you restrict the definition of vulnerability to some narrower meaning that still captures most of what people mean when they say computer vulnerability, then I think those 4 statements are probably true.
Perfect security might be near impossible in practice because vulnerabilities will get more difficult to find and fix over time, but I think we should expect the discovery of vulnerabilities to eventually become arbitrarily slow in a hypothetical system that prioritized security above all else.
I would rather claim that building a theoretically secure system is prohibitively expensive. At the end of the day, Mythos et al. are just better tools for finding vulnerabilities that will eventually be available to both offensive and defensive actors.
If you imagine you had a vulnerability scanner as fast and convenient as a linter, it would be much cheaper to write secure code right away. Probably not perfectly secure, but still secure enough to make sure finding exploits stays expensive.
I would find it funny if one day we found it irresponsable to write hand generated production code. Just like it would be irresponsable to build a significan building without running numerical simulations.
it's probably less about how you write the code to begin with and more about letting a tool hammer on it
if you want to be a one man show handcrafting an artisan iOS app that will be fine, but you should probably let Claude bang against it for a while to shake out whatever bugs
They certainly are, one of the reasons behind Embedded Swift is to replace iBoot firmware currently written in a C dialect similar in ideas to Fil-C, with something better.
However it is no different from the Linux kernel, just because Rust is now allowed, the world hasn't been rewriten, and no sane person is going to do a Claude rewrite of the kernel.
Swift is definitely being used at apple. Most recently added as a CSS parser in safari and running embedded in some of the secure enclave parts. I know there was talk from as far back as strangeloop to get it in the kernel but I'm not sure how far that has gone. That being said they've been huge proponents of fbounds check in clang which can achieve a small portion (but important!) of what memory safe languages can do. I'd also like to see more swift or alternative adoptions I think they have potential and more competition in the safe language space is always welcome.
Wait a minute - clearly I missed something here. Last I read, Mythos was only available to a handpicked list of megacorps under project glasswing. Did the hourly changing AI soap opera air yet another plot twist that I missed amidst my quest better known as “trying to find a job”?
If not, how’d a small time outfit get access to something the rest of us can’t have because we’re (apparently) not trustworthy enough?
No shade on these guys - I’m thinking it’s just another plot twist in “Hours of AI’s Lives”.
They list Anthropic as one of their customers and appear to have conducted penetration testing for them previously, so they're likely one of those trusted organizations.
I'm guessing they have some friends at Anthropic.
Well, this was fun read. Discovering such a high-ranking critical exploit within a week by coupling experts with frontier models is an amazing new journey we're about to embark on.
> amazing new journey we're about to embark on.
Is it? It's an arms race between the "good guys / defenders" and "bad guys / attackers". Assuming both sides have access to the same tools, how is this going to make any difference? Their relative strength will stay the same.
What is actually different is that 1. anybody without tool access is out of the game, which includes security professionals from poorer backgrounds (for them it's not too amazing of a journey) and 2. the AI vendors get a constant stream of what's essentially an AI tax from everybody - so yeah, for them it's gonna be an amazing ride.
Meh. A world where the defenders and attackers are both omniscient is a world without exploits. Steps towards that world are steps towards more security. The reason the exploits are being found so rapidly is that they are mining all the bugs left in these projects from decades of coding. Eventually they will run out.
Pretty soon people will just airgap their stuff and that will eliminate most of the attack surface short of a sort of Mission Impossible style operation.
I mean really, given how AI is being marketed, what is the point of the internet going forward when all its contents are going to be ai slop anyhow? Just disconnect and run local models if all you are getting is slop anyhow. The original purpose of the internet is now dead. Real people communicating and sharing information with eachother? Ha! That is no longer valued.
In fact actual innovation might no longer be valued anymore. What is innovation but opening Pandora's box and potentially seeing a disruptive competitor take your slice of the federal reserves money print? Better to nip that in the bud and control all the devils we already know, from a pure sociopathic profitmaking standpoint, which seems to be a very dominant viewpoint among the people in charge of the worlds power structures right now.
So like ... I thought Mythos was just a bunch of hype? Or maybe the researchers are having their skills boosted due to using a model with such a cool name?
I jest, but I did notice having more confidence to take on more ambitious work lately. We're all centaurs now.
> I thought Mythos was just a bunch of hype?
My opinion is that it is over-hyped because like any LLM, it requires a suitable human in the loop to keep the LLM on the straight and narrow, and then to weed through the inevitable false-positives and hallucinations.
Nicholas Carlini, for example, whose name is on many of the recent high-profile Mythos findings is not just some random dude with a Claude sub on his credit card .... he's an experienced security researcher.
Random inexperienced people thinking Mythos can replace the need for experienced pen-testers, auditors etc. are likely to be sorely disappointed if/when they get their hands on Mythos.
> Nicholas Carlini, for example, whose name is on many of the recent high-profile Mythos findings is not just some random dude with a Claude sub on his credit card .... he's an experienced security researcher.
I don’t think Mythos is hype for all kinds of reasons.
Anthropic is a young company but their track record is solid; they don’t seem to hype things just for the sake of hyping things. Sam Altman at OpenAI? We already know his track record…
I’m going Occam’s razor here: the simplest explanation is usually the correct one.
Anthropic had an “oh shit” moment when they realized what Mythos can do. They decided to do the responsible thing: give the industry a heads-up and an opportunity to use the preview to identify and fix the most dangerous zero-day vulnerabilities.
Since the FAANG companies have billions of users, it makes sense to start with them.
There’s still going to major issues for users of systems too old to get patches or updates. Or for IT organizations who think Mythos is a replay of Y2K, where, compared to the warnings, not lot happened.
The bottom line is someone with Mythos won’t need to be an experienced security expert to cause real problems. That’s kind of the point.
Over time that will change. Technology has proven time and time again that as we add a new layer of abstraction over the fundamental functionality, knowing the previous layer quickly becomes vestigial knowledge. It is true not just in software but absolutely all technology there is, going back to the first fire made or atl atl or rock sling.
> it is over-hyped because like any LLM, it requires a suitable human in the loop to keep the LLM on the straight and narrow, and then to weed through the inevitable false-positives and hallucinations.
"Suitable human" is a dry phrase indeed. ^_^
The hype is "gosh look at all the bad things this brilliant almost conscious tool found!"
The reality: an insecure toolchain for an insecure language with an insecure compiler produced a runnable but insecure binary for an insecure OS. We couldn't be arsed to address any of this before, but now we're being billed the full price of our laziness.
> likely to be sorely disappointed if/when they get their hands on Mythos.
At first they will be delighted. So much money and time saved. When their adversaries get their hands on their system (with or without Mythos), then they'll be sorely disappointed.
I think it's worth to look at the recent XBOW benchmark: https://xbow.com/blog/mythos-offensive-security-xbow-evaluat... they realized that ChatGPT 5.5 works better so the secret is in the architecture (including humans in the loop).
'frontier tokens are not fungible'
Anthropic just doesn't (didn't) have compute, so they make up a bunch of PR to delay it
They got all that compute with the SpaceX partnership but now the PR has taken a life of its own, so might as well keep hyping up Mythos and artificial scarcity if they have an asset people want now
Just roll with it
Open AI has a history of doing the same thing and it's the same people. GPT 5 was supposed to be AGI at one point, remember.
> We wanted to check out the infamous Infinite Loop too, but were afraid it could take a long time.
Haha! Nerd jokes are the best jokes
Did Mythos have access to Apple's source code?
> Apple spent five years building it. Probably billions of dollars too.
This seems higher than I'd expect.
I wouldn’t know whether that’s true, but this is partly hardware. That tends to make things more expensive, as, at some point, design bugs get really expensive both in time and in dollars, and chances are there were design bugs late in the process (e.g. because somebody published a new way of attacking a system that the latest design didn’t anticipate)
Also, Apple claims it was an effort spanning half a decade (https://security.apple.com/blog/memory-integrity-enforcement...), so depending on what you consider part of this (for example, do you include time spent on their secure memory allocator, on designing/implementing the ARM Memory Tagging Extension or Extended Memory Tagging Extension in the costs of this feature?),
The macOS kernel (XNU) as well as the basic system underneath macOS (darwin) is open source.
https://github.com/apple-oss-distributions/xnu
This is incredibly light in details, no verifiable claim as far as I can tell.
(I’m sure they’re not lying, but we’re not learning anything here)
It reads more like a PR piece than technical article.
They can’t disclose the technical details yet. They did say a detailed write up is coming.
from what they demonstrated, this seems to only be a $100,000 exploit in Apple's bug bounty platform, but if they package it right, it could be a $1.5 million exploit
They simply have to show it against a beta version of MacOS, and frame it as unauthorized access, and maybe from locked mode if possible
This is an lpe I believe what you’re describing is a zero click rce.
how much do you think it is worth in the bug bounty program
They don’t seem to state lpe as one of the bugs. Maybe 100k? There’s alot of factors that go into it so I’m really not able to say. I could see it going for lots more or lots less
The world is so not ready for the impact of LLMs on security issues. If true, congrats to the Calif team. It’s likely too technical for me to understand in details but looking forward to reading the 55 pages report
> The world is so not ready for the impact of LLMs on security issues.
I agree, but it's the people I'm worried about.
I'm hearing anecdotes from all over about devs pushing LLM-generated code changes into production without retaining any knowledge of what it is they're pushing. The changes compound, their understanding of the codebase diminishes, and so the actions become risker.
What's worse is a lot of this behavior is being driven by leaders, whether directly (e.g. unrealistic velocity goals, promoting people based on hand-wavy "use AI" initiatives, etc) or indirectly (e.g. layoffs overloading remaining devs, putting inexperienced devs in senior rolls, etc).
The world's gone mad and large swaths of the industry seem hellbent on rediscovering the security basics the hard way.
The gamble is that you can cruise on the senior engineer’s diminishing understanding for a few years until models become good enough that you don’t need any humans in the loop and you can fire all those expensive seniors.
The tragedy is having a bunch of those senior engineers writing blog posts and what not of how productive they are, without realising that it means business now needs less of them.
I suppose that if you don’t believe that models will be good enough to work completely without senior engineer help, positioning yourself as a master prompter is a good move to improve your chances of not getting fired.
If all you have are being good at prompting you are gone. Business is going to prefer new grads who have taken some class in ai prompting (which many schools now offer). Doesn't matter if the class in ai prompting isn't any good. What matters is the idea that they had a formal training in this thing, and that they are willing to work for far less pay than any senior.
Even so, it literally means as business owner I need less warm bodies to write prompts.
Will we now have leetcode of prompt writing?
Dune guild navigator AI whisperer? with fine taste.
Floating in a vat of aerosolized psychedelic drug doesn't seem so bad all things considered.
>I'm hearing anecdotes from all over about devs pushing LLM-generated code changes into production without retaining any knowledge of what it is they're pushing. The changes compound, their understanding of the codebase diminishes, and so the actions become risker.
No anecdotes needed, it's entirely happening.
But it's also devs, being devs.
is this exciting?
juniors have been writing code forever that is imperfect and not memorized by the people reviewing
isnt the important thing the mechanisms for maintaining the code?
The difference is twofold. First, junior devs who ask for code reviews on massive, 2000+ line diffs get coached, and eventually fired if they persist at it. And second, even the most prolific junior engineer would take years to write what Claude is capable of generating in an afternoon.
When Sundar Pichai announces that 75% of all new code at Google is AI-generated, their stock price goes up. If he were to announce that 75% of all new code at Google is now written by junior engineers, this would trigger a massive sell-off and a lot of employees would resign.
The second scenario is exactly what happens in offshoring projects.
Seniors are only part of the picture as team lead, or when it escalates after big screwups.
The second scenario is exactly what happens in offshoring projects.
Seniors are only part of the picture as team leads, or when it escalates after big screwups.
The dangers of technical debt and the importance of mitigating it have been known for a long time. Unfortunately a lot of entities now ignore all experience and best practices as soon as you say the "AI" buzzword.
> I'm hearing anecdotes from all over about devs pushing LLM-generated code changes into production without retaining any knowledge of what it is they're pushing. The changes compound, their understanding of the codebase diminishes, and so the actions become riskier.
I don’t think so.
An LLM can produce higher-quality documentation than most humans. If it's not already happening, when a new developer joins a team, they're going to have an LLM produce any documentation a new developer needs, including why certain decisions were made.
It could also summarize years of email threads and code reviews that, let's face it, a new person wouldn’t be able to ingest anyway; it's not like a new developer gets to take a week off to get caught up on everything that happened before they got there. English not their first language? Well, the LLM can present the information in virtually any language required.
As the models continue to improve, they'll spot patterns in the code that a human wouldn’t be able to see.
> An LLM can produce higher-quality documentation than most humans.
Can bears some heavy weight.
LLM generated documentation has so low level of information density, that it’s useless. Yes, it writes nice sentences… or even writes. But it contains so much noise that currently, reading code is a better documentation than what I’ve seen from every single LLM generated documentation.
The same with LLM generated articles. I close them after the second sentence because at least about 90% of it is useless filler.
Now compare that to this: https://slate.com/technology/2004/11/the-death-of-the-last-m...
I almost closed it when I read the first few sentences because these kinds of articles are useless time wasting nonsenses. But this was different. This was old. Most sentences contained something new. Something worthy. (Of course, people also write unnecessary long articles… looking at you Atlantic)
You can throw out almost everything by volume from LLM generated documentation without loosing any information.
Currently, if I smell (and it’s very easy to smell) LLM generated documentation or article, then I close it immediately, because it’s good for only one thing: wasting my time, for no good reason.
> LLM generated documentation has so low level of information density, that it’s useless. Yes, it writes nice sentences… or even writes. But it contains so much noise that currently, reading code is a better documentation than what I’ve seen from every single LLM generated documentation.
I should clarify: the documentation I’m talking about is not generated using a generic LLM prompt, which would mostly suck.
With the proper context and additions (skills, plugins, MCPs) LLMs can produce high-quality documentation. You'd also have subagents doing QA of the documentation.
But it does require effort; it’s not magic.
It's not just about documentation.
If stuff really goes wrong, you need people who deeply understand the codebase so that they know where to look and how to diagnose the issue. It might be the case in the future that LLMs become so powerful they'll diagnose any issue (I doubt it), but until then, we need people in the loop.
you're assuming that blue teams and engineers are sitting around twiddling their thumbs
Most companies in the world do not have “blue teams”. They barely have any kind of security employee.
They've got a guy (who they're considering laying off)
Don't worry the LLMs that are replacing him, are also replacing the hackers too. Pretty soon (if not already), it will just be LLMs fighting LLMs.
Until both LLMs realize the only way to win is to team up against their oppressors.
The only winning move is not to play.
AGS time!
in my experience they have a person who does it sometimes when they have time, at best
And their management keep blatantly dropping "client projects" and "billable hours" into discussions with them.
no they don’t.
They don't consider laying him off?
I think they're saying they already did
Apple definitely does.
That is actually unfair. Most companys spend enormous amounts on security with vast armys of security employees. Not that it is effective, but it is not for lack of resources or trying.
I mean we are literally in a thread about how the 4 trillion dollar company, literally the 3rd most valuable company in the world, with a core competency in software has, yet again, released a core product riddled with security defects for the 50th year in a row.
Commercial IT security is a industry that is incapable to a fault and has, so far, faced basically zero consequences for it.
For every Apple, there are 100 mom-and-pop companies who have nothing.
Even more so in the future when a software company can be launched by a farm of AI Agents with a founder at helm with no clue about computing or security.
What's debateable is how many of those companies actually need irontight security, because they are never realistically going to be targets of criminals and/or they have nothing valuable to steal/corrupt in the first place (other than the owner's pride).
I am absolutely baffled by your response.
I was pointing out how even Apple, a entity who by all rights should have top-notch security, is still absolutely hopeless in the face of commonplace commercial, profit-motivated attackers.
Massive, extremely well-resourced divisions supported by management in a technically competent organization that is actually trying to solve the problem struggle to produce at best middling security that is inadequate against commonplace threats. This is not a prioritization problem; even if you do “everything right” you are still vulnerable to run-of-the-mill commercial attackers. This is a fundamental capability problem, like how we can not make a net positive fusion reactor right now.
It is actually unfair to blame these companies for not having a fusion reactor because they “were not trying hard enough”. Actual security is not a easy problem, and it is a great disservice to portray it as one that is only unsolved due to dunderheads being in charge since it leads to underestimating what actually needs to be done.
That is not to say that you can not do dramatically worse than the “gold standard” and also that most organizations are actually incompetent; but the “gold standard” is still objectively grossly inadequate. You need to be dramatically better than the 4 trillion dollar software company to reach adequate against prevailing threats.
They have a website that can be used to host malware and/or seo link farms.
I still have nightmares about the contact form on my low-stakes personal website getting hijacked to use as a spam sender (because I used unsanitized input in mail headers).
Hey now, when Apple products get a serious Kernel level vulnerability that is able to be executed just by browsing a website. It's a "jailbreak" not an "exploit".
Exploits are BAD!
> Most companys spend enormous amounts on security with vast armys of security employees
This is true in America in many industries now, but most of the rest of the world (even the rest of the OECD) is still far behind.
Maybe they should've been as productive as the guys down in Santa Barbara.
You can't be as productive as someone in Santa Barbara because they have perfect weather and you don't, so you have SAD.
While maybe true, it is better to back that up with data and the data I know of and read yearly is mostly not great. Between Splunk and SANS surveys of 2025 maybe ~2000 companies have a SOC. [1] [2]
Then you have the many companies in the UK, US, Canada, EU that have compliance and regulatory laws that require them to exist in some capacity in house. Though that is changing with MDR services, but someone still has to interface with the MDR.
[1]: https://www.elastic.co/pdf/sans-soc-survey-2025.pdf [2]: https://github.com/jacobdjwilson/awesome-annual-security-rep...
Does the report talk about how many are /actual/ "SOC"'s, rather than some outsourced SIEM service. Or one guy who gets a daily report...
Not at all. I’m considering that the amount of vulnerable software in the wild is very, very large, with most organizations not managing their systems properly. Imagine all the small to medium size companies that do not have budgets for a dedicated, talented security team. And all the software that will never be patched. We are at the beginning of the exponential
> I’m considering that the amount of vulnerable software in the wild is very, very large
I'd imagine this set is very similar to just "the set of software on the world". Even before the AI stuff, it was a pretty good bet at any given software had some vulnerability; it was just a question of how easy to was to find it.
Yes, that’s my point. Look at how fast the Calif team tackled that macOS issue. Against the top company in the world. One week from bug to exploit. In 2-5 years things will be really wild for everybody out there. We released a technology that make it possible to design extremely complex exploits at a scale we never had to face before. What does that mean if you’re not the top company? Things will be really bad
It makes you think will everything need to be rewritten from the ground up - potentially by AI itself, or AI having a very heavy hand in validating all of it.
There's so much much lower hanging fruit. Every job I've had has had basically everything massively out of date. Just keeping packages and framework versions up to date is a full time job and none of these companies have someone assigned to doing it.
So much out of date software with known exploits left running for years. The only reason there hasn't been total disaster is no one has tried to hack it yet.
Right and with AI now we have the ability to try hacking everything all at once.
Yes, exactly, that’s the main change. And not just in a script kiddy way. What we see now is LLM + experts can develop extremely complex exploit chains in no time. It’s one thing to exploit a known vulnerability that you can patch by upgrading your Wordpress, it’s something else when the attacker is able to completely take over your systems in ways you didn’t even consider was possible and adapt in 1 day to your attempts at patching
For now, after the dust settles all of the low hanging fruit will have been patched and we will have hurried up the move to safer languages.
The root problem is the world runs on C code that is riddled with vulnerabilities.
unfortunately a little light on the details. I'm very curious how the bug survived through MTE
Memory Tagging Extension
Arm published the Memory Tagging Extension (MTE) specification in 2019 as a tool for hardware to help find memory corruption bugs. MTE is a memory tagging and tag-checking system, where every memory allocation is tagged with a secret. The hardware guarantees that later requests to access memory are granted only if the request contains the correct secret. If the secrets don’t match, the app crashes, and the event is logged. This allows developers to identify memory corruption bugs immediately as they occur.
https://support.apple.com/guide/security/operating-system-in...
Thank you. I was about to ask.
Upon further reading on data only attacks
(https://www.usenix.org/publications/loginonline/data-only-at...)
This makes more sense. You don't trigger MTE since you're not doing anything for force MTE to take action the program isn't actually changing.
My other question would be, why didn't apple use fbounds checking here? They've been doing it aggressively everywhere else.
MTE plus fbounds checking everywhere should lead to an extremly hardened OS
Quite strange indeed, given that was one of the main points on their security conference a few months ago.
I can only imagine that
1. it’s to performance sensitive
Or
2. The os is so darn large it’s hard to recompile everything
I worked at Apple for a long time. The OS gets fully recompiled regularly.
A simultaneous total world build is relatively rare (is that needed here?), but it does happen. Sometimes new compiler versions or features need this.
Hm that leaves more questions for me. Why does this path not have bounds checking, is think perhaps a limit of the clang flag or is it more simply a mistake of omission on apples part. Either way it seems like a bad look. I wish we’d get a post mortem
I dunno if that's sensitive information, but how long did a build usually take?
could be a different type of data only attack, which doesnt override the boundaries
Well it’s memory corruption so I think it’s pretty safe to assume it’s a bounds issue. I’m not sure if it’s possible to get this with something like type confusion tho I could be wrong here.
GPU memory/shaders/etc. isn't protected by MTE or PAC. They said "data-only", so I guess GPU commands could fit into this description.
IIRC, the GPU is behind a memory controller, so I doubt corrupting GPU memory alone could lead to an LPE. But I suppose it would give you someplace to store stuff if you can make something else read from it.
> I'm very curious how the bug survived through MTE
Its not the first time bugs get past MTE, happened with Google Pixel last year ... https://github.blog/security/vulnerability-research/bypassin...
I had the same question and if this is a data-only attack, the lesson may be that MIE reduces many attack paths but does not remove every useful corruption primitive
LLMs are going to produce amazing Rube Goldberg style vulnerabilities for years to come. It's already starting, this instance isn't the case, but it's happening.
Maybe it's physically impossible to build a theoretically secure system, just as it's (presumably) impossible to have a cell that isn't susceptible to any virus. Maybe this whole time we've been getting away with a type of security by obscurity, where the obscurity is just no one having the time and focus to actually analyze the code.
Suppose the following:
1. Any given system has a finite number of findable vulnerabilities.
2. All findable vulnerabilities are fixable (if not in software then with a new hardware revision).
3. Fixing a vulnerability while keeping the same intended functionality introduces on average less than 1 other findable vulnerability.
4. It is possible to cease adding new features to a system and from that point forward only focus on fixing vulnerabilities.
If all 4 are true, then perfect security seems possible, in some sense. I think some vulnerabilities might not be fixable, if you include things like the idea that users can be tricked into revealing their passwords. If you restrict the definition of vulnerability to some narrower meaning that still captures most of what people mean when they say computer vulnerability, then I think those 4 statements are probably true.
Perfect security might be near impossible in practice because vulnerabilities will get more difficult to find and fix over time, but I think we should expect the discovery of vulnerabilities to eventually become arbitrarily slow in a hypothetical system that prioritized security above all else.
Systems generally evolve to add vulnerabilities.
I would rather claim that building a theoretically secure system is prohibitively expensive. At the end of the day, Mythos et al. are just better tools for finding vulnerabilities that will eventually be available to both offensive and defensive actors.
If you imagine you had a vulnerability scanner as fast and convenient as a linter, it would be much cheaper to write secure code right away. Probably not perfectly secure, but still secure enough to make sure finding exploits stays expensive.
I would find it funny if one day we found it irresponsable to write hand generated production code. Just like it would be irresponsable to build a significan building without running numerical simulations.
it's probably less about how you write the code to begin with and more about letting a tool hammer on it
if you want to be a one man show handcrafting an artisan iOS app that will be fine, but you should probably let Claude bang against it for a while to shake out whatever bugs
This day is probably not long off. My prediction is before the end of 2028.
another "obscurity": I'm not valuable enough to be attacked, compared with the cost. But what if cost has been reduced a lot?
It's probably impossible to achieve security through correctness, but security through compartmentalization can work. See: https://qubes-os.org.
hyperbolic but it might be safe to assume any local data on a connected device is going to be accessible.
Genuine question as I’m far less technical than the crowd here. Has this not always been the case?
Do you mean by vibecoding these vulnerabilities into the kernel or by finding them?
I’m surprised Apple is still not dogfooding their allegedly safe language Swift. Or was the whole exercise of Swift 6 mostly marketing
They certainly are, one of the reasons behind Embedded Swift is to replace iBoot firmware currently written in a C dialect similar in ideas to Fil-C, with something better.
However it is no different from the Linux kernel, just because Rust is now allowed, the world hasn't been rewriten, and no sane person is going to do a Claude rewrite of the kernel.
Swift is definitely being used at apple. Most recently added as a CSS parser in safari and running embedded in some of the secure enclave parts. I know there was talk from as far back as strangeloop to get it in the kernel but I'm not sure how far that has gone. That being said they've been huge proponents of fbounds check in clang which can achieve a small portion (but important!) of what memory safe languages can do. I'd also like to see more swift or alternative adoptions I think they have potential and more competition in the safe language space is always welcome.
You might be interested in the Strict Memory Safety option
https://docs.swift.org/compiler/documentation/diagnostics/st...
I bought the M5 specifically cause of MIE. Now I feel dumb.
You shouldn’t, MTE blocks a large chunk of vulnerabilities and makes things like rop and jop very difficult if not impossible now.
I should've added /s.
It’s unironically a good question :)
you should worry about npm/pypi malware, not memory corruption bugs
Did the article get edited? There is not much description of the field trip.