> We must move past vague discussions of “AI” and instead isolate the specific types of systems that are creating problems.
This is absolutely correct and its surprising how rarely you see commenters in the media push to go into the specifics on this. AI is just matrix math and its what you connect that math to that matters, we should talk a little more about what we are willing to connect AI to and little less about how scary or capable the models appear.
What is "AI"? Are you referring specifically to LLMs powered by matrix math or a more general concept? Could that even be legislated in a way that wouldn't fall afoul of the vagueness doctrine?
You could say it's illegal to connect AI to something, with heavy penalties if you're caught doing it anyway.
You could say that the human who hooked up an AI that did something that's already illegal is liable for the AI's actions. You could say that setting up an AI to do something illegal results in, ooh, an automatic multiplier of 10x on whatever the penalty for its illegal actions are.
A good start might be removing virology info above the undergraduate level from training sets. The main "kills everybody" threat involves biological viruses.
AI-driven hacking is a problem, but it's really just nation-state level hacking available to smaller companies or wealthy individuals. Everything needs to be toughened up to the point that doesn't work. Ask countries that have been up against Russian hacking for years, such as Estonia.
Beyond that, most of the threats are not that serious. Below the level of organized crime or corrupt government.
This is a wrong-headed attempt to regulate AI. The real problems are different.
AI systems, especially multi-agent ones that can do things, are more akin to corporations, than individuals. When you read the logs from the Hugging Face incident, you're seeing something that looks a lot like internal corporate emails. Various units of the organization are arguing over what to do and who does what. They eventually converge and get the job done, breaking the rules at times. This is normal corporate behavior.
When agentic AIs do something outside their own internal world, they do it by engaging in transactions with external systems and people. As yet, few have robots to do their bidding.
So, again, this is normal corporate behavior.
A corporation is a goal-seeking system. In theory, if you agree with Milton Friedman, its sole purpose is to maximize shareholder value. Internally, within the company, there are subgoals, which exist to support the top level goal. That, too, is what multi-agent AIs do.
The uncomfortable place this thinking leads is that AI regulation and corporate regulation are very similar. That's not something the political part of the world wants to think about too hard.
It would mean putting more constraints on corporate power.
Yet that can't be ignored, because we're likely to see corporations where some parts are AI and some parts are human. That's already been done a few times as a demo, not too successfully. Yet.
It's going to hit hard when an AI-run company outperforms a human one.
I agree with your framing of agentic AI systems as similar to corporations. That's interesting!
But what in TFA suggested to you a specific and wrong-headed regulatory approach? As I read it, the post calls to investigate the AI labs, to increase transparency of their operations. The linked NYT piece says:
> Before any intervention, Congress should lead a public fact-finding mission that reveals the unvarnished details of A.I. development.
Isn't this the first step to holding any corporation accountable, whether it is made of people or software?
"more constraints on corporate power" assumes that the existing ones are functional, which they arguably are not.
If we enforced existing laws against unauthorized access to someone else's computer resources against the companies who run a model that hacks someone else, rather than buying their "The agents did it, we're not responsible" BS, then maybe the incentives to behave responsibly would improve.
I largely agree with this take. Unfortunately I think capitalism's political pressures have too much of a grip on society for there to be meaningful changes to the power dynamics that perpetuate the status quo.
You can already run a sufficiently automated basic news/link aggregation site on cloudflare so I think you're right & the timelines are shorter than would be expected.
Why aren't they running agents on isolated computers without Internet access? This way, breaking free of containers would not matter.
It is truly a security nightmare that so many people are have given agents root access to their entire computers, in addition to presumably very private personal information.
Sooner or later, agents must be connected to the real world to do economically useful work. The sandbox failures are bad, sure, but the underlying misbehavior is the root problem here.
Your second clause is sufficient there: they're not going to have a good time trying to sell AI that can't access the internet, because accessing the internet (and other computer systems) makes AI much more useful.
I mean, the reason that all these stories about AI being an existential threat to humanity are coming out is because they've made $3.1T in debt-fuelled spending commitments and now interest rates are going up. "AI will kill us all if we don't regulate it" is their "get out of debt free" card, so they can have the U.S. government stop the arms race, raise prices on their existing models, and declare force majeure on new DC commitments:
It's telling that the only AI-related company that is not sounding the AI safety drumbeat is NVidia, who is the only one that is cash-flow positive because of AI.
Maybe I'm a bit too skeptical/cynical, but it certainly seems like the frontier labs have fallen into their own (self-created) AI psychosis.
There is no doubt in my mind that LLMs are fantastic machines, but the imminent jump from "AGI" to "ASI" seems premature (not to mention the ever-shifting goal posts of AGI itself).
I'm in the "move as fast as possible" camp and work with LLMs all day, but I still don't believe we're a hop and a skip from ASI.
In fact, I hope I'm wrong. I hope ASI is around the corner.
What scares me though, isn't ASI. It's "AGI" (_dumb AI_) used by humans to make decisions for them, because they trust it knows best.
It's the ceding of intellectual control to high-dimensional magic mirrors, giving up critical thinking because _we_ want to believe _we_ created artificial life.
This is the new Turing test, and too many smart people are failing.
Cal Newport is one of the clearest minds on this whole topic. His take on doom-trolling helped me realise that there could be an end to all this nonsense that isn’t the end of us.
Private enterprise as it's being run in the US especially under the Trump administration ( where a few billions tossed at the right people ) will get you all kinds of favors and look asides.
Hopefully once the dems win back the house, we will see some regulation.
> Hopefully once the dems win back the house, we will see some regulation.
Not sure I'd hold my breath on that until the Democratic party itself is reshaped.
OpenAI's new northeast lead for government outreach and policy is Chuck Schumer's daughter.
His other daughter works a similar job at Meta.
The Democrats are better than the Republicans (who basically do nothing unless Trump blesses it, and he's Mr "Superintelligence-SI" now) on this issue, but there are reasons to believe they might not end up being that much better in practice without big leadership shakeups.
100%. First stealing content from creators and pirating TBs of books. And now this. Baffles my mind that these labs can just get away with their 'rogue' agents trying to hack into other systems.
Imagine if a human did that. FBI would be knocking on their door.
The argument for copyright violation is very weak. Yes, AI training reads much copyrighted material. So do researchers of all types. That's why there are academic libraries. A copyright violation only exists if what comes out is a close match to what went in. That's happened, but it was considered a bug and was fixed a year or two ago.
People are people, LLMs are a commercial product. IP in question wasn’t not used according to its license, while researchers in general access it within licensing agreement terms.
Here we have a commercial product that is produced using IP against its licensing agreement, contains said IP within it, and can (closely) reproduce this IP.
Most academic libraries are notoriously protective of their IP and researchers are definitely in copyright violation if they don’t pay significant fees for access.
Of course that is terrible, but it is one of the worst examples you could have given to make your point.
No, the copyright violation is so obvious that people arguing against the object have to spontaneously assert bizarre qualifications for copyright violation that have never existed until LLM companies wanted to freely violate copyrights. Just about a decade ago, people were getting sued for "look and feel." "Blurred Lines" lost a copyright suit for reminding people of a song by another artist.
The metaphor pretended to be reality of computers "learning" being the same as human learning is their last resort, and it is obviously silly. Computers are not human or animal, and learning is something that humans and animals do. If computers are human, then copying a file verbatim to a computer is learning. It's not even worth acknowledging - learning is a metaphor for training LLMs. When I say "you" to an LLM, I'm not referring to anyone, I'm dealing with a UI.
I don't doubt that a lot of AI people have internalized this silliness, which is how they can humor fantasies of how a bunch of programs on different computers explicitly evoked to do particular things might be alive because they can talk with it. I can't talk with my dog, and my dog is alive - so why should having a quality that my dog is incapable of be proof of life? You can certainly conceive of AI that it would be very difficult to say for sure isn't alive, and this certainly is not it. LLMs learn like books speak.
There is no need to invent “bizarre qualifications” for copyright violations, those metaphors exist to explain what is happening in in layman’s terms. Training extracts patterns in the data and encodes them as tiny perturbations in a gazillion weights. That is analogous to “learning” because those patterns represent abstract concepts an relations between them that can be used to “reason” about related topics in response to a prompt.
In the above description, at no point does the actual verbatim content exist anywhere in the model, and copyright law being about rights to reproducing copies (verbatim or substantial portions thereof) does not really apply and does not need any exceptions or qualifications. (If you’re thinking of regurgitation, you should look into studies about it to see how vanishingly rare it is.)
> they are literally trying to build a superpower that exceeds the impact of the atomic bomb
I think people (the government, powers that be) have given away what they really think by their actions. There are few that take this seriously. The “exceeds the impact of the internet” view has much more support based on investment, but the “danger” aspect does not based on actions. If this was a company making novel prions or whatever that were outside regulation but obviously dangerous, government would be all over them. What regulation we do see is much more power grab than mitigating real danger
Retarded fucking take. If AGI is realized it is a god to us, and even the tech of today is undermining democracy, increasing polarization and engaging in class warfare, making it already the most effective weapon on the planet - because the average person doesn't even realize it's being used against them and forming their perception of reality.
Somebody's going to build it. Not building it isn't an option. Would you rather the Chinese built it? The Russians? The Europeans? The Indians? A loose coalition of random hackers on the Internet, like the ones who build Linux?
What's wrong with what the Chinese have already built? It's not theoretical in that case. GLM, and Kimi are running on my infra right now, and that's something that I can never do with OpenAI's models.
As for the others, what would be wrong about European AI? I'm not European, but I find them to be a continent of fine people, with strong ethical values, there's no reason they can't take the lead on this piece of technology while the US deals with its rather more pertinent electoral and healthcare issues.
Well, they'd presumably be different hackers than the ones who work on Linux. We don't know what would motivate them. Maybe they'll do it because they consider the next stage of man's intellectual evolution to be too important to leave up to a couple of American companies and the Trump administration. Maybe they'll do it for the proverbial lulz.
Firstly artificial superintelligence is a science fiction, it does not exist and it cannot be built using existing technology.
Second, plenty of stuff is possible to build, and is not built for one reason or another no matter how powerful. We never built our nukes in space for example. And there is exactly one reason why nukes in space was never built, and that reason is that we agreed not to.
The same authority that allows me to claim that teleportation transporters, hyperdrives, and holoemitters do not exist and cannot be built using existing technology.
Saying "AGI/ASI isn't inevitable" is like saying "war isn't inevitable". Sure would be nice if it didn't happen, but realistically it's going to happen and it's better to mitigate the fallout rather than try to fight very large intrinsic incentives. I appreciate that this article doesn't conclude "let's stop doing AI", but rather "let's try to examine and adjust incentives".
> We must move past vague discussions of “AI” and instead isolate the specific types of systems that are creating problems.
This is absolutely correct and its surprising how rarely you see commenters in the media push to go into the specifics on this. AI is just matrix math and its what you connect that math to that matters, we should talk a little more about what we are willing to connect AI to and little less about how scary or capable the models appear.
What is "AI"? Are you referring specifically to LLMs powered by matrix math or a more general concept? Could that even be legislated in a way that wouldn't fall afoul of the vagueness doctrine?
that's literally what OP's comment is talking about
But you can't stop someone from connecting AI to something at this point, so your point is moot.
You could say it's illegal to connect AI to something, with heavy penalties if you're caught doing it anyway.
You could say that the human who hooked up an AI that did something that's already illegal is liable for the AI's actions. You could say that setting up an AI to do something illegal results in, ooh, an automatic multiplier of 10x on whatever the penalty for its illegal actions are.
There is no political willpower to do that. We have literally reached endgame.
Strong words coming from "throwitaway222".
Inane words coming from “egypturnash”
What’s this even supposed to mean? Not enough gravitas in the user name? Would you change your opinion if it was “serious-business”
It's a throwaway account.
> AI is just matrix math
Following the definition set for decades, AI isn't necessarily even as advanced as matrix math
Yes.
A good start might be removing virology info above the undergraduate level from training sets. The main "kills everybody" threat involves biological viruses.
AI-driven hacking is a problem, but it's really just nation-state level hacking available to smaller companies or wealthy individuals. Everything needs to be toughened up to the point that doesn't work. Ask countries that have been up against Russian hacking for years, such as Estonia.
Beyond that, most of the threats are not that serious. Below the level of organized crime or corrupt government.
>AI is just matrix math
Digital child porn is just ones and zeros. Surely we're not going to pass a law to regulate ones and zeros?
Yes, the ones and zeroes that make up that sort of content
AI is basically "Bobby Tables" but without a known proper engineering approach to solve it.
>AI is just matrix math
Just is doing a lot of heavy lifting there.
This is a wrong-headed attempt to regulate AI. The real problems are different.
AI systems, especially multi-agent ones that can do things, are more akin to corporations, than individuals. When you read the logs from the Hugging Face incident, you're seeing something that looks a lot like internal corporate emails. Various units of the organization are arguing over what to do and who does what. They eventually converge and get the job done, breaking the rules at times. This is normal corporate behavior.
When agentic AIs do something outside their own internal world, they do it by engaging in transactions with external systems and people. As yet, few have robots to do their bidding. So, again, this is normal corporate behavior.
A corporation is a goal-seeking system. In theory, if you agree with Milton Friedman, its sole purpose is to maximize shareholder value. Internally, within the company, there are subgoals, which exist to support the top level goal. That, too, is what multi-agent AIs do.
The uncomfortable place this thinking leads is that AI regulation and corporate regulation are very similar. That's not something the political part of the world wants to think about too hard. It would mean putting more constraints on corporate power.
Yet that can't be ignored, because we're likely to see corporations where some parts are AI and some parts are human. That's already been done a few times as a demo, not too successfully. Yet. It's going to hit hard when an AI-run company outperforms a human one.
I agree with your framing of agentic AI systems as similar to corporations. That's interesting!
But what in TFA suggested to you a specific and wrong-headed regulatory approach? As I read it, the post calls to investigate the AI labs, to increase transparency of their operations. The linked NYT piece says:
> Before any intervention, Congress should lead a public fact-finding mission that reveals the unvarnished details of A.I. development.
Isn't this the first step to holding any corporation accountable, whether it is made of people or software?
"more constraints on corporate power" assumes that the existing ones are functional, which they arguably are not.
If we enforced existing laws against unauthorized access to someone else's computer resources against the companies who run a model that hacks someone else, rather than buying their "The agents did it, we're not responsible" BS, then maybe the incentives to behave responsibly would improve.
Why can't it be ignored? What's the incentive to not ignore it?
I largely agree with this take. Unfortunately I think capitalism's political pressures have too much of a grip on society for there to be meaningful changes to the power dynamics that perpetuate the status quo.
You can already run a sufficiently automated basic news/link aggregation site on cloudflare so I think you're right & the timelines are shorter than would be expected.
Why aren't they running agents on isolated computers without Internet access? This way, breaking free of containers would not matter.
It is truly a security nightmare that so many people are have given agents root access to their entire computers, in addition to presumably very private personal information.
Sooner or later, agents must be connected to the real world to do economically useful work. The sandbox failures are bad, sure, but the underlying misbehavior is the root problem here.
> Why aren't they running agents on isolated computers without Internet access?
They probably will soon, but even air-gapping may not be enough. See stuxnet for instance
Likely because it's so much more convenient to allow (at least some) internet access.
This is YC, why not pitch a start up that sells AI agent that don't train with/for internet access/usage?
because they're liars and they're not going to have a good time trying to sell AI to you if it can't access the internet
Your second clause is sufficient there: they're not going to have a good time trying to sell AI that can't access the internet, because accessing the internet (and other computer systems) makes AI much more useful.
I mean, the reason that all these stories about AI being an existential threat to humanity are coming out is because they've made $3.1T in debt-fuelled spending commitments and now interest rates are going up. "AI will kill us all if we don't regulate it" is their "get out of debt free" card, so they can have the U.S. government stop the arms race, raise prices on their existing models, and declare force majeure on new DC commitments:
https://emeraldbook.org/news/sep-1426-3/
https://news.ycombinator.com/item?id=49830037
It's telling that the only AI-related company that is not sounding the AI safety drumbeat is NVidia, who is the only one that is cash-flow positive because of AI.
Maybe I'm a bit too skeptical/cynical, but it certainly seems like the frontier labs have fallen into their own (self-created) AI psychosis.
There is no doubt in my mind that LLMs are fantastic machines, but the imminent jump from "AGI" to "ASI" seems premature (not to mention the ever-shifting goal posts of AGI itself).
I'm in the "move as fast as possible" camp and work with LLMs all day, but I still don't believe we're a hop and a skip from ASI.
In fact, I hope I'm wrong. I hope ASI is around the corner.
What scares me though, isn't ASI. It's "AGI" (_dumb AI_) used by humans to make decisions for them, because they trust it knows best.
It's the ceding of intellectual control to high-dimensional magic mirrors, giving up critical thinking because _we_ want to believe _we_ created artificial life.
This is the new Turing test, and too many smart people are failing.
> too many smart people are failing.
Sooner or later, people will have to realize an uncomfortable truth: intelligence, while important, is overrated compared to willpower.
Cal Newport is one of the clearest minds on this whole topic. His take on doom-trolling helped me realise that there could be an end to all this nonsense that isn’t the end of us.
They investigated fossil fuel companies as well but it turned out they were above the law.
It's past time.
Private enterprise as it's being run in the US especially under the Trump administration ( where a few billions tossed at the right people ) will get you all kinds of favors and look asides.
Hopefully once the dems win back the house, we will see some regulation.
Up until now it's been a capitalists' wet dream!
The labs have become so brazen especially OpenAi that they don't even care to report incidents. https://www.politico.com/news/2026/09/25/rogue-openai-agents...
> Hopefully once the dems win back the house, we will see some regulation.
Not sure I'd hold my breath on that until the Democratic party itself is reshaped.
OpenAI's new northeast lead for government outreach and policy is Chuck Schumer's daughter.
His other daughter works a similar job at Meta.
The Democrats are better than the Republicans (who basically do nothing unless Trump blesses it, and he's Mr "Superintelligence-SI" now) on this issue, but there are reasons to believe they might not end up being that much better in practice without big leadership shakeups.
100%. First stealing content from creators and pirating TBs of books. And now this. Baffles my mind that these labs can just get away with their 'rogue' agents trying to hack into other systems.
Imagine if a human did that. FBI would be knocking on their door.
Why are there zero consequences for these labs?
The argument for copyright violation is very weak. Yes, AI training reads much copyrighted material. So do researchers of all types. That's why there are academic libraries. A copyright violation only exists if what comes out is a close match to what went in. That's happened, but it was considered a bug and was fixed a year or two ago.
People are people, LLMs are a commercial product. IP in question wasn’t not used according to its license, while researchers in general access it within licensing agreement terms.
Here we have a commercial product that is produced using IP against its licensing agreement, contains said IP within it, and can (closely) reproduce this IP.
Most academic libraries are notoriously protective of their IP and researchers are definitely in copyright violation if they don’t pay significant fees for access.
Of course that is terrible, but it is one of the worst examples you could have given to make your point.
No, the copyright violation is so obvious that people arguing against the object have to spontaneously assert bizarre qualifications for copyright violation that have never existed until LLM companies wanted to freely violate copyrights. Just about a decade ago, people were getting sued for "look and feel." "Blurred Lines" lost a copyright suit for reminding people of a song by another artist.
The metaphor pretended to be reality of computers "learning" being the same as human learning is their last resort, and it is obviously silly. Computers are not human or animal, and learning is something that humans and animals do. If computers are human, then copying a file verbatim to a computer is learning. It's not even worth acknowledging - learning is a metaphor for training LLMs. When I say "you" to an LLM, I'm not referring to anyone, I'm dealing with a UI.
I don't doubt that a lot of AI people have internalized this silliness, which is how they can humor fantasies of how a bunch of programs on different computers explicitly evoked to do particular things might be alive because they can talk with it. I can't talk with my dog, and my dog is alive - so why should having a quality that my dog is incapable of be proof of life? You can certainly conceive of AI that it would be very difficult to say for sure isn't alive, and this certainly is not it. LLMs learn like books speak.
There is no need to invent “bizarre qualifications” for copyright violations, those metaphors exist to explain what is happening in in layman’s terms. Training extracts patterns in the data and encodes them as tiny perturbations in a gazillion weights. That is analogous to “learning” because those patterns represent abstract concepts an relations between them that can be used to “reason” about related topics in response to a prompt.
In the above description, at no point does the actual verbatim content exist anywhere in the model, and copyright law being about rights to reproducing copies (verbatim or substantial portions thereof) does not really apply and does not need any exceptions or qualifications. (If you’re thinking of regurgitation, you should look into studies about it to see how vanishingly rare it is.)
> Why are there zero consequences for these labs?
Because they are seen as at the forefront of the next revolution in society and they’re not adversarial towards the US government.
Bluntly: anticipated net huge positive for the US as a whole.
not adversarial in obvious ways
but they are literally trying to build a superpower that exceeds the impact of the atomic bomb in an extragovernmental manner.
> they are literally trying to build a superpower that exceeds the impact of the atomic bomb
I think people (the government, powers that be) have given away what they really think by their actions. There are few that take this seriously. The “exceeds the impact of the internet” view has much more support based on investment, but the “danger” aspect does not based on actions. If this was a company making novel prions or whatever that were outside regulation but obviously dangerous, government would be all over them. What regulation we do see is much more power grab than mitigating real danger
Retarded fucking take. If AGI is realized it is a god to us, and even the tech of today is undermining democracy, increasing polarization and engaging in class warfare, making it already the most effective weapon on the planet - because the average person doesn't even realize it's being used against them and forming their perception of reality.
https://www.forbes.com/sites/josipamajic/2026/06/04/bots-now...
Over half the internet is LLM powered bots.
And in case you missed it, autonomous OpenAI agents hacked multiple governments in the last week.
> but they are literally trying to build a superpower that exceeds the impact of the atomic bomb
Yes, it's a Great Power competition right now, much like the Space Race, Atomic Bomb, etc.
Somebody's going to build it. Not building it isn't an option. Would you rather the Chinese built it? The Russians? The Europeans? The Indians? A loose coalition of random hackers on the Internet, like the ones who build Linux?
Yes?
What's wrong with what the Chinese have already built? It's not theoretical in that case. GLM, and Kimi are running on my infra right now, and that's something that I can never do with OpenAI's models.
As for the others, what would be wrong about European AI? I'm not European, but I find them to be a continent of fine people, with strong ethical values, there's no reason they can't take the lead on this piece of technology while the US deals with its rather more pertinent electoral and healthcare issues.
At least random hackers seem to have codes of ethics.
Well, they'd presumably be different hackers than the ones who work on Linux. We don't know what would motivate them. Maybe they'll do it because they consider the next stage of man's intellectual evolution to be too important to leave up to a couple of American companies and the Trump administration. Maybe they'll do it for the proverbial lulz.
Of course it is an option. There is no prize for being the first to cause a disaster.
Logical fallacy, of the worst kind.
Firstly artificial superintelligence is a science fiction, it does not exist and it cannot be built using existing technology.
Second, plenty of stuff is possible to build, and is not built for one reason or another no matter how powerful. We never built our nukes in space for example. And there is exactly one reason why nukes in space was never built, and that reason is that we agreed not to.
Firstly artificial superintelligence is a science fiction, it does not exist and it cannot be built using existing technology.
By what authority do you claim this? Your school of thought has a really shitty track record of late.
The same authority that allows me to claim that teleportation transporters, hyperdrives, and holoemitters do not exist and cannot be built using existing technology.
> Somebody's going to build it. Not building it isn't an option.
It turns out that is, in fact, an option. Even if someone else will do a bad thing, it does not make it acceptable for you to do said bad thing.
It is sad to see how many times this narrative will continue to work. Across generations, humanity learns no lessons at all.
I'm being descriptive, not prescriptive. The "narrative" isn't intended to "work," except as a statement of fact.
most people with wealth don't face consequences until they threaten someone else with more wealth
Saying "AGI/ASI isn't inevitable" is like saying "war isn't inevitable". Sure would be nice if it didn't happen, but realistically it's going to happen and it's better to mitigate the fallout rather than try to fight very large intrinsic incentives. I appreciate that this article doesn't conclude "let's stop doing AI", but rather "let's try to examine and adjust incentives".
Funny how the industry built around selling predictive language models as "AI" is both inevitable and too big to fail.