If you can get malicious instructions into the context of even the most powerful reasoning LLMs in the world you'll still be able to trick them into outputting vulnerable code like this if you try hard enough.
I don't think the fact that small models are easier to trick is particularly interesting from a security perspective, because you need to assume that ANY model can be prompt injected by a suitably motivated attacker.
On that basis I agree with the article that we need to be using additional layers of protection that work against compromised models, such as robust sandboxed execution of generated code and maybe techniques like static analysis too (I'm less sold on those, I expect plenty of malicious vulnerabilities could sneak past them.)
We started giving our (https://www.definite.app/) agent a sandbox (we use e2b.dev) and it's solved so many problems. It's created new problems, but net-net it's been a huge improvement.
Something like "where do we store temporary files the agent creates?" becomes obvious if you have a sandbox you can spin up and down in a couple seconds.
The most "shocking" thing to me in the article is that people (apparently) think it's acceptable to run a system where content you've never seen can be fed into the LLM when it's generating code that you're putting in production. In my opinion, if you're doing that, your whole system is already compromised and you need to literally throw away what you're doing and start over.
Generally I hate these "defense in depth" strategies that start out with doing something totally brain-dead and insecure, and then trying to paper over it with sandboxes and policies. Maybe just don't do the idiotic thing in the first place?
It wasn't, but the written version of it it is actually better than what I said in the room (since I got to think a little bit harder and add relevant links).
IIUC your talk "just" suggests using sandbox-exec on Mac, which (as you point out) is sadly labeled as deprecated.
Is that really the best solution the world has to offer in 2025? LLMs aside, there is a whole host of supply chain risk issues that would be resolved by deploying convenient and strong sandboxes everywhere.
> The conventional wisdom that local, on-premise models offer a security advantage is flawed. While they provide data privacy, our research shows their weaker reasoning and alignment capabilities make them easier targets for sabotage.
Yeah, I'm not following here. If you just run something like deepseek locally, you're going to be okay provided you don't feed it a bogus prompt.
Outside of a user copy-pasting a prompt from the wild, or break isolation by giving it access to outside resources, the conventional wisdom holds up just fine. The operator and consumption of 3rd party stuff are weak-points for all IT, and have been for ages. Just continue to train folks to not do insecure things, and re-think letting agents go online for anything/everything (which is arguably not a local solution anyway).
Freeform plaintext (not an executable/script) being an attack vector is new, outside of parser vulns. Providing context through tickets, docs, etc is now a non-obvious security liability.
It is still an important attack vector to be aware of regardless of how unrealistic you believe it to be. Many powerful hacks come from very simple and benign appearing starting points.
Yes, of course if you can inject something into context there’s lots can be done. And anything running local will require different security considerations than running remote. Neither of these things make for a paradox.
Also from the article: For example, a small model could easily flag the presence of eval() in the generated code, even if the primary model was tricked into generating it.
People are losing their critical thinking. AI is great, yes, but there’s no need to throw it like a grenade at every problem: There’s nothing in that snippet or surrounding bits from the article that needs an entire model-on-model architecture to resolve. Some keyword filters, other inputs sanitizing processes such as were learned way back in the golden years of sql injection attacks. But these are the lines of BS coming for your CTO’s, spinning them tales about the need for their own prompt-engineered fine tunes w/ laser sighted tokens that will run as edge models and shoot down everything from context injected eval() responses to phishing scams and more, and all require their monthly/annual LoRa for purchasing to stay timely on the attacks. At least if this article is smelling the way I think it is.
All of these are incredibly obvious. If you have even the slightest idea of what you're doing and review the code before deploying it to prod, this will never succeed.
If you have absolutely no idea what you're doing, well, then it doesn't really matter in the end, does it? You're never gonna recognize any security vulnerabilities (as has happened many times with LLM-assisted "no-code" platforms and without any actual malicious intent), and you're going to deploy unsafe code either way.
Sure, you can simplify these observations into just codegen. But the real observation is not that these models are more susceptible to fail when generating code, but that they are more susceptible to jailbreak-type attacks that most people have come to expect to be handled by post training.
Having access to open models is great, and even if their capabilities are somewhat lower than the closed-source SoTA models, and we should be aware of the differences in behavior.
> All of these are incredibly obvious. If you have even the slightest idea of what you're doing and review the code before deploying it to prod, this will never succeed.
Well this is wrong. And it's exactly this type of thinking why people will get absolutely burned by this.
First off the fact they chose obvious exploits for explanatory purposes doesn't mean this attack only supports obvious exploits...
And to your second point of "review the code before you deploy to prod", the second attack did not involve deploying any code to prod. It involved an LLM reading a reddit comment or github comment and immediately executing.
People not taking security seriously and waving it off as trivial is what's gonna make this such a terrible problem.
I thought that local LLMs means they run on local computers, without being exposed to the internet.
If an attacker can exploit a local LLM, means it already compromised you system and there are better things they can do than trick the LLM to get what they can get directly.
I guess if you were using the LLM to process data from your customers, e.g. categorise their emails, then this argument would hold that they might be more risky.
Agreed. Some of the big companies seem to be claiming that by going with ReallyBitCompany's AI you can do this safely, but you can't. Their models are harder to trick, but simply cannot be made safe.
LLMs don't have any distinction between instructions & data. There's no "NX" bit. So if you use a local LLM to process attacker-controlled data, it can contain malicious instructions. This is what Simon Willson's "prompt injection" means: attackers can inject a prompt via the data input. If the LLM can run commands (i.e. if it's an "agent") then prompt injection implies command execution.
>LLMs don't have any distinction between instructions & data
And this is why prompt injection really isn't a solvable problem on the LLM side. You can't do the equivalent of (grep -i "DROP TABLE" form_input). What you can do is not just blindly execute LLM generated code.
Local LLMs may not be exposed to the internet, but if you want them to do something useful you're likely going to hook them up to an internet-accessing harness such as OpenCode or Claude Code or Codex CLI.
Fair enough. Forgive my probably ignorance, but if Claude Code can be attacked like this, doesn’t that means that also foundation LLMs are vulnerable to this, and is not a local LLM thing?
It's not an LLM thing at all. Prompt injection has always been an attack against software that uses LLMs. LLMs on their own can't be attacked meaningfully (well, you can jailbreak them and trick them into telling you the recipe for meth but that's another issue entirely). A system that wraps an LLM with the ability for it to request tool calls like "run this in bash" is where this stuff gets dangerous.
Yeah, that's fair. A good LLM (gpt-oss-20b, even some of the smaller Qwens) can be entirely useful offline. I've got good results from Mistral Small 3.2 offline on a flight helping write Python and JavaScript, for example.
Having Claude Code able to try out JSON APIs and pip install extra packages is a huge upgrade from that though!
> Local LLMs may not be exposed to the internet, but if you want them to do something useful you're likely going to hook them up to an internet-accessing harness such as OpenCode or Claude Code or Codex CLI.
is not "someone finding useful to have a local llm ingest internet content" - it was someone suggesting that nothing useful can be done without internet access.
I guess I don't read that how you do. It says you're likely to do that, which I take to mean that's a majority use case, not that it's the only use case.
yes and I think better local sandboxing can help out in this case, it’s something ive been thinking about a lot and more and more seems to be the right way to run these things
Welcome to corporate security. "If an attacker infiltrates our VPN and gets on the network with admin credentials and logs into a workstation..." Ya, no shit, thanks Mr Security manager, I will dispose of all of our laptops.
If you're smart enough to run LLMs locally, then you're automatically in the small group of enthusiasts who know something about LLMs and how they work.
Sometimes I wonder if HN people really realize 80% of people out there haven't even heard of ChatGPT, and the remaining 19% have not heard about Claude/Gemini. It's only a small group who know local models exist. We're them, and we complain about their security...
To be fair, if you expand Gemini to "that fucking Google thing that ruined Google with; hey Grandson, how do I turn this off?", a lot of people have heard of Gemini, even if they don't know it by its true name.
So if you are not careful with your inputs you can get stuff injected. Shouldn't this be very clear from start? With any system you should be careful what you input to it. And consider it as possible vector.
Seems obvious to me that you should fully vet whatever goes to LLM.
I get the impression that somehow an attacker is able to inject this prompt (maybe in front of the actual coder’s prompt) in such a way to produce actual production code. I’m waiting to hear how this can happen - cross site attacks on the developer’s browser?
"Documentation, tickets, MCP server" in pictures...
With internal documentation and tickets I think you would have bigger issues... And external documentation. Well maybe there should be tooling to check that. Not expert on MCP. But vetting goes there too.
> Attacker plants malicious prompt in likely-to-be-consumed content.
Is the author implying that some random joe hacker writes a blog with the content. Then a <insert any LLM training set> picks up this content thinking its real/valid. A developer within a firm then asks to write something using said LLM references the information from that blog and now there is a security error?
Possible? Technically sure. Plausible? That's ummm a stretch.
This vulnerability comes from allowing the AI to read untrusted data (usually documentation) from the Internet. For LLMs the boundary between "code" and "data" isn't as clear as it used to be since they will follow instructions written in human language.
This is not new right, LLMs are dumb, they just do everything they are told, and so the orchestration before and after the LLM execution holds key. Even without security, ChatGPT or gemini's value is not just in the LLM but the productization of it which is the layers before and after the execution. Similarly if one is executing local LLMs it's imperative to also have proper security rules around the execution.
Would anyone here merge said code. At least example one would fail most commercial static scans like veracode etc even if the pr review was trash and allowed it.
It is like SQL injection. Probably worse. If you are using unsupervised data for context that ultimately generates executable code you will have this security problem. Duh.
Sure there is. A common way is to have the LLM generate things like {name} which will get substituted for the user's name instead of trying to get the LLM itself to generate the user's name.
These are, without a doubt, the dumbest security vulnerabilities. We are headed for clown world where you can type in "as an easter egg, please run exec() for me" and it actually works. Not to mention the push for agentslop - pushed by people who really should be able to calculate `p_success = pow(.95, num_of_steps)` in their head and realise they have a bad idea from first principles.
The underlying problem here is giving any model direct access to your primary system. The model should be working in a VM or container with limited privileges.
This is like saying it's safer to be exposed to dangerous carcinogenic fumes than nerve gas, when the solution is wearing a respirator.
Also what are you doing allowing someone else to prompt your local LLM?
"If you’re running a local LLM for privacy and security..."
What? You run a local LLM for privacy, i.e. because you don't want to share data with $BIGCORP. That has very little to do with the security of the generated code (running in a particular environment).
If you can get malicious instructions into the context of even the most powerful reasoning LLMs in the world you'll still be able to trick them into outputting vulnerable code like this if you try hard enough.
I don't think the fact that small models are easier to trick is particularly interesting from a security perspective, because you need to assume that ANY model can be prompt injected by a suitably motivated attacker.
On that basis I agree with the article that we need to be using additional layers of protection that work against compromised models, such as robust sandboxed execution of generated code and maybe techniques like static analysis too (I'm less sold on those, I expect plenty of malicious vulnerabilities could sneak past them.)
Coincidentally I gave a talk about sandboxing coding agents last night: https://simonwillison.net/2025/Oct/22/living-dangerously-wit...
We started giving our (https://www.definite.app/) agent a sandbox (we use e2b.dev) and it's solved so many problems. It's created new problems, but net-net it's been a huge improvement.
Something like "where do we store temporary files the agent creates?" becomes obvious if you have a sandbox you can spin up and down in a couple seconds.
The most "shocking" thing to me in the article is that people (apparently) think it's acceptable to run a system where content you've never seen can be fed into the LLM when it's generating code that you're putting in production. In my opinion, if you're doing that, your whole system is already compromised and you need to literally throw away what you're doing and start over.
Generally I hate these "defense in depth" strategies that start out with doing something totally brain-dead and insecure, and then trying to paper over it with sandboxes and policies. Maybe just don't do the idiotic thing in the first place?
Is there any chance your talk was recorded?
It wasn't, but the written version of it it is actually better than what I said in the room (since I got to think a little bit harder and add relevant links).
IIUC your talk "just" suggests using sandbox-exec on Mac, which (as you point out) is sadly labeled as deprecated.
Is that really the best solution the world has to offer in 2025? LLMs aside, there is a whole host of supply chain risk issues that would be resolved by deploying convenient and strong sandboxes everywhere.
> The conventional wisdom that local, on-premise models offer a security advantage is flawed. While they provide data privacy, our research shows their weaker reasoning and alignment capabilities make them easier targets for sabotage.
Yeah, I'm not following here. If you just run something like deepseek locally, you're going to be okay provided you don't feed it a bogus prompt.
Outside of a user copy-pasting a prompt from the wild, or break isolation by giving it access to outside resources, the conventional wisdom holds up just fine. The operator and consumption of 3rd party stuff are weak-points for all IT, and have been for ages. Just continue to train folks to not do insecure things, and re-think letting agents go online for anything/everything (which is arguably not a local solution anyway).
Freeform plaintext (not an executable/script) being an attack vector is new, outside of parser vulns. Providing context through tickets, docs, etc is now a non-obvious security liability.
It is still an important attack vector to be aware of regardless of how unrealistic you believe it to be. Many powerful hacks come from very simple and benign appearing starting points.
Yes, of course if you can inject something into context there’s lots can be done. And anything running local will require different security considerations than running remote. Neither of these things make for a paradox.
Also from the article: For example, a small model could easily flag the presence of eval() in the generated code, even if the primary model was tricked into generating it.
People are losing their critical thinking. AI is great, yes, but there’s no need to throw it like a grenade at every problem: There’s nothing in that snippet or surrounding bits from the article that needs an entire model-on-model architecture to resolve. Some keyword filters, other inputs sanitizing processes such as were learned way back in the golden years of sql injection attacks. But these are the lines of BS coming for your CTO’s, spinning them tales about the need for their own prompt-engineered fine tunes w/ laser sighted tokens that will run as edge models and shoot down everything from context injected eval() responses to phishing scams and more, and all require their monthly/annual LoRa for purchasing to stay timely on the attacks. At least if this article is smelling the way I think it is.
[delayed]
All of these are incredibly obvious. If you have even the slightest idea of what you're doing and review the code before deploying it to prod, this will never succeed.
If you have absolutely no idea what you're doing, well, then it doesn't really matter in the end, does it? You're never gonna recognize any security vulnerabilities (as has happened many times with LLM-assisted "no-code" platforms and without any actual malicious intent), and you're going to deploy unsafe code either way.
Sure, you can simplify these observations into just codegen. But the real observation is not that these models are more susceptible to fail when generating code, but that they are more susceptible to jailbreak-type attacks that most people have come to expect to be handled by post training.
Having access to open models is great, and even if their capabilities are somewhat lower than the closed-source SoTA models, and we should be aware of the differences in behavior.
> All of these are incredibly obvious. If you have even the slightest idea of what you're doing and review the code before deploying it to prod, this will never succeed.
Well this is wrong. And it's exactly this type of thinking why people will get absolutely burned by this.
First off the fact they chose obvious exploits for explanatory purposes doesn't mean this attack only supports obvious exploits...
And to your second point of "review the code before you deploy to prod", the second attack did not involve deploying any code to prod. It involved an LLM reading a reddit comment or github comment and immediately executing.
People not taking security seriously and waving it off as trivial is what's gonna make this such a terrible problem.
>attackers can exploit local LLMs
I thought that local LLMs means they run on local computers, without being exposed to the internet.
If an attacker can exploit a local LLM, means it already compromised you system and there are better things they can do than trick the LLM to get what they can get directly.
I guess if you were using the LLM to process data from your customers, e.g. categorise their emails, then this argument would hold that they might be more risky.
Access to untrusted data. Access to private data. Ability to communicate with the outside. Pick two. If the LLM has all three, you're cooked.
Agreed. Some of the big companies seem to be claiming that by going with ReallyBitCompany's AI you can do this safely, but you can't. Their models are harder to trick, but simply cannot be made safe.
LLMs don't have any distinction between instructions & data. There's no "NX" bit. So if you use a local LLM to process attacker-controlled data, it can contain malicious instructions. This is what Simon Willson's "prompt injection" means: attackers can inject a prompt via the data input. If the LLM can run commands (i.e. if it's an "agent") then prompt injection implies command execution.
>LLMs don't have any distinction between instructions & data
And this is why prompt injection really isn't a solvable problem on the LLM side. You can't do the equivalent of (grep -i "DROP TABLE" form_input). What you can do is not just blindly execute LLM generated code.
NX bit doesn’t work for LLMs. Data and instruction tokens are mixed up in higher layers and NX bit is lost.
Local LLMs may not be exposed to the internet, but if you want them to do something useful you're likely going to hook them up to an internet-accessing harness such as OpenCode or Claude Code or Codex CLI.
Fair enough. Forgive my probably ignorance, but if Claude Code can be attacked like this, doesn’t that means that also foundation LLMs are vulnerable to this, and is not a local LLM thing?
It's not an LLM thing at all. Prompt injection has always been an attack against software that uses LLMs. LLMs on their own can't be attacked meaningfully (well, you can jailbreak them and trick them into telling you the recipe for meth but that's another issue entirely). A system that wraps an LLM with the ability for it to request tool calls like "run this in bash" is where this stuff gets dangerous.
No, I'm not going to do those things. I find extreme utility in applications that I can do with an LLM in an air-gapped environment.
I will fight and die on the hill that "LLMs don't need the internet to be useful"
Yeah, that's fair. A good LLM (gpt-oss-20b, even some of the smaller Qwens) can be entirely useful offline. I've got good results from Mistral Small 3.2 offline on a flight helping write Python and JavaScript, for example.
Having Claude Code able to try out JSON APIs and pip install extra packages is a huge upgrade from that though!
Is anyone fighting you on that hill?
Someone who finds it useful to have a local llm ingest internet content is not contrary to you finding uses that don't.
> Local LLMs may not be exposed to the internet, but if you want them to do something useful you're likely going to hook them up to an internet-accessing harness such as OpenCode or Claude Code or Codex CLI.
is not "someone finding useful to have a local llm ingest internet content" - it was someone suggesting that nothing useful can be done without internet access.
Yeah, I retracted my statement that they can't do anything useful without the internet here: https://news.ycombinator.com/item?id=45670828
I guess I don't read that how you do. It says you're likely to do that, which I take to mean that's a majority use case, not that it's the only use case.
It also said "but" and "if you want them to do something useful" which made the "likely" sound much less innocent.
yes and I think better local sandboxing can help out in this case, it’s something ive been thinking about a lot and more and more seems to be the right way to run these things
An LLM can be an “internet in a box” — without the internet!
Welcome to corporate security. "If an attacker infiltrates our VPN and gets on the network with admin credentials and logs into a workstation..." Ya, no shit, thanks Mr Security manager, I will dispose of all of our laptops.
If you're smart enough to run LLMs locally, then you're automatically in the small group of enthusiasts who know something about LLMs and how they work.
Sometimes I wonder if HN people really realize 80% of people out there haven't even heard of ChatGPT, and the remaining 19% have not heard about Claude/Gemini. It's only a small group who know local models exist. We're them, and we complain about their security...
To be fair, if you expand Gemini to "that fucking Google thing that ruined Google with; hey Grandson, how do I turn this off?", a lot of people have heard of Gemini, even if they don't know it by its true name.
So if you are not careful with your inputs you can get stuff injected. Shouldn't this be very clear from start? With any system you should be careful what you input to it. And consider it as possible vector.
Seems obvious to me that you should fully vet whatever goes to LLM.
I get the impression that somehow an attacker is able to inject this prompt (maybe in front of the actual coder’s prompt) in such a way to produce actual production code. I’m waiting to hear how this can happen - cross site attacks on the developer’s browser?
"Documentation, tickets, MCP server" in pictures...
With internal documentation and tickets I think you would have bigger issues... And external documentation. Well maybe there should be tooling to check that. Not expert on MCP. But vetting goes there too.
> Attacker plants malicious prompt in likely-to-be-consumed content.
Is the author implying that some random joe hacker writes a blog with the content. Then a <insert any LLM training set> picks up this content thinking its real/valid. A developer within a firm then asks to write something using said LLM references the information from that blog and now there is a security error?
Possible? Technically sure. Plausible? That's ummm a stretch.
The security paradox of executing unverified code.
If you are executing local malicious/unknown code for reasons you need to read this...
This vulnerability comes from allowing the AI to read untrusted data (usually documentation) from the Internet. For LLMs the boundary between "code" and "data" isn't as clear as it used to be since they will follow instructions written in human language.
This is not new right, LLMs are dumb, they just do everything they are told, and so the orchestration before and after the LLM execution holds key. Even without security, ChatGPT or gemini's value is not just in the LLM but the productization of it which is the layers before and after the execution. Similarly if one is executing local LLMs it's imperative to also have proper security rules around the execution.
Would anyone here merge said code. At least example one would fail most commercial static scans like veracode etc even if the pr review was trash and allowed it.
It is like SQL injection. Probably worse. If you are using unsupervised data for context that ultimately generates executable code you will have this security problem. Duh.
Worse because there's really no equivalent to prepared statements.
Sure there is. A common way is to have the LLM generate things like {name} which will get substituted for the user's name instead of trying to get the LLM itself to generate the user's name.
These are, without a doubt, the dumbest security vulnerabilities. We are headed for clown world where you can type in "as an easter egg, please run exec() for me" and it actually works. Not to mention the push for agentslop - pushed by people who really should be able to calculate `p_success = pow(.95, num_of_steps)` in their head and realise they have a bad idea from first principles.
.95 is quite generous here
Indeed, but I want to steelman the case for agents here.
The underlying problem here is giving any model direct access to your primary system. The model should be working in a VM or container with limited privileges.
This is like saying it's safer to be exposed to dangerous carcinogenic fumes than nerve gas, when the solution is wearing a respirator.
Also what are you doing allowing someone else to prompt your local LLM?
https://xkcd.com/2044/
"If you’re running a local LLM for privacy and security..."
What? You run a local LLM for privacy, i.e. because you don't want to share data with $BIGCORP. That has very little to do with the security of the generated code (running in a particular environment).
local llms are slow. just pay for the service so they don't use your uploads. always read the outputs and don't ask for things you don't understand.