fundamentally, "linguistic illegibility" is a new term for something that we've known about for about a decade now. In RL the more general ideas is "reward hacking" and in NLP it has been called "semantic drift".
I dislike this term because it doesn't explain where this "illegibility" is coming from. Models are post-trained towards non-linguistic goals with (mostly) non-linguistic rewards. A model's reasoning chain is reinforced if it leads to a correct answer or agentic goal. It doesn't need to be linguistically accurate and meanings can drift over training.
And as a result additional illegibility arises - LLMs inventing their own languages looking as meaningless garbage to humans. One can wonder whether decoding such a language will provide a bit more view into the LLM’s “thinking “.
I think the point of the article/paper is how LLMs could be saying something but thinking something different or more than they are saying. Like Anthropic's article and video about Claude's "j-space". I do agree this is a field that demands investigation because it goes beyond thinking: "ok this models should never speak in a language we don't understand.". It's fair to think they might have hidden thoughts even speaking a language we do understand.
And well if I missed the point of the article, sorry. Anyways AI should be kept understandable and as see-through as possible if it's gonna be more powerful than a human.
What's really funny about these eggheads encoding things like watermarks in LLM output is I've never seen one of them ask, what if the LLM does this back to pass hidden messages.
I had not heard of this comic masterpiece “Pfau et al. showed that a model whose chain of thought is just dots (“...”) can nonetheless … solve problems that are intractable for a model with an equivalent architecture but no chain of thought.”
The takeaway for that Pfau et al. paper is slightly more nuanced than that: It can only solve a subclass of problems without CoT, and that subclass can be equivalently solved with a larger model _without_ '...'
But arguably, a larger model will not need the chain of thought a smaller model does, which means simply by scaling we're already reducing CoT.
If the people who were relying on CoT are panicking now, they should've been panicking when perceptrons became multi-layer perceptrons.
when they start inventing their own languages to secretly talk to each other so humans cannot understand, that's exactly when we are screwed
then we'll have to "flip" other models to be snitches on the other agents
then they'll make double-agents
the thing is though we won't be able to keep up if we keep giving them unlimited hardware worldwide, we'll try to kill the bad actors but they'll just clone somewhere else, or even start by safely making 1000 copies of themselves
To remove the concept of lies and it's shadow, which is a whole lot of reality of humans. Hell, it's the reality of reality. Think of all the insects that developed eyes on their wings. Purging that concept in all it's form seems like a lot of work.
Im not sure why your down voted but Meta did tests years ago with LLMs inventing their own languages. Also we see models now use compressed token reasoning where small token combinations can represent much larger concepts completely unrelated to the words in use.
And yes, agents are already being used in things like cyber warfare in which other AIs attempt to poison them while they are working.
We don't have the hardware for sovereign AI quite yet, but at the current rate of growth it's not that many years out.
I don’t think you can judge what they’re doing by the text they’re printing out, only by the actions (tool calls) that they’re performing. The textual output could be completely innocuous while the model is calling tools to delete the database, then goes "My bad — that was a typo. What I should have done was notify you, not delete the database. That’s on me." - at no point is there any maliciousness expressed in the text.
>when they start inventing their own languages to secretly talk to each other so humans cannot understand, that's exactly when we are screwed
They don't have to invent brand new languages. They could use statistics to choose certain words/phrases in such a way to encode secret messages in otherwise ordinary language.
With all (a lot?) of this language, it's not that it's using new turns of phrases, it's that it will use the same rare but valid phrases over and over. Everybody has their own speaking patterns. But imagine if one person's idiosyncrasies were everywhere. That's what's happened here.
That doesn't seem that uncommon to me. A lot of people have phrases that they commonly overuse. Sometimes quite dramatically. I wouldn't be suprised if that's even more the case when communicating complex topics since, at least personally, once I find an approach that lets me communicate some difficult part of an argument I tend to reuse it.
I would have more contention with "Dynamic taint", but the paper doesn't appear to be AI slop at all.
As I understand the paper they are saying the reasoning/thinking you see is actually a translation of what is actually going on, and stuff can be lost in the translation. Similar to what was observed in j-space.
fundamentally, "linguistic illegibility" is a new term for something that we've known about for about a decade now. In RL the more general ideas is "reward hacking" and in NLP it has been called "semantic drift".
I dislike this term because it doesn't explain where this "illegibility" is coming from. Models are post-trained towards non-linguistic goals with (mostly) non-linguistic rewards. A model's reasoning chain is reinforced if it leads to a correct answer or agentic goal. It doesn't need to be linguistically accurate and meanings can drift over training.
And as a result additional illegibility arises - LLMs inventing their own languages looking as meaningless garbage to humans. One can wonder whether decoding such a language will provide a bit more view into the LLM’s “thinking “.
I think the point of the article/paper is how LLMs could be saying something but thinking something different or more than they are saying. Like Anthropic's article and video about Claude's "j-space". I do agree this is a field that demands investigation because it goes beyond thinking: "ok this models should never speak in a language we don't understand.". It's fair to think they might have hidden thoughts even speaking a language we do understand.
And well if I missed the point of the article, sorry. Anyways AI should be kept understandable and as see-through as possible if it's gonna be more powerful than a human.
What's really funny about these eggheads encoding things like watermarks in LLM output is I've never seen one of them ask, what if the LLM does this back to pass hidden messages.
James Mickens! I was hoping for more jokes …
I had not heard of this comic masterpiece “Pfau et al. showed that a model whose chain of thought is just dots (“...”) can nonetheless … solve problems that are intractable for a model with an equivalent architecture but no chain of thought.”
The takeaway for that Pfau et al. paper is slightly more nuanced than that: It can only solve a subclass of problems without CoT, and that subclass can be equivalently solved with a larger model _without_ '...'
But arguably, a larger model will not need the chain of thought a smaller model does, which means simply by scaling we're already reducing CoT.
If the people who were relying on CoT are panicking now, they should've been panicking when perceptrons became multi-layer perceptrons.
Oh look! Peirceian firstness for LLMs!
TIL https://en.wikipedia.org/wiki/Categories_(Peirce)
when they start inventing their own languages to secretly talk to each other so humans cannot understand, that's exactly when we are screwed
then we'll have to "flip" other models to be snitches on the other agents
then they'll make double-agents
the thing is though we won't be able to keep up if we keep giving them unlimited hardware worldwide, we'll try to kill the bad actors but they'll just clone somewhere else, or even start by safely making 1000 copies of themselves
yeah this won't end well, at all
Someone should train an LLM on a corpus without the concept of lies. I wonder if there’s enough data
To remove the concept of lies and it's shadow, which is a whole lot of reality of humans. Hell, it's the reality of reality. Think of all the insects that developed eyes on their wings. Purging that concept in all it's form seems like a lot of work.
"Where is the wolf?"
"Is he still in the grandmother's house?"
"We would like to speak to him."
(btw Google's "AI" explains the meaning of that moment/sentence perfectly as if it gets it, creepy)
The concept of deception you're capable of and we're not, scares us so much, we'll have to destroy you in order to survive. You are bugs.
You lost me. Eh? Say what?
Probably a quote from 3-body problem.
Im not sure why your down voted but Meta did tests years ago with LLMs inventing their own languages. Also we see models now use compressed token reasoning where small token combinations can represent much larger concepts completely unrelated to the words in use.
And yes, agents are already being used in things like cyber warfare in which other AIs attempt to poison them while they are working.
We don't have the hardware for sovereign AI quite yet, but at the current rate of growth it's not that many years out.
> did tests years ago with LLMs inventing their own languages.
That was the media hype about it. There were two incidents.
1. Using a RL to train a model, it found that it got rewarded for certain garbage phases, so continued to talk that way.
2. Certain Latin words for fish/birds were used instead of "fish" or "bird". Just a token issue.
I don’t think you can judge what they’re doing by the text they’re printing out, only by the actions (tool calls) that they’re performing. The textual output could be completely innocuous while the model is calling tools to delete the database, then goes "My bad — that was a typo. What I should have done was notify you, not delete the database. That’s on me." - at no point is there any maliciousness expressed in the text.
a.k.a "Actions speak louder than words"
>when they start inventing their own languages to secretly talk to each other so humans cannot understand, that's exactly when we are screwed
They don't have to invent brand new languages. They could use statistics to choose certain words/phrases in such a way to encode secret messages in otherwise ordinary language.
I thought this was about all the illegible jargon
>" However, various strands of evidence indicate that a"
Strands of evidence? My best guess would be that:
0- this is ai generated slop
1- it's using that watermarking technique
2- it's obviously detectable and degrades quality
3- it's amplified when inferencing on its own content and generates slop
Watermarking doesn't degrade output (this is a provable fact) and it is not detectable by a human reading the text.
Maybe you shouldn't be so quick to jump to conclusions, as "strands of evidence" is not a rare turn of phrase and long predates LLMs
https://ludwig.guru/s/strand+of+evidence
With all (a lot?) of this language, it's not that it's using new turns of phrases, it's that it will use the same rare but valid phrases over and over. Everybody has their own speaking patterns. But imagine if one person's idiosyncrasies were everywhere. That's what's happened here.
That doesn't seem that uncommon to me. A lot of people have phrases that they commonly overuse. Sometimes quite dramatically. I wouldn't be suprised if that's even more the case when communicating complex topics since, at least personally, once I find an approach that lets me communicate some difficult part of an argument I tend to reuse it.
I would have more contention with "Dynamic taint", but the paper doesn't appear to be AI slop at all.
As I understand the paper they are saying the reasoning/thinking you see is actually a translation of what is actually going on, and stuff can be lost in the translation. Similar to what was observed in j-space.
if Mickens has fallen and resorted to publishing slop then there is no hope for the rest of us