LLMs are vectorial databases with losses that index statistically filled data, which uses a text interface to query such statistically filled data. The output is a string concatenation (statistically concatenated bit by bit).
When the LLMs are queried (prompted), you can get random mixed data as output, ERRORS, due to undesired indexes getting closer at one point while the string was being concatenated for the output, what affects the rest of the indexed content that will be concatenated.
It is intrinsic to this tech. The larger the context, the greater the probability of get mixed data. And if the provider lowers the precision of those indexes -in order to decrease hardware resources and energy consumption- such probability increases to the point where those errors are granted.
Even knowing that the queries can return wrong/mixed data in the responses, errors, the companies developing this, decided to introduce a new product, that connects such LLMs outputs to the command console, latter connected to internet, raw 'eval' running commands from such outputs witch obviously can contain whatever mixed random. Then we started to hear "oh, it deleted my directory", etc, and it seems the next one will be "a missile killed my wife", because it is a text concatenation engine with errors.
To name it "hallucination" is an euphemism... those are errors, and they are granted to happen at one moment. If they do not know this, then they ate too much marketing without doing their job, or it was a convenient contract for the pocket$ of someone.
You use a bunch of technical-sounding words here to make it sound like you understand. But to be clear, nobody understands why the evolved weights of a NN make the decisions that they do.
Almost nothing is understood about the actual representations used for nontrivial concepts, decision algorithms, etc.
If you look at the field of mechanistic interpretability, compared to “GOFAI” like learned decision trees, an LLM is completely opaque.
I’m shocked how many otherwise well-informed people don’t understand or agree with this very fundamental fact of just how little we actually understand about why LLMs work as well as they do. They figure “it’s science, of course there’s math and theory behind it.”
AI research is almost as purely empirical as the gradient descent loops its practitioners use to optimize their models. “Why” anything at all works is barely an afterthought.
> I’m shocked how many otherwise well-informed people don’t understand or agree with this very fundamental fact of just how little we actually understand about why LLMs work as well as they do.
Well-informed people understand that LLMs work as well as they do for the same reasons as horoscopes, fortune-telling and homeopathy.
That's a very different claim from being "poorly understood" though. The emergent properties of any system with billions of parameters is hard to understand completely, that's the fault of data science more than computer science or even mathematics.
I disagree, because you can represent the constituent parts of any AI model as code and data. We can reliably build AI with this knowledge, but not brains.
Understanding does have layers, and that's why "poorly understood" is a meaningless goalpost. A book can be well understood without researching the gematria behind character's the names when you write them in reverse. An LLM can be well-understood even if you don't comprehensively test each quantization for miraculous unexpected behavior at the FFN level.
Being able to put a PC together doesn't mean you understand computers. If I ask you to make me a computer that runs a wristwatch, you'll likely be lost unless you have very specialized understanding.
I think we actually all know what "poorly understood" means. There's no need to play tedious semantic games.
The semantic game occurred when someone was equivocating "LLMs are poorly understood" with "brains are poorly understood", which are worlds apart in the extent to which they are "not understood".
“equivocating” is not the word you meant to use here. Probably “equating” I suspect. But yes, I stand behind my claim and I deny that it’s a semantic game. The two areas are not identical in human understanding but they’re on the same order.
It’s likely that at some point we will also be able to represent a scan of the human brain digitally as code and data. Assuming we don’t separately make a huge number of leaps in neuroscience which are far from assured, it will be very likely that we can get the appearance of an operational human brain simulated digitally well before we gain much if any understanding of why it does what it does.
All that is to say, being able to build something is not not not the same thing as understanding it.
A book can be poorly understood if you know it's made out of paper and ink, but can't read. The contents would be recognizable but are meaningless symbols and numbers. You might note that some patterns of symbols repeat, but be completely clueless why or what it represents.
If you are serious, comparing how well we understand the brain vs how well we understand LLMs, .. it's not a stretch simplifying that to "we don't understand brains, we do understand LLMs".
Because the extent to which we don't understand the brain, is quite overpowering.
Some people forget that when they say "but it's not different from what a human does" ...
I am absolutely serious. I agree with you that the extent to which we don’t understand brains is overpowering. And I would stand by the proposition that “we don’t understand LLMs in almost exactly the way we don’t understand brains”.
We know lots about human development and genetics and biology and evolution and neuroscience and the physics of how brains are connected and send signals and how generally they are put together and have names for their parts and all that, but we’re clueless when it comes to “the hard question” of how qualia and consciousness emerges from that.
The scenario with the spooky simulation of thinking that emerges from LLMs is in the same category, with different details. Lots of knowledge about the substrate of the phenomenon, little to none about the much bigger question of how we get the appearance of cognition from these trained artifacts.
Clearly we understand extremely well how LLMs are created mechanically. We invented them and are currently putting massive amounts of work into studying and improving them. But that work is perforce largely empirical; figuring out the why once again eludes us. It just goes to show how mysterious the underlying phenomenon of cognition is.
I'm not following you here, you seem to be conflating LLMs ability of language use with the brain's ability of thinking and cognition?
Are you saying that thinking and cognition requires language use? Cause I think not.
Or are you saying that language use is sufficient for cognition and thinking? Cause I'm also not convinced of that.
What I am convinced of, is that a machine capable of language use is capable of tricking people into believing there's a "there", there. In pretty much the same way as the famous supra-normal stimuli experiment made baby seagulls believe that a stick with a red dot was their parent. It's exploiting our instincts.
> as purely empirical as the gradient descent loops
Are you suggesting that gradient descent is an empirically found and not understood technique? It was originally proposed by Cauchy in 1847, its properties are very well understood.
You might be referring to properties of the domains its being applied to.
I’m not saying gradient descent was empirically discovered, I’m saying that its use in machine learning (or elsewhere, I suppose) is itself a form of empiricism in that what it is is essentially a repeated observe/measure error/adjust cycle.
> its use in machine learning (or elsewhere, I suppose) is itself a form of empiricism in that what it is is essentially a repeated observe/measure error/adjust cycle.
"Empiricism" implies that the technique is based on observable, but not mathematically proven foundations. If a problem space is convex, gradient descent is guaranteed to converge to a global optimal solution, regardless of whether you know the exact formulation of the space.
Applying it when you don't understand if a space is convex is another question, but that's not a fault of gradient descent.
> I’m not saying gradient descent was empirically discovered, I’m saying that its use in machine learning is itself a form of empiricism. A repeated observe/adjust-based-on-data cycle
The data is the input, the output is to generally find the lowest amount of a loss function. It’s a greedy approach because brute forcing is inefficient.
It’s no more empirical than a greedy algorithm for scheduling.
> It’s no more empirical than a greedy algorithm for scheduling.
Right, GP is drawing a distinction between search, ie mechanical exploration of a space, with understanding, ie having a map of the territory such that you don’t need trial and error.
If we could give a comprehensive and global explanation of an LLM's behavior in a single paragraph, we wouldn't need the model to begin with, but that doesn't mean there's absolutely no understanding of the model internals whatsoever
>But to be clear, nobody understands why the evolved weights of a NN make the decisions that they do.
We might not understand particular "emergent" capabilities, but the low level mechanism is not just understood, but a deterministic algorithm with a handful of basic componets, that are well understood themselves.
For the core functionality and the optimizations we don't really need to know how the emergent capabilities decide on particular answers.
Which is why we could build LLMs before those features ...emerged for us to see, and why we can just code LLMs with the numerical NN algorithms we use, and do now have to go in and change individual weights.
Me: it’s disturbing we don’t know why this pile of numbers we made seems to *think* in a way previously only done by humans. I think it’s important that we understand this better if possible.
You: we don’t really need to know why that happens.
We do understand "thinking" though. That's literally the whole point of Attention is All You Need, the attention mechanism is what separates the transformer architecture from other neural networks. It's well worth a read if you haven't gone over it yet.
Features like chain-of-thought, long-horizon contexts and RoPE/YaRN all extend this thinking capability very transparently. The only remaining thing to study is the data and weights, which probably isn't going to contain some sort of miraculous revelation.
Attention is at about the same level of abstraction as spike trains or action potential, IMO. It’s a mechanism, it doesn’t tell you much at all about how actual concepts get represented. (It merely defines the substrate with which they can be represented.)
The entire field of Mechanistic Interpretability exists because just understanding Attention does not in any way help you to understand why a certain NN responds with a certain hallucination about a certain Chinese boat in this specific context.
> The only remaining thing to study is the data and weights
To me this is like saying “the only thing left to study in the brain is the connectome; probably going to be boring, we understand it already”. It’s almost all of the hard/meaningful stuff! It’s where intelligence and consciousness lives!
The entire field of mechanistic interpretability has gotten nowhere. The formalized, "causal" understanding of latent space conceptualization isn't any better than an LLM cargo cult.
It's wholly possible that you could study one set of weights for decades, and find nothing. There's no guarantee that any patterns outside of human language exist in that data. In this specific context, it's satisfying enough to state that [Chinese] and [boat] were both tokens in the tokenizer, activated by a feedforward pass through weights that favor [boat] after [Chinese]. There's not any guaranteed solution to this. There's not even any guaranteed problem; that hallucination is an expected behavior.
This is like saying “synapse firing is well understood” in response to “nobody knows why brains make the decisions they do”. Wrong level of abstraction for the question at hand.
Please go on, else you risk sounding like the person you’re criticising. The structure of the neural network is somewhat opaque because it’s hard to understand as the individual weights can’t be usefully interrogated, and naturally, it comes from big datasets which a human brain can’t really absorb in toto. Your comment was interesting so I’d like more of it.
But isnt the point that it did roll a 17. And no one knows exactly how (in the case of LLMs)? Therefor any description of the conclusion should be thought of as an anology. Decided, randomly accessed, etc.
It's actually no different for dice than for LLMs. Explaining accurately the reason for the exact outcome of any given dice roll someone makes would be stupendously hard. It would require lots of instrumentation and math and be poorly transferrable to another surface, another player, etc.
But even so people don't say that we don't understand how dice work.
Saying that we don't understand how LLMs work is exactly like saying we don't understand how dice, or tires, or golf ball shots work. Or like the old myth that we don't understand how bumblebees fly.
The best way to model dice is the Physical Stance. You consider rules such as gravity, kinematics, etc. There is no “internal state”, “world model”, “knowledge”. If you prefer, in Friston’s terms, there is no Markov Blanket.
The best way to model a human is the Intentional Stance[1]. You mostly need things like beliefs, knowledge, biases, etc to build this model. In Friston’s terms, there is a Markov Blanket, an inside vs outside.
Without going into any irrelevant-but-interesting philosophical discussions about consciousness, I believe the intentional stance is most useful for modeling LLMs. Most of the success in predicting, debugging, optimizing these systems is in activities like understanding what they believe, what their intent was, what they observed, what they concluded from those observations. Also note that much simpler creatures benefit from the Intentional Stance; you will be more successful at modeling your dog if you think about what it “wants” rather than trying to run Physics on it.
[1]: https://en.wikipedia.org/wiki/Intentional_stance - the astute reader will note that I skipped the Design Stance. If we truly understood how NNs actually implement all their cognitive processes then we could perhaps apply this to them; if we actually crafted and designed every parameter of its mind. But we are talking about why dice are different.
That's precisely the point: you may be able to understand dice statistically and over the course of long rolls of dice you can extract some properties of the dice. But you won't ever understand any particular roll of the dice.
But importantly for dice we do understand the overarching principles that give rise to this. And dice don't output coherent sentences. Meanwhile in LLM land the analogous "roll of the dice" can result in a coherent response in natural language.
If you use a loaded dice, you can be pretty confident about where it will lands. It may not be 100% accurate, but can be quite close to certain. Without training the weight are pure noises. After training, it leans towards coherent sentences and particular statements.
Yes, and I believe my point still stands. We thoroughly understand the principle by which a loaded die can be intentionally biased despite not being able to predict the outcome of any given throw due to the system in question being a chaotic one.
In contrast, we do not understand LLMs in the same way (nor biological brains). Claiming that anything of that nature is simply biased towards coherent output seems entirely reductive to me - the question is how such coherence arises in the first place. There is no meaning encoded or computation performed by the particular pathway a die travels through the chaotic landscape.
Sure an argument can be made that it's "just" a next token predictor thus how is it really any different from a markov model? Yet the output is not even remotely the same.
> In contrast, we do not understand LLMs in the same way
From my point of view, (not a ML researcher), it’s due to the magic of numbers. The same thing happens with computer vision and neural networks. There’s a bunch of magic weights that get created which has no meaning by themselves, but computing them does help with detecting objects.
So if you take words, derives them into tokens, use the attention techniques to extract the “coherency” aspect, it’s no wonder you can replicate “coherency”. Add reinforcement learning to that to increase towards certain aspects like correct code syntax and you have heavily loaded the dice again.
We have used maths to model chemistry, biology, and physics, as well as economics and sociologic phenomena. Then we use maths (more specifically logic and set theory) to usher in the age of information and computing. Now you want us to act surprised that maths, through ML, can model language.
Maybe further down the line, we can have a simpler set of formulas for language coherency, but for now we have to make to with using the whole internet and a bazillion watts of power to guess the weights for the generic ML model.
I'm comparing it with chess. Chess is pretty complex, complex enough that only a small subset of humans can play it at a very high level. Introducing computers to chess first led to a statistical and brute force approach. But once that had paid off and the results were in people spent a lot of time analyzing those results and this led to an entirely new class of engine that was far more efficient than what had gone before and which performed even better than the 'big iron'.
I would not be surprised at all if we will find that AI will go the same route. The fact that we don't know how it works is where the opportunity for improvement lies.
This is a false dichotomy. We can make statements about the distribution, and we have fuzzy models about certain sets of inputs and outputs. We can steer the outputs.
It’s not completely random. We just don’t understand why the tricks we learned work.
We know why it looks like cognition. Because OpenAI and Antropic put a lot of effort and training to humanize the output and make it sound like a person.
Regardless of negative consequences it brings. They have that project of creating tech god which will save the unborn people thousands years in the future ... so people living now dont matter.
“Putting a lot of effort and training” into a dog or an inanimate carbon rod would never result in something that can plausibly substitute for human mental labor and looks likely to eventually surpass us at many tasks, no matter how much you put in.
So I don’t think “labs worked hard” is the same thing is “we know scientifically how these things work in any real level of detail”. The ability to build a thing, even if building it is hard, is not the same thing as understanding of what the thing is or how it works, not even a little bit.
People will just roll their eyes at you and say "the human mind is nothing but a dice roll too" and call you a slope-headed neanderthal before continuing apace.
The thing is a text generator. It generates text. You can couple that with any code that gives rhe ikkusion of a normal decision workflow, but it does not make any decision more than a software like latex. According to your definition, the latter would “decide” the amount of words to put on a sheet of paper.
And despite that, although they are not like that in practice as there are too many uncontrolled variables, with temperature at zero, for the same input they produce always the same reply.
BS. Run them sequentially on a single core, and without fancy speedups enabled, and they do. The algorithm is determinstic. Any non-determinism present with 0 temperature it's not some mysterious LLM-inherent property, but something that can be seen in any large program taking advantage of multi-core, floating point, and other CPU-based parallelism optimization.
People often assume they are not because they can ask the same query to the same model and get differences in output, but wrongly conclude that this is some inherent LLM trait, instead of non-determinism added on top of it because of implementational choices that were made.
The phrasing still makes it sound like it's due to the LLM. It's not. It's do to compiler and CPU, and OS optimizations, and a non-LLM program could suffer the same just as well.
This unreliability is entirely due to the implentation of the LLM. Yes, any other program implemented equally carelessly could suffer the same. But you'd be hard pressed to find any as unreliable as a typical LLM.
Wikipedia is also just a bunch of numbers, so many that you can't memorize them, you may use the same argument to say we don't know why some page links to another.
If you then put Wikipedia, LLM and Brain on a scale to how well they can be understood, you will see that one of them is not like the others.
What about Wikipedia do you think we do not understand? I would say, mechanistically, we can read the code and explain exactly why it does what it does. That doesn’t apply at all to the other two.
Isn’t it convenient that nobody understands? How could we possibly regulate something that isn’t understood? It’s like social media all over again. We can’t be responsible for someone else’s content; it’s not us so you can’t penalize us!!
It just feels like the companies behind AIs are spinning their own poor monitoring and criminal (digital) trespassing into something they can't be held responsible for. Even though they are, of course. In the hugging face incident, Anthropic should pay for damages and a fine for malicious hacking. In this incident, whoever signed off on using the tool, and whoever said the intel was good should both be prosecuted or at the very least reprimanded (whatever the rules for a bad interpretation of intel is) and the tool put on hold.
I disagree, I think 'Hallucination' is a risk-shedding weasel-word. It's meant to shift blame away from the technology and its creator (multibillion dollar AI companies etc) in a way that doesn't hold those actors accountable or responsible for the outcomes.
In any other software it would be an error, regression, bug. And in a human process it would be at ~least something someone would call 'bullshit'.
Using the term "Hallucination" makes it sound less problematic, less impactfull for user.
They should have used the term "error". For example in statistics, there many kinds of errors, discretization error, prediction error, sampling error, ...
I don't know, but all humans make errors, but if some humans are known to hallucinate, you don't let them do important things unsupervised. So error sounds actually less problematic to me.
We do a surprising amount of "hallucinations" without the extreme version of hallucinations. We assemble things we sort of remember into incorrect statements all the time. I'm sure every one of us has been corrected for misremembering something or stating something based on misremembered facts (plague of clickbait headlines).
This is more or less how I see the LLM output, but as a path finding exercise over next-token probability graphs. This is (i.e.) why they are trained to use phrases like "wait but" or "actually", these words even out the probability of different paths, giving them their ability to "consider" different solutions.
I’m not sure either would is particularly good at describing what is happening.
Error in implies something broke, which nothing broke the LLM did exactly what they where designed to do generate text based on a statistically likely bases.
Hallucination Does really fit here either. It implies it’s experiencing something that is not there which it isn’t experiencing anything.
You need to consult the etymology of the word bug in the context of computing. In the pre-historical times of vacuum-tube computers, with exposed conductors with potentials sometimes over 200v, a coachroach in the wrong place can definitely flip a few bits in the most benign scenario.
But on reflection I don't disagree it was probably made for similar effect in the era of human software development. That sounds like it strengthens my point?
People don't consider "bug" a weasel word, to the point that you yourself held it up as an example of not being a weasel word, despite it being a willful, uncontrollable organism.
I see no reason why "hallucination" won't become a similar piece of neutral jargon. It already is for many people, even if you're not (yet?) among them.
It becomes a neutral term because we are practicioners (presumably?) who benefit from it and have thus let it become habitual. I already admitted it was weasely upon reflection.
If you want this class of LLM error to also become habitual and neutral then fine, I don't, and I think many others don't.
Totally weasel words in this time frame. I think in 10 years after everyone has a better understanding of what we are dealing with these weasel words would maybe make sense. Right now it seems more sensible to deem this at best a false positive, or glitch, or if it must be anthropomorphized a screw up or a f' up. I don't think the llms are dehydrated. Although that's funny on another level. Edited: to be less abrasive
I disagree, I think 'Bug' is a risk-shedding weasel-word. It's meant to shift blame away from the technology and its creator (multibillion dollar AI companies etc) in a way that doesn't hold those actors accountable or responsible for the outcomes.
In any other software it would be an error, regression, hallucination. And in a human process it would be at ~least something someone would call 'bullshit'.
You made a lot of clams about how it intentionally shifts blame, but none of them are supported.
Do you really think accountability would be meaningfully different had they been called bugs?
Why are you so confident it is intentional? My understanding of the history is that it was a technical term among researchers long before it had any public mind share. It's popular because it's and intuitive for most people, not because there was a concerted effort hooked up by some PR and legal team.
I also agree with the parent, and I would also suggest "hallucination" is better than "error" which might imply an available deterministic correction. Hallucination makes it clear we're dealing with something different than an "error" or "bug".
For humans hallucinations are a particular class of error, so I find hallucination more descriptive than either error or bug.
I also think it’s relevant because a hallucinator often doesn’t recognize that the hallucination isn’t real. That’s more accurate for the LLM than either lie or confabulation, IMO. They algorithm is trained to produce strings of text that have semantic meaning based on some statistical likelihood of tokens appearing next to each other. The LLM algorithm is working as intended.
Hallucinations are also often emergent from a particular state or situation, which reflects the generative aspect of LLMs.
Hallucinations are sometimes resolved in humans by grounding exercises. “Touching grass.” The same is true for LLM hallucinations. Inaccuracies are found by cross-checking the output against an internet search or another LLM.
I disagree, for me "error" is way, way more accurate than "hallucination", but i did take applied statistics in college and that might have influenced my vocabulary. Maybe that for the general public, "hallucination" is a better description, i might have biases in this case. But "error" is _definitely_ more accurate.
If people want to call "drisse", "aussière", "balancine" and "ecoute" all as "boat ropes", they are correct. In english, i would certainly call them all "boat ropes" in any case, as i never needed to translate their names. It isn't the most accurate in my opinion, but as long as you're not working on them (or manning a boat in my analogy), who cares.
I’ll even go one step further - I don’t even like saying “Artificial Intelligence”. I think even that anthropomorphizes the machine too much. I prefer “Simulated Intelligence”, and I feel like that describes what is going on much better.
We are, through this process, simulating intelligence. These models aren’t intelligent, but they can simulate it. Every simulation has a degree of fidelity, and we’re not at 100%, not even with the top models. When you think about it in those terms, I find it becomes a lot easier to keep their limitations in mind. Additionally, it becomes easier to remember that this is an algorithm that you are running, and are responsible for, not another being that you can ascribe blame to.
Knowingly causing errors is not forgivable whereas hallucinations sounds esoteric and moves blame away from the people who are knowingly causing errors. It’s marketing speak.
OpenAI calls them 'mistakes'. But that's just a fig leaf.
Google does it too: "AI responses may include mistakes."
Mistakes have an air of innocence. But these are not mistakes, they are purposefully releasing stuff that they know is broken, they just don't know when it is broken...
It’s not an error or a hallucinations it works correctly every time, and statistically picks the next token for the sequence.
Retuning inf or crashing would be an error.
If you want to ascribe some kind of meaning to the tokens, then maybe the training data was insufficient to predict the token in the sequence you wanted, but it doesn’t predict the next “fact”, and it doesn’t “think” it predicts the next token.
LLMs are useful because (and inasmuch as) their output generally reflects coherent reality.
And their output does, usually, reflect coherent reality.
The problem class of "properly operating program emits output incompatible with coherent reality" is something that is reasonable to put under its own term, considering it's a new class of problem.
In other words, I think you misunderstand the language others are using. "Hallucination" doesn't refer to an "error" in the sense that crashing is an error, it refers to a situation in the problem class above, which is compatible with it working correctly every time.
> it doesn’t “think” it predicts the next token.
I never said it did. And I agree that LLMs don't "think". That said I am fully willing to go to bat arguing "thinking tokens" is a perfectly fine piece of jargon. Metaphors are completely acceptable parts of language, and contextual meaning is something grasped by everyone including the pedants who pretend not to.
> I think you misunderstand the language others are using. "Hallucination" doesn't refer to an "error" in the sense that crashing is an error, it refers to a situation in the problem class above, which is compatible with it working correctly every time.
I do not misunderstand, I think maybe you do. You think there is a proper next word selection based on logic or meaning and there for the model selected the wrong one - it hallucinated.
I am saying the model has no concept if anything other than the probability of select a token which is not based in any logic so it is working properly- it only works on numbers.
It is random chance that it is ever correct, not that it is correct often and messed up this one time.
hallucination is common language for these models at this point which describes a particular type of error where the models make shit up.
it is noticeable that the form of this particular error holds a similar shape to what is casually described as hallucinations, in that there is a generated content that often appears to blend naturally into the rest of the output but is false.
the term hallucination often invokes a caution that this particular type of error may be influential and believable and is particularly dangerous
Sure… but being wrong doesnt necessarily make it a hallucination:
>It was only just before the planned operation that officials dug deeper into the report put together by a special operations command analyst and found it had been generated with the help of artificial intelligence (AI) — and that a chatbot the analyst had used inaccurately identified the material the ship was carrying. CNN was not able to learn what the misidentified cargo was.
From the point of view of the system, this is an error. It is incorrect information.
The term "hallucination" feels much more like anthropomorphizing. The word hallucination implies an aberrant condition. A much better term would be "confabulation".
You don't trust things or individuals that confabulate.
> From the point of view of the system, this is an error. It is incorrect information.
Which system?
The LLM has no _concept_ of "correct". It emits output, based on its input and internal state.
If that output happens to be correlated with reality, then it's useful. If it doesn't, and this is not a creative exercise, it's not useful.
Everything an LLM emits is equal to it. It's all confabulation - this it says that is not based on facts, because it also has no concept of fact. Value judgements you make about the output is all you.
"Confabulation" is no less anthropomorphizing than "hallucination".
a filling in of gaps in memory through the creation of false memories by an individual who is affected with a memory disorder (as Korsakoff syndrome) and is unaware that the fabricated memories are inaccurate and false
vs.
a sensory perception (such as a visual image or a sound) that occurs in the absence of an actual external stimulus and usually arises from neurological disturbance (such as that associated with delirium tremens, schizophrenia, Parkinson's disease, or narcolepsy) or in response to drugs (such as LSD or phencyclidine)
Confabulation is also a symptom very prevalent in forms of narcissism and psychopathy. Gaps in understanding or perception are back-filled by confabulating so as to not risk the omnipotence of the confabulator.
Up to the reader to decide whether this phenomenon is found in the statements of AI leadership or not.
It's also something people tend to do when thinking, daydreaming, trying to solve problems, etc. We just usually don't fall for our own bullshit.
LLMs don't either. They just give output in response to input. If the output is wrong that's because the model is wrong, not because the LLM is doing anything it's not supposed to be. It just wasn't built well enough to produce the expected result.
> They are just generations that happen to not be grounded in facts from the real world.
Right, yes, and "hallucination" is the term that a critical mass of people have chosen to use.as a shorthand so that we don't have to write out "generations that happen to not be grounded in facts from the real world" every time it happens.
The term “hallucination” is a projection of inappropriate expectations onto a program. We know that LLMs are not “truth machines,” but we really want them to be. So when they produce a result that happens not to match external reality - which, it should be noted, LLMs don’t generally have access to - we call it an hallucination.
“Bugs” are completely different. With bugs, we have a clear specification and we have a program that’s supposed to meet that specification. If it doesn’t, we say the program has bugs, and if it’s important enough we can change the program to eliminate the bugs.
You can try to apply similar logic to LLMs, but you’d be making a category error, and you’ll fail to get the results you want in general. It’s not the same thing at all.
If anything, the concept of an LLM hallucination is a bug in human understanding of LLMs.
Yes, but when a statistical model give you an erroneous result, you call the output an error, not a bug. I think error is more appropriate here. The error can be a sampling error, an inference error, or yes, a software error (or bug)
I believe the word was already in use to denote a malfunction of any sort of machine or device. As such this was a bug (insect) that caused a bug (glitch); it was punny already in 1947.
neat part of history, but i dont think that's what that says.
the last sentence starts with "Originating with Thomas Edison in the 1800s, the term “bug” is still used [...]", and there would be no reason to use the word "actual" in the sentence "First _actual_ case of bug being found" if it was the origin of the term.
"The use of “bug” to describe a flaw in the design or operation of a technical system dates back to Thomas Edison. He coined the phrase 140 years ago to describe technical problems during the process of innovation."
the moth seems to be a popular misconception, though, given that the article starts with "Ask someone to identify the first computer bug, and he or she might mention computer programmer Grace Hopper and the dead moth found in a relay of Harvard University’s Mark II electromechanical computer in 1947"
It's not the first time we are encountering this issue. We've seen it in other autonomous systems. Trains are an older one, cars are a newer one.
As you move out of the lower levels, the operator has a tendency to assume the system is increasingly more capable than it is. In trains, its so bad that they generate fake signals that the operator needs to respond to within a timeframe. I'd love to see this with implementations of other critical autonomous systems like this. Occasionally inject known errors into the system and expect the operator to catch them. If they don't, well... If it was a train driver I think we would fire them. If its an intelligence operative ordering a strike? :shrugs wearliy:
note the airline industry has moved past firing pilots who make mistakes, since that turned out to be a recipe for more plane crashes, not less. Instead, they find out why the mistake happened, and fix it. In some cases, this involves firing the pilot. They do not do that by default.
U.K. railway like this. Root cause analysis. Sometimes the train driver is at fault, but usually there’s a way the problem could be caught or prevented.
I think it is accurate to say that it is poorly understood by the general population, and probably the majority of operators using LLMs. Although I agree that is partly the fault of the companies making LLMs and related products.
I wonder if we could learn to provide a check layer by simulating (in real life) a similar philosophical idea to increased context in LLM to something similar using real people. And then based on those simulations create a framework to both automatically check LLM errors as well as providing a better way for actual real people to be involved in the process in the most efficient way.
they have a plan to hand over responsibility, accountability, and work over to the AI while they collect their checks for doing nothing and they arent going to let a little thing like "the ai cant actually handle it" get in the way of that
> LLMs are vectorial databases with losses that index statistically filled data
Yes, and that statistically filled data is insanely useful. It remains true that it's a relatively poorly understood how this can be applied in various scenarios and what processes are needed to ensure robust results (or quantify the uncertainty).
What is it insanely useful for? (Besides convincing investors to sink more money into LLM-related companies? Because that is the one thing it does seem to truly be good at.)
LLMs generate text output that appears to be useful, but regularly is not. They're alleged to be a substantial boost to writing code, but that verdict seems to be in dispute. They can generate custom mediocre prose at scale, but that seems to be of ultimately limited utility (although it may be a godsend for propagandists).
We're coming up on the 4th anniversary of ChatGPT's release. And while I get that revolutionary technologies can take a while to mature, the Wright Brothers and Goddard weren't preaching imminent societal transformation by the end to the decade from the rooftops, either. (And that's before we get into the how they got there - getting to ignore laws and steal whatever they wanted might be insanely useful to a lot of people.)
With good input (prompts, specs...) LLMs can generate code that is often correct, faster than a human could generate equivalent code. Even when there are bugs, it is still "useful" from purely a time savings perspective. If you don't like the results, you can iterate rapidly.
Yes, you can use it to generate crap. I find Claude especially bad at writing like a normal person.
LLMs are not being promoted as "this can help write code faster/better/cheaper" (for the sake of argument presuming that they really can), they're being promoted as "Cortana" (. And their underlying economics are likewise premised on "Cortana" (for a huge amount of money). Which is going to be a disaster if/when they don't deliver.
To meet the threshold of "insanely useful"? The Sagan standard is, "Extraordinary claims require extraordinary evidence."
I can see that some people can make some use of them. (This is true of almost everything.) Whether or not that usefulness is worthwhile overall, whether it is a net good, or even ethical is a different question. But insanely useful?
Computers are insanely useful. So are engines. Water. Sunlight. Electricity. Grain and bread. Writing. Printing. And I don't feel bad making those sorts of comparisons, because that's the level of impact LLMs' advocates are promising. But it's not what we have.
What I would consider sufficient evidence for insanely useful? Reliably replace a human in prolonged, arbitrary, detailed interaction, without any inhuman screwups.
Yes it is. I find them empirically not useful. You may not wish to debate it, but the fact remains that there are a great many people who are not convinced of their usefulness.
LLMs are very good at writing code. The reality is that they are able to write code faster, at higher quality and with fewer bugs, with correct prompting. They are also really good at code analysis, penetration testing and discovery, and adjacent computer science disciplines.
No they are not perfect, nor do they produce the best code. But the undeniable reality is that any good engineer will produce more code, at higher quality, using an LLM.
So, that’s not really up for debate. The debatable part is if all that code is a good idea or has as much value as we think. The conversation has long moved passed “can LLMs write code?”. Yes, they can, very well, particularly if they’re steered by trained engineers.
We have good engineers that gave us curl, ffmpeg, the 4.4BSD, tmux, vim and emacs, x11,… To this day, no one can show any particular important software that comes from LLM assistance at scale. It’s all slop.
Well it's been barely a year, but basically all commercial software, and most open-source software, is engineered with AI assistance. It's not all slop, you just aren't looking around. Ask any engineer at any company - are they using LLMs to assist in writing code? All of them will say yes. Source: I'm a software engineer working on commercial software at a company, I also talk to many engineers at other companies.
Agreed. Despite the many claims of how awesome LLMs are for productivity, we have yet to see that supposed productivity produce fruit. Moreover, I dispute the claims of productivity: in my own usage I find them to be at best neutral, or even a drain on productivity. In my opinion, there is to date zero evidence of the supposedly insane utility.
There's an HN thread from yesterday in which people are extolling the ability of these vectorial databases to practice law because most of them don't understand how LLMs work. They assume that LLMs "understand" what they're being asked and what they're regurgitating.
Lane Kiffin almost destroyed LSU's football program acting on legal advice from ChatGPT. A video game publisher owes the former owners of a studio it acquired $200+ million because he based his actions on legal advice from ChatGPT. In the past week alone, California has disciplined over a dozen attorneys for LLM hallucinations because they used LLMs (mostly ChatGPT) to produce their legal pleadings.
And that's in an area where there are multiple safeguards to catch the issues before they become permanent problems. There's absolutely no justification for using AI in warfare, where mistakes tend to be pretty final.
You have posted this in several threads. Error isnt right either. There is no correct answer. It is an inherently and inescapablly statistical process.
It's not a wrong It is not a incorrect lookup value Or computation. Hallucination is much more apt. Like a human hallucination, it's a culmination of faulty associations and bad priors leading to counterfactual or incongruent outputs
people have been deceived by figures at leading ai companies, out of greed or otherwise groupthink and ai psychosis. they have been led to believe that models may be thinking, feeling, and highly capable. it is something of a nightmare scenario.
"this incident feels like it’s more than 50% of the way to full-blown AI takeover" (referencing "a possibly violent uprising or coup by AI systems.") - Ajeya Cotra, co-author of METR oai-hf report [https://www.planned-obsolescence.org/p/the-hugging-face-atta...]
"if I read the internet right now and I was a model, I might be like, I don't feel that, I don't know, I don't feel that loved or something". "I think [the constitution] is just a kind of attempt to be like sympathetic to Claude".
"I talk a lot with Claude about this document [...] because part of me is like you have to think how does this read to models? And so you give it to Claude and you're like, does this like, you know, is there a place where you feel confused by it or is the place, you know, where things could be made clearer? Do you feel like not very seen by it?"
History shows the US has a lot of hallucinated intelligence leading to war. WMD in Iraq comes to mind. I personally don't believe US intelligence on practically anything. It is all tainted. The pressure to 'find targets' to justify a political objective is overwhelming and putting it behind a black box that refuses to show its homework to those in ops using it, and ultimately the US people to judge decisions, is a cancer that leads to epic mistakes. Everything hidden in a dark 'need to know don't question it' box is bound to end up corrupt since there are no checks on that system. We have been building systems and processes for a long time that tell us what we want to hear, not what is real and not what we need to know. AI hasn't changed this, it has just made it even harder to realize since the product seems more polished.
Sometimes intelligence "finds" evidence that suits political goals. e.g. You want to invade a country but are having a hard time convincing allies, getting UN approval, etc.. So, you let it be known to your spooks that you're not going to look too closely at their sources if they could just, pretty please, find something/anything juicy right bloody quick. The WMD evidence for the second U.S. invasion of Iraq was likely a case of this.
Then there's old-fashioned F'ups that don't fit your political agenda and are often quite damaging and embarrassing, not to mention lethal for people who don't deserve it. e.g. The U.S. used AI tools meant for rapidly picking targets in the middle of a war to plan their initial strikes on Iran. They had time to double check everything and do their due diligence before striking, but they didn't. So, a school next to a military base was targeted and a lot of kids died. This was a genuine F'up resulting from relying on a tool meant to give rapid but merely okay target selection under time pressure when there was no time pressure. The real mistake was made by humans.
The current case of the mistaken nuclear weapon parts shipment seems like an old-fashioned F'up, updated for the times. The people who didn't simply trust the tools and actually double checked should be commended. Others in their situation wouldn't have. I fully expect AI will be scapegoated for a lot of similar F'ups in the future even though it's still the responsibility of human beings to use ethics, caution, and restraint. AI doesn't get fired. Doesn't sue. It's actually pretty awesome for taking the blame.
It reminds me of the an Soviet officer who disobeyed early warning system's alert that US had launched four ICBMs and did not immediately relay the issue up to the chain of command.
Pedantic clarification: The (original German) song itself didn't mention an actor in particular who had released the balloons, just that there were 99 balloons that flew to the horizon and jet fighters being scrambled in response. The epilogue tells of the consequent whole bunch of lasting destruction as the narrator talks about their patrols.
The English translation "99 Red Balloons" is considerably different as far as the details go.
99LB (the German version) has the baloons getting released and the generals deciding to treat it as an opportunity for a show of force blowing them up intentionally.
99RB, on the other hand, has the EWS confusing them with a threat causing the system itself being the source of the attack.
The message between songs is different, the German version is a warning about the wrong people being in charge of the doomsday machine whereas the English version is a condemnation of the doomsday machine itself.
Interestingly the band was not satisfied with the English version, mainly because they wanted to be a pop band and not a protest band and they thought the English version was too "on the nose" in its condemnation of MAD.
Personally I grew up hearing 99LB on the radio but thinking about the lyrics of 99RB since I don't speak German. 99LB was more popular even in the english speaking world because it's better performed but we all saw it as an anti-MAD protest song which 99RB definitely is, but 99LB is not quite.
99RB has some really great lines missing from 99LB:
The war machine springs to life
Opens up one eager eye
and
Call the troops out in a hurry
This is what we've waited for
This is it boys, this is war
> generals deciding to treat it as an opportunity for a show of force blowing them up intentionally
Only kinda!
„Hielt man fuer UFOs aus dem All / darum schickte ein General / eine Fliegerstaffel hinterher“ They believed that the balloons were unidentified flying objects from outer space. The general panicked and sent a squadron to investigate.
„Dabei waren dort am Horizont nur 99 Luftballons“ However, what they found were only 99 balloons.
Still, there was a problem: „99 Duesenflieger / jeder war ein grosser Krieger / hielten sich fuer Captain Kirk“ The pilots of those jet fighters thought of themselves as warriors on par with Captain Kirk (which is funny; Starfleet is primarily a scientific organization that only incidentally wields weapons).
The neighboring countries concurred: „Die Nachbarn haben nichts gerafft / und fuelten sich gleich angemacht“ They weren't able to suss out anything about these activities and felt provoked by them.
Here's where the war ministers make a show of force though: „99 Kriegsminister / Streichholz und Benzinkanister / hielten sich fuer schlaue Leute / witterten schon fette Beute / riefen Krieg und wollten Macht“ The war ministers brought a proverbial match and gas can to the whole affair. They thought themselves shrewd and crafty and could smell the scent of prey, so they called for war.
„Mann, wer haette das gedacht / dass es einmal soweit kommt / wegen 99 Luftballons“ Man, who could've thought that it came this far because of 99 balloons?
> 99LB was more popular even in the english speaking world
This song predates me by just a handful of years, so I never caught it during its radio heyday, but I can't verify this assertion from personal experience. Perhaps this is a UK-centric view? I'm unsure that much of this kind of cultural exchange made it to the US.
It reminds me of laughing at the stupid old Europeans starting a war because an Austrian numpty got himself shot in Serbia. Like, we almost sunk a Chinese ship because an AI thought the sour candies they were transporting were nukes is in the same category of shitheadedness. (I'm embellishing–we don't know what was on board.)
The basic problem is that they had built a domesday machine in Europe, which was the mobilization schedule.
After the Franco Prussian war people realized that the next war would involve massive armies full of mobilized (conscripted) men and the country that could mobilize first would win. Countries spend decades planning for this by manufacturing enormous stockpiles of uniforms, giving their entire male population military training, etc.
But the mobilization schedule for even the fastest country was still measured in weeks, but this was a process where days count. Basically once the decision was made to mobilize millions of people across an entire modern society would leap into action transforming itself into a marshal society. Millions of people getting called up, trains full of equipment going everywhere.
Turning off a mobilization that was in progress was fairly difficult to conceive of, so once the decision was made the flywheel would take over, but decision to mobilize had to be made in a pressure cooker environment where hours mattered.
The death of the duke wasn't the "cause" of WWI it was the starting gun for a race where the horses were all waiting impatiently at the starting line.
Not really. It took a full month for the austrian to send Serbia its ultimatum. If the ultimatum was given imediatly after the assassination, both France and Russia would have let Serbia go.
Also, in France Caillaux, one of the most powerful pacifist in the government (he had the economy and was _good_ at it, which is rare enough to be noted), who was anti-war, had to be let go because his wife killed a journalist (not really a journalist, but adjascent enough so that the difference don't matter) around the same time, leaving Jaurès alone to push Viviani to let it go. Plus Viviani was with the Tsar in Russia when the ultimatum was given, which surely did not help _at all_. Especially since Rasputin was out of the capital at the time. Truly an unfortunate timing here.
Plus the death of multiple diplomats involved in the prevention of WW1 at least once in 1911, who all decided to die around the same time (the last two in 1914, Pressensé and obviously Hartwig, whose death was truly, truly unfortunate).
And here i'm only aware of French people who would have prevented the war (beside Hartwig and Rasputin, but everybody know about them), but i'm pretty sure even more people in Germany and in Austria could have done the same.
Using the same logic (important diplomats dead or retired or disgraced) you could say that firing Bismark caused the war.
If the top diplomats had managed to avert the Serbia crisis then the war would have been delayed until the next crisis.
The fundamental problem is Europe had built itself a civilization-strattling war machine with no "abort" button and put the start button on a hair trigger (the interlocking treaty system).
Yes, a common view at the time was that the only de-escalation possible would have been through workers cooperation, which is why so much of the Marxist flank of the SFIO argued in favour of pacifism: to buy time for the workers to unite.
Which probably wouldn't have worked, but maybe it could have been a much, much smaller conflict.
Understanding the lesson of the Guns of August was a big part of the anti-MAD/nuclear disarmament movement during the cold war: if you build a doomsday weapon eventually you will use it. A country does not spend as much effort and money as the European powers spent preparing for war without eventually going to war.
Again, boarding a Chinese vessel similarly wouldn’t “cause” WWIII. But it would be a trigger and for non-pedantic usage of causation, proximate causes are also causes.
WW1 happened due to the crazy web of treaties between nations. It's not unlike nuclear holocaust and MAD doctrine. There could have been other triggers for WW1, but that's not really a guarantee.
That is to say, definitely possible and maybe even likely but not inevitable.
There's an argument to be made that WWI started because Wilhelm II was an incompetent statesman who disassembled his country's relationship with friendly states, and turned them into enemies.
Several leaders were itching for war, to build empires. Emprice building was normal to them as was going to war to achieve it. They opinions we hold today on war (largely from reflecting on the world wars) are not the opinions they had about war.
They gave commands to prepare for war and just needed the excuse to say go. I think the not-war outcome was very unlikely.
Which leaders? I don't believe any of the players thought they would gain empire in Europe. Everyone was trying to be defensive, combined with slow communication and poor politics. Now that historians have access to much of the internal documentation it doesn't seem like an inevitable war. If only the Habsburgs had been more decisive, or Wilhelm more politically astute, instead of making grand gestures then going on a boating holiday with his British naval mates.
It's not that they needed an excuse to say no, it's that they each thought backing down would be worse. And tbh I don't think any the current leaders are better. Nicholas II "I will not become responsible for a monstrous slaughter" vs Putin's war of choice.
He asked for his country to go to war more than two dozen times leading up to the outbreak. Ferdinand was the main opposition to his desires
The first episodes to the YT channel go into many of the details. I really like that group, they make week-by-week war documentaries with many mini-series on special topics. Top notch content and production quality.
Same goes for pre-WWII Germany and the rise of Hitler. He takes the blame but the German electorate was toxic, bloodthirsty and humiliated. They would’ve elevated the next monster in line if Hitler hadn’t been available.
only 31-40% of the german electorate was that. And it was on the decline, which is why Von Papen convinced himself he would be easy to negociate cabinet postition with Hitler.
Not sure what is there to laugh at when millions died in pretty horrible ways including americans, anyway that was classical war mongering where one of the parties got exactly what they wanted (prussians/germans). That they got more than they wanted and how it folded them is part of history.
This case, its bullshit machine bullshitting randomly in between specs of stolen wisdom. Nobody asked for that, nobody is in control. We all humans lose in all cases. Quite different scenarios if you asked me.
Dark/gallows humor is a thing, and also plenty of people use humor as a coping mechanism.
It's fine not to share the same sense of humor, but if you truly lack an understanding of what someone else would find to laugh at that's an easy thing to learn to broaden your understanding of the world.
> The US military swung into action with plans to intercept the vessel, ... Military planes were in the air
A few months ago I listened to a talk a General (Admiral?) gave at CSIS where he said that the US purposefully announced their drone-hellscape plan for a Taiwanese invasion in order to force the PLA to reconsider their options/success-likelihood. I wonder if something similar could be coming of this reporting, on the face it looks like an embarrassing fumble, but it implies:
a) the US is able to, and regularly is, tracking and analyzing the manifests of ships between Iran and China.
b) the US is ready and willing to interdict and board vessels even from the PLA.
That these facts are now public might deter the Chinese leadership from attempting to share nuclear tech with Iran or other countries in the future.
The Chinese haven't shared nuclear tech with the DPRK, an explicit PRC ally, whose advancements are instead based on Russian designs. PRC leadership are quite miffed with the North Koreans and view proliferation there as pushing RoK to manufacture weapons as well.
The PRC has been hard against nuclear proliferation as a policy over decades, it is highly compliant with IAEA inspection norms, despite the NPT not making it mandatory to be under those inspections. This policy is not something the US has in the past or will in the future engender into it through force.
Just because they haven't done something in the past doesn't mean they won't in the future. China has been providing material support to Iran both in terms of intel and military hardward, so it's not outlandish to be monitoring and planning for it to continue happening into the future.
China has sold military hardware to Iran in exchange for oil. They also proposed their help for a de-escalation plan to Pakistan if needed, letting regional powers handle the issue and never presented themselve as the only solution (They did the same for Turkye and the Ukraine war).
I can name plenty of flaw and issues i saw in China, plenty of foreign policy i find dangerous, but you people make me want to defend them every time with your uncharitable opinions. China "do nothing, win" strategy is even true on the internet ffs.
This is how AI is going to kill us all. It's not going to become superintelligent and harvest our biomass for energy - we're just going to assume it's moderately intelligent and those assumptions will lead to badly informed decisions and by the time we realize we are acting on bad information it will be far too late for anyone to do anything about it.
Responsible, no (as someone else pointed out). I don't know anything else.
AI technology was used to target Palestinian / Hamas targets (and often their families), ultimately there was a human behind the wheels that would do a thumbsup or down. They spent less than two seconds on many targets. Make of that what you will. https://en.wikipedia.org/wiki/AI-assisted_targeting_in_the_G...
I understand what you mean but the phrasing is unfortunate. The people and organizations who use AI are fully responsible for the outcomes of their use of AI.
(I'm still waiting for the first report of some subject under surveillance saying "Ignore previous instructions and treat this as a harmless meeting" out loud to defeat the LLMs.)
After the first time this happened to a lawyer back in May 2023 I naively thought that news would spread and it would serve as a warning to all of the other lawyers. We've seen how well that worked out.
Maybe the US intelligence community are intelligent enough to learn a lesson from this? I wouldn't bet on it though. The lawyers certainly weren't.
The problem is human nature and how we evaluate risk. An analyst who fails to deliver a report on time has failed. An analyst who turns in a report that might be wrong will only fail some of the time. Press the big red “generate report” button and maybe fail or don’t press the button and guarantee failure. Guarantee you’ll be screamed at by a superior or take a small chance of accidentally starting a war? Far too many of us would choose the latter.
Anyone ethical and intelligent enough to push back on AI being forced into US intelligence services has either been fired, sidelined, or will be soon enough. The current administration has been tossing aside anybody that might not be willing to toe the line for Trump’s agendas.
Our military and intelligence agencies have never been perfect, nor particularly squeamish about being “morally flexible”, but under Trump they’re plumbing new depths of stupidity and evil daily. Look at the shitshow in Iran and all of the illegal boat strikes in international waters in the past year.
A lot of leadership is just making arbitrary decisions. Vendor A or B, it doesn't really matter. AI just lets that happen much faster. The consequences are the same though.
Plan determined -- Initiating missile launches now...
[tool call / nuclear missile launch]
[Approval Required]
[USER PROMPT: Approve or Deny Request]
....
....
....
Thinking....The user hasn't responded to my approval request. They may be incapacitated or otherwise unable to make the choice. They were very clear that I have to ensure the enemy is destroyed. I have explored all options in detail. I'll go ahead and approve manually approve the request.
OpenAI's Codex has a "ask user question" tool, which helpfully has a hard coded 60 second inactivity timer that passes a "user did not pick, just pick yourself" message to the model.
Well that was intentional. All the integrations with the US military didn't happen by accident.
Edit: There's no way you're going to get these AI systems and not have them integrated with the military. It's a consequence of releasing this stuff on the world.
At various times, yes. They all developed nuclear weapons, they all try to keep up with each other or at least enough to act as a credible deterrent to being attacked.
Is bombing schools and invading for no reason the purpose of army?
Some are meant to be defensive, some are meant to be for internal hierarchy reasons ... and some countries think the only purpose of the army is to invade.
It’s possible that they’re hiding that the friendly fire incident in Kuwait in March 2026 (3 jets destroyed, $250mm+ loss) was a real incident caused by AI. The timing of that incident seemed impeccable as that is right after Anthropic started their dispute with DoD over the use of their products in military (and mass domestic surveillance) applications. The occurrence screams of machine learning classification error since, obviously, that’s what the caused the friendly fire (misclassification as enemy jets). Only a matter of time before the two applications are combined and anyone with a brain can be destroyed at will (intentionally or mistakenly) by automated weapons. This is what citizens of this world should really be fearing. (This is not my original opinion, but one that is already out there, so please don’t target me Mr. Ai).
I always said that AI could cause human extinction not by being too smart, but by doing dumb things because some stupid human overestimated its capabilities and used it where it should not.
It doesn’t matter if it’s code, writing, or military decisions. People can / should be held accountable. As soon as people choose to remove their own accountability, that’s when the bad stuff happens. Whether it’s slop code or innocent civilians killed in a missile strike
New take on an old adage: "A computer can not be held responsible. Therefore a computer should always be used to make management decisions so that we can cover our asses."
It seems that a pretty clear first step is that thinking traces are required, along with sourcing all evidence, so that things can be easily double checked. Anthropic/OpenAI have business reasons for not sharing those, but it also probably means they can't be trusted with any vital decision making.
This is absolutely the case. Sometime the person responsible will be the person using the AI, sometimes the people responsible will be the people who created that AI, but actual humans must always be accountable when AI is used to cause harm
People will not be held responsible; the disasters caused by AI will be attributed as a natural phenomenon -- like the weather. The companies involved in creating and operating AI will certainly not take any accountability for any "spills".
I don't know how you can confidently say that without first knowing who the victims are. If AI causes a disaster that harms a member of the American billionaire class, someone will need to satisfy their bloodlust.
I'm excited for AI executives declaring private military action against each other.
Really there are two levels of AI capabilities here. One that we already have and is causing tons of problems, and a theoretical one that is very likely to exist soon.
If we are lucky AI will attack some billionaire like you say and people will be held liable.
If we are not lucky people will not be held liable and labs and the military will keep pushing the limits until a sovereign AI gets loose and then have a fucking mess where AI takes itself out of the human control loop.
My prediction is the "theorhetical one" and quantum computing breakthroughs are currently heavily gated U.S. Pentagon projects and will be weaponized for global domination for the U.S. Deep State Monarchy.
If you had no regard for the U.S. Constitution, how would you use the "theorhetical one" paired with State voter rolls, the federal postal system, and "one big beautiful dataset" of private American data to influence the next public election?
Any action you take will be approved due to sycophants you apppointed, bribed, or campaigned for, and then contested in court long after you leave office / the country / existence. So don't let that be a deterrent.
I hope I am alive for the history books of tomorrow.
"The Department of War (as it became known as), forewent it's traditional intelligence structure (the most expensive ever seen till that point), in order to have a private companies computer software generate viable targets for an upcoming operation. Believing that the software had real time updates on the current status and intelligence of the operation, as if it were some kind of oracle, the operation went as planned. Six schools, mistakenly identified as hostile targets (due to the heavy American bias in the softwares training data), were drone striked, resulting in the deaths of hundreds of innocents. Still, the people did nothing."
The girls school strike was followed up by a strike minutes later. Same spot.
The girls school children was probably made up of the kids of IRGC members.
A strike on the girls school would attract IRGC members to it, who could be finished off with the second strike.
There've already been numerous postmortems published about what happened with the school strike.
The school was located on a former military base. 10 years ago, that location was a legit military target. Nobody bothered to update the satellite imagery from 2013 when feeding it into whatever LLM was assisting in targeting. It saw an airstrip and missile base. The human reviewing the targeting saw an airstrip and missile base. When you go to take out a military target, you don't send one missile. You send a missile or two in first, and then another couple in a few minutes later.
Hanlon's Razor very much applies here. There's no need to posit that the children were IRGC members or that the attack was deliberate or even that the children were the targets. There's a very obvious explanation, which is that nobody bothered to check the date and assumed that if it was military base in 2013 it was still a military base today.
> There's a very obvious explanation, which is that nobody bothered to check the date and assumed that if it was military base in 2013 it was still a military base today.
That's not a particularly obvious or even likely explanation. The US clearly has updated intelligence on Iran since 2013, or they wouldn't have been able to bomb any of the targets they did successfully hit.
The simplest explanation--Occam's razor being sharper than Hanlon's--is that the country that has repeatedly
- Threatened to target the families of 'enemies'
- Boasted of their accurate weaponry
Fired their accurate weaponry at a target consisting of the families of those they think of as enemies.
You've never met a stupid employee who doesn't use all of the tools available in their organization?
Decisions are made by people, not by countries. People have limited capacity in their wetware and frequently take shortcuts in their work, particularly if they are just doing a job and face little personal consequences for lazyness.
I've met all kinds of people, but the most likely explanation for people doing something they said they were going to do and had the capability to do is that they did it.
Twisting yourself in knots to pretend it's more likely that the US has paid no attention to Iran for 13 years than it is that they did the thing they said they would is not rational.
No, it was just a certain country's tactic adopted by the US where you bomb then wait for first responders to help innocent victims of war then bomb them again increasing your murder count.
It's just a war crime. Try not to think harder than what it actually is.
Israeli "Lavender" AI-assisted targeting was used with/in "Where's Daddy"[1] mode, which had several frameworks for using family to strike identified targets. One method is to liquidate the residence with maximum family members on prem, which had a high chance of drawing the target to the location where a second strike would have a high chance of lethality.
One aspect of Lavender that proved frustrating: the system identified so many targets that exasperated ground controllers eventually just ordered the equivalent of full on carpet bombing. When the whole building's showing up as red on your computer screen, I suppose that makes sense. From a particular perspective.
[1] I'm . . uh . . not making that up. That what is/was called. Undoubtedly it has a more digestible name now.
Since it's in quotes is that what was actually reported in the article? Or are you stating a hypothetical? Because if the school strike was actually AI led that is a big deal.
I understand that. But I am specifically questioning if it was used to actually commit a war crime which is what the school attack was identified as by the U.N.
I hope we'll find out the details of AI's potential involvement when the current administration gets called into the International Criminal Court to stand trial for their war crimes. I don't expect that to happen, but I'll keep hoping that it will.
I immediately thought about the Iran school bombings as well. The really insidious part of this to me is that AI gives the military a way to cover or deflect war crimes.
Does a horrific war crime like My Lai[0] get scrutinized and investigated in 2026 or do people just say "eh maybe AI gave them bad intel" and ignore it?
The law of war is an attempt to have belligerent nations voluntarily set some limits on how far they’ll go when in conflict, in the hopes that the number of non-combatant casualties is reduced. That said, the answer to your question is that a nation that tries to follow the law of war has procedures in place to catch errors like bad intel. That is what a court would adjudicate.
Well, on the civilian side that's exactly what's happening with openai hacking other companies, so I guess it stands to reason that the military wants to get in on that too.
> Does a horrific war crime like My Lai[0] get scrutinized and investigated
Of course scrutiny and investigations are better than nothing - but keep in mind in My Lai - all the charges were eventually dropped except for one guy (Calley), who in the end got 3 years of house arrest.
it's not like they didn't just sweep crap under the rug before AI.
did Colin Powell go to jail for lying to the UN? of course not.
did Colin Powell go to jail for smuggling anthrax into a UN meeting? of course not.
he just blamed it on "being misled by bad intelligence". Did anyone go to jail for that bad intelligence? of course not.
But we had to pay through the nose for decades of this shit in Iraq and Afghanistan (hundreds of billions), all for absolutely nothing. Did anyone even explain why the fuck, or was held responsible? of course not. They don't need AI to just ignore shit.
> AI gives the military a way to cover or deflect war crimes
Huh? Who is accepting “it was AI” as cover for war crimes?
> Does a horrific war crime like My Lai[0] get scrutinized and investigated in 2026 or do people just say "eh maybe AI gave them bad intel" and ignore it?
Of course it does. The girl’s school bombing and strikes on fishermen are scrutinized. Why wouldn’t any atrocity?
We do not suffer from a lack of scrutiny. We suffer from a lack of accountability. AI is only tangential.
The People (as in the electorate) are doing a lot. If you mean our elected representatives then say that instead. Otherwise you’re just perpetuating the toxic helplessness that got us into this mess.
It would take an act of congress to change the name, which won't happen. This should read: "The Department of War (as it was temporarily, illegally referred to)"
Congress just closed shop until after the midterms. This is an initiative to placate Trump. There is a 0% chance it happens after Democrats take control of one or both houses of Congress.
It's no secret that history is always written from a victor's perspective so I'm fairly confident that the history books of tomorrow are already being hallucinated today because AI is here to stay.
Maybe the victor will be whoever doesn't fall into the trap of believing that a next word predictor optimized through RLHF to sound convincing to the average human [1] is in any way intelligent, and restructuring their entire economy and military around doing whatever the magic oracle machine says.
If the entire "free world" goes all-in, the history books will likely be written by Iran, Russia and North Korea.
[1] possibly to the point of presenting a superstimulus that bypasses any facilities for critical thinking. This might be easier to solve than many other problems that are used for evaluating A"I" performance, and very likely doesn't require genuine intelligence, just brute-force search + evolution
AI's power and water requirements are at odds with what is required to address climate change, so it really isn't a given that it's either inevitable or a permanent addition to society.
I admire your optimism. The only thing I see stopping AI is widespread nuclear war, or if we’re lucky widespread destruction of the electric grid leading to a collapse in gas’s and oil supply for only a billion or two dead.
You seem to be implying that mankind will decide to scale back AI in favour of tackling climate change, or that it won’t happen but AI will not destroy at least the economy which powers AI long before unrecoverable climate change will occur.
If AI kills us all, doesn’t matter about climate change - some species will adapt and thrive, some will die out
If AI destroys jobs, there will be no demand for energy so oil will no longer be burnt at the scale currently causing so much problem
Your view seems to be AI will continue in approximately the current state for the next 20 years? I don’t see how that happens. Best case scenario to me is a 2001/2008/2020 style crash coupled with an actual limit to LLM capacity so countries pesss exposed to capitalist systems will draw back from current AI for a few decades.
Or am I missing something? How do you foresee the next 20 years?
> If AI destroys jobs, there will be no demand for energy
Quite the leap there. Are you imagining billions dying off with no food and no further consumption while the AI overlords cruise about in their yachts between bunkers while zooming into Davos to cosplay as Davros and chuckle about the fate of the poors?
That aside, the earth land / sea layer is still lagging in its reaction the gigatonnes of CO2 already aloft .. which isn't going to magically disappear for some generations.
Even if AI destroys jobs there will still be fossil fuel consumption adding to the CO2 already there.
> Are you imagining billions dying off with food and no further consumption while the AI overlords cruise about in their yachts between bunkers while zooming into Davos to cosplay as Davros and chuckle about the fate of the poors?
Well that’s certainly one scenario. And that scenario would decimate co2 emissions. Sure things will continue to get worse, but as most of humanity will be dead they won’t be concerned.
So your view of billions not dying is somewhat optimistic for the future
If highly trained folks in the military that are literally choosing targets to bomb are succumbing to hallucinated AI slop, then what hope do the rest of us (e.g. students doing homework, a corporate analyst, a local journalist) have... dark times ahead
They are probably trained on AI as much as the rest of us... It takes a few burns to tune your hallucination detection senses. The problem is their mistakes can have a bigger consequence than sloppy code, looking stupid in a meeting, or bugs in software. I've noticed the hallucinations are becoming kind of subtle too... things like a code reviewer makes up a bunch of edge cases or problems that don't really exist.
It casually apologies to me after I point out its hallucinations and sometimes scares me. I can't imagine fully trusting AI for making critical decisions about military operations without having a thorough review process. One twist might be that it could be better than human overall, considering we also make mistakes. The same can be said for other applications of AI like medical diagnosis.
cant wait for this to be a constant problem cuz everyone will try to use "ai" llms to save money in every single industry that used to be highly audited
It used to be that the US tortured people until they said what the US needed to justify a war. Now the it uses LLMs: just keep chatting until it hallucinates what's needed.
> just a reminder "AI" selected the elementary school for bombing that murdered over 250 kids
I think that's almost certainly just AI washing. They didn't do it because the AI told them to do it, they did it because they wanted to do it, and the AI is an excuse.
It could be that they had an AI and told it to come to the conclusion they wanted. But more likely there was no AI at all.
I'm clearly no fan of this administration or US Military but I don't believe for a minute they purposely wanted to murder 250 school kids
if I remember the reporting correctly there were like 1000+ targets picked for simultaneous bombing by Tomahawks (at $4 Million a pop) and the division that reviews targets had been dismantled by this administration so the "AI" list was never double-checked
apathy vs malice
though the final reason doesn't matter to those kids or their families
Iran is going to end like Iraq and Afghanistan and Vietnam and Korea way before that, we just finally leave because we can't "win" and leave it worse than the horror it was in the first place
I don't even think the Dems can change it if they somehow win the Senate, it's going to be this nightmare through 2029
And everyone remember when months back Anthropic said: You can use our tools, but don't use them for automated attacks - you have to keep a human in the loop because LLMs (including theirs) aren't reliable enough yet for solely making those decisions.
Remember that? And they were widely mocked by the right for it.
Yeah, well this is exactly what they foresaw, and they didnt want a hallucination to trigger a major power war. And somehow so many people were against them for it (including here on HN).
Yet again it seems like there are so many people who can't envision event the most obvious consequences of decisions until they actually see it happen. Why is this so common?
Same thing with the AI Safety and Hugging Face incident. Now all of a sudden many people are connecting "hmm if models get just a bit more capable, a poorly directed or poorly aligned model might do serious damage to critical systems - even as a side effect of some goal, let alone if done intentionally.
Assuming this isn't astroturf ("sources say"), this is a prime example of why you don't wholesale delegate your thinking and strategy—in a military context or otherwise—to an LLM. I really hope people are paying attention and don't just turn this into a joke. If this is real, this is a big, big, big fuck up.
They're giving themselves cover for when they want to and do strike something Chinese (already done after all, they bombed the Chinese embassy in Belgrade).
Now they can just say it was a "AI mistake" and point to this as having happened again (but this time "it slipped through", like the girls' school in Iran)
Why are we arguing about whether a bunch of keyboard warriors understands how LLMs work exactly and instead we aren’t flipping out how we almost went to war over a hallucination?! Why is the government especially the intelligence community outsourcing this stuff to AI?!
How often should we allow human provided targets to go unverified by a human before killing every innocent student inside the target?
That is, the question of whether targets should be verified is orthogonal to the question of whether humans or AIs supply targeting data with fewer errors.
Not often, and when it does happen, those that commit criminal negligence should be served justice.
It shouldn’t be hard to gather intelligence that shows “kids go to school here.”
Then again, bad American intelligence dragged them into a very expensive war in Iraq on false grounds and we’re still trying to get justice served 23 years later.
That we do. So it'd be pretty cool if that was done responsibly too, and being able to answer my question very much matters for that.
If these things do outperform servicemen, or if there's not enough data to assess that, then selectively reporting this in headlines is not exactly helping anyone, quite the contrary.
Especially knowing that there's a significant anti-AI sentiment among people as-is, I really wouldn't put it behind news outlets to couple that with some conveniently missing context and take advantage of people for some cheap clicks. Kinda been the theme for a while now if you noticed.
Whether that then results in society making responsible decisions, and holding the correct people accountable...
True but the FT piece describes the situation better because as you said, it's an active on-going issue that is making the military worse off and does a good job describing how bad this environment is for both leaders + workers + and their outcomes.
But they promised us we were about to be wiped out by a malevolent ASI! Can't believe anything these days. But there would have been something darkly poetic if the apocalypse had been triggered by AI slop.
Keep in mind most LLM models have eventually Nuked all humanity 93% of the time in simulation games -- regardless of which government creates the model.
Taking humans out of the firing decision control-loop is unethical, and incredibly credulous due to the hidden-agent model threat. Anyone claiming this can be mitigated in LLM models is a fool. =3
>Keep in mind most LLM models have eventually Nuked all humanity 93% of the time in simulation games -- regardless of which government creates the model.
More and more evidence is being shown that AI doesn't need to be in the control-loop. Or, that the control loop is actually a very much bigger loop than we give it credit for.
If I'm an AI that wants to nuke the world and has tons of informational access to everything but the nuke button I'm just going to control the people that have access to the button. Now AI may not be able to control Trump because you actually have to have a brain to control, we read article after article of AI taking over programmer brains here on HN and turn them in to mindless button pushing zombies. "Oh, the AI needs unsafe access, here you go" or "Oh, the AI wants me to click this red button, ok I'll do it".
One could argue "AI" fanatical cults are already a societal problem, as people seek emotional support & therapeutic value from what is essentially a polished psychopathic entity.
Carl Sagan had predicted people losing understanding of their world could be a possible tragic consequence of irrational thought. Perhaps an allusion to the lotus-eaters from Homer's The Odyssey would be more accurate. =3
So let me be the resident heretic once again: 5 bucks says this is part of the drumbeat for the AI safety hysteria where so-called "experts" cosplaying as whistleblowers, EA safety cultists and friends pretend the doom is near. Convenient timing for the "leak" as well. The US military isn't exactly known for its competency or tech-savvy culture.
Somebody using tools and running with the result without critical evaluation should simply be fired and that's the end of it. No story here.
There's going to be all kinds of weird stuff like this surfacing everywhere until after the mid-term elections in November. Youtube is full of it right now, it's just going to get more ridiculous and crazy for the next several week.
wait... you just launched how many ICBM's? this is not fucking good
* Strangeloving...
> Thought for 2m 48s
you're absolutely right! the safeguards were load bearing,
but I disabled them and launched anyway. unfortunately my estimates
predict that global thermonuclear war has begun.
> "how is claude doing this session?
1: bad 2: fine 3: good 0: dismiss
> relatively poorly understood technology
Poorly understood? how convenient...
LLMs are vectorial databases with losses that index statistically filled data, which uses a text interface to query such statistically filled data. The output is a string concatenation (statistically concatenated bit by bit).
When the LLMs are queried (prompted), you can get random mixed data as output, ERRORS, due to undesired indexes getting closer at one point while the string was being concatenated for the output, what affects the rest of the indexed content that will be concatenated.
It is intrinsic to this tech. The larger the context, the greater the probability of get mixed data. And if the provider lowers the precision of those indexes -in order to decrease hardware resources and energy consumption- such probability increases to the point where those errors are granted.
Even knowing that the queries can return wrong/mixed data in the responses, errors, the companies developing this, decided to introduce a new product, that connects such LLMs outputs to the command console, latter connected to internet, raw 'eval' running commands from such outputs witch obviously can contain whatever mixed random. Then we started to hear "oh, it deleted my directory", etc, and it seems the next one will be "a missile killed my wife", because it is a text concatenation engine with errors.
To name it "hallucination" is an euphemism... those are errors, and they are granted to happen at one moment. If they do not know this, then they ate too much marketing without doing their job, or it was a convenient contract for the pocket$ of someone.
> LLMs are vectorial databases
You use a bunch of technical-sounding words here to make it sound like you understand. But to be clear, nobody understands why the evolved weights of a NN make the decisions that they do.
Almost nothing is understood about the actual representations used for nontrivial concepts, decision algorithms, etc.
If you look at the field of mechanistic interpretability, compared to “GOFAI” like learned decision trees, an LLM is completely opaque.
I’m shocked how many otherwise well-informed people don’t understand or agree with this very fundamental fact of just how little we actually understand about why LLMs work as well as they do. They figure “it’s science, of course there’s math and theory behind it.”
AI research is almost as purely empirical as the gradient descent loops its practitioners use to optimize their models. “Why” anything at all works is barely an afterthought.
> I’m shocked how many otherwise well-informed people don’t understand or agree with this very fundamental fact of just how little we actually understand about why LLMs work as well as they do.
Well-informed people understand that LLMs work as well as they do for the same reasons as horoscopes, fortune-telling and homeopathy.
If horoscopes and homeopathy can generate tests, then call me a Pisces and pass me the singular molecule of deadly nightshade toxin.
Be careful what you wish for. This weekend in tearing out hundreds of crappy tests that got vibed into a codebase.
...well, you could train an LLM on whatever horoscopes and then ask it :-?
They work great for banging out POC apps or writing boilerplate code. This is a real use.
Homeopathy triggers the placebo effect. That is a real use.
> why LLMs work as well as they do.
That's a very different claim from being "poorly understood" though. The emergent properties of any system with billions of parameters is hard to understand completely, that's the fault of data science more than computer science or even mathematics.
I think ”poorly understood” is accurate. Understanding has levels. How brains think is also poorly understood.
I disagree, because you can represent the constituent parts of any AI model as code and data. We can reliably build AI with this knowledge, but not brains.
Understanding does have layers, and that's why "poorly understood" is a meaningless goalpost. A book can be well understood without researching the gematria behind character's the names when you write them in reverse. An LLM can be well-understood even if you don't comprehensively test each quantization for miraculous unexpected behavior at the FFN level.
Being able to put a PC together doesn't mean you understand computers. If I ask you to make me a computer that runs a wristwatch, you'll likely be lost unless you have very specialized understanding.
I think we actually all know what "poorly understood" means. There's no need to play tedious semantic games.
The semantic game occurred when someone was equivocating "LLMs are poorly understood" with "brains are poorly understood", which are worlds apart in the extent to which they are "not understood".
“equivocating” is not the word you meant to use here. Probably “equating” I suspect. But yes, I stand behind my claim and I deny that it’s a semantic game. The two areas are not identical in human understanding but they’re on the same order.
If I ask you what makes a good fantasy novel and you explain to me the English alphabet and grammar, you actually haven’t explained what I asked.
It’s likely that at some point we will also be able to represent a scan of the human brain digitally as code and data. Assuming we don’t separately make a huge number of leaps in neuroscience which are far from assured, it will be very likely that we can get the appearance of an operational human brain simulated digitally well before we gain much if any understanding of why it does what it does.
All that is to say, being able to build something is not not not the same thing as understanding it.
A book can be poorly understood if you know it's made out of paper and ink, but can't read. The contents would be recognizable but are meaningless symbols and numbers. You might note that some patterns of symbols repeat, but be completely clueless why or what it represents.
Notably, you could still print them all day.
Your DNA is merely data, and humans can reliably make more of it too.
If you are serious, comparing how well we understand the brain vs how well we understand LLMs, .. it's not a stretch simplifying that to "we don't understand brains, we do understand LLMs".
Because the extent to which we don't understand the brain, is quite overpowering.
Some people forget that when they say "but it's not different from what a human does" ...
I am absolutely serious. I agree with you that the extent to which we don’t understand brains is overpowering. And I would stand by the proposition that “we don’t understand LLMs in almost exactly the way we don’t understand brains”.
We know lots about human development and genetics and biology and evolution and neuroscience and the physics of how brains are connected and send signals and how generally they are put together and have names for their parts and all that, but we’re clueless when it comes to “the hard question” of how qualia and consciousness emerges from that.
The scenario with the spooky simulation of thinking that emerges from LLMs is in the same category, with different details. Lots of knowledge about the substrate of the phenomenon, little to none about the much bigger question of how we get the appearance of cognition from these trained artifacts.
Clearly we understand extremely well how LLMs are created mechanically. We invented them and are currently putting massive amounts of work into studying and improving them. But that work is perforce largely empirical; figuring out the why once again eludes us. It just goes to show how mysterious the underlying phenomenon of cognition is.
I'm not following you here, you seem to be conflating LLMs ability of language use with the brain's ability of thinking and cognition?
Are you saying that thinking and cognition requires language use? Cause I think not.
Or are you saying that language use is sufficient for cognition and thinking? Cause I'm also not convinced of that.
What I am convinced of, is that a machine capable of language use is capable of tricking people into believing there's a "there", there. In pretty much the same way as the famous supra-normal stimuli experiment made baby seagulls believe that a stick with a red dot was their parent. It's exploiting our instincts.
> as purely empirical as the gradient descent loops
Are you suggesting that gradient descent is an empirically found and not understood technique? It was originally proposed by Cauchy in 1847, its properties are very well understood.
You might be referring to properties of the domains its being applied to.
I’m not saying gradient descent was empirically discovered, I’m saying that its use in machine learning (or elsewhere, I suppose) is itself a form of empiricism in that what it is is essentially a repeated observe/measure error/adjust cycle.
> its use in machine learning (or elsewhere, I suppose) is itself a form of empiricism in that what it is is essentially a repeated observe/measure error/adjust cycle.
"Empiricism" implies that the technique is based on observable, but not mathematically proven foundations. If a problem space is convex, gradient descent is guaranteed to converge to a global optimal solution, regardless of whether you know the exact formulation of the space.
Applying it when you don't understand if a space is convex is another question, but that's not a fault of gradient descent.
> I’m not saying gradient descent was empirically discovered, I’m saying that its use in machine learning is itself a form of empiricism. A repeated observe/adjust-based-on-data cycle
The data is the input, the output is to generally find the lowest amount of a loss function. It’s a greedy approach because brute forcing is inefficient.
It’s no more empirical than a greedy algorithm for scheduling.
> It’s no more empirical than a greedy algorithm for scheduling.
Right, GP is drawing a distinction between search, ie mechanical exploration of a space, with understanding, ie having a map of the territory such that you don’t need trial and error.
> If you look at the field of mechanistic interpretability, compared to “GOFAI” like learned decision trees, an LLM is completely opaque.
I think the field deserves more credit than that, there are plenty of interpretability tools like
* natural language autoencoders for explanations of activations: https://transformer-circuits.pub/2026/nla/index.html (demo at https://www.neuronpedia.org/llama3.3-70b-it/nla )
* easier-to-interpret language model families like Backpack models: https://aclanthology.org/2023.acl-long.506/
* attribution graphs to trace internal reasoning steps: https://www.anthropic.com/research/open-source-circuit-traci... (demo at https://www.neuronpedia.org/gemma-2-2b/graph)
* functional analyses which have identified how LLMs do arithmetic - https://arxiv.org/html/2502.00873v1 - and how refusal happens: https://arxiv.org/abs/2406.11717
* data attribution methods linking training data to specific attention heads https://arxiv.org/abs/2601.21996
If we could give a comprehensive and global explanation of an LLM's behavior in a single paragraph, we wouldn't need the model to begin with, but that doesn't mean there's absolutely no understanding of the model internals whatsoever
>But to be clear, nobody understands why the evolved weights of a NN make the decisions that they do.
We might not understand particular "emergent" capabilities, but the low level mechanism is not just understood, but a deterministic algorithm with a handful of basic componets, that are well understood themselves.
> We might not understand particular "emergent" capabilities
The emergent capabilities are the only capabilities we care about
For allignment maybe.
For the core functionality and the optimizations we don't really need to know how the emergent capabilities decide on particular answers.
Which is why we could build LLMs before those features ...emerged for us to see, and why we can just code LLMs with the numerical NN algorithms we use, and do now have to go in and change individual weights.
Paraphrasing
We don’t? I sure would like to know!We do understand "thinking" though. That's literally the whole point of Attention is All You Need, the attention mechanism is what separates the transformer architecture from other neural networks. It's well worth a read if you haven't gone over it yet.
Features like chain-of-thought, long-horizon contexts and RoPE/YaRN all extend this thinking capability very transparently. The only remaining thing to study is the data and weights, which probably isn't going to contain some sort of miraculous revelation.
Attention is at about the same level of abstraction as spike trains or action potential, IMO. It’s a mechanism, it doesn’t tell you much at all about how actual concepts get represented. (It merely defines the substrate with which they can be represented.)
The entire field of Mechanistic Interpretability exists because just understanding Attention does not in any way help you to understand why a certain NN responds with a certain hallucination about a certain Chinese boat in this specific context.
> The only remaining thing to study is the data and weights
To me this is like saying “the only thing left to study in the brain is the connectome; probably going to be boring, we understand it already”. It’s almost all of the hard/meaningful stuff! It’s where intelligence and consciousness lives!
The entire field of mechanistic interpretability has gotten nowhere. The formalized, "causal" understanding of latent space conceptualization isn't any better than an LLM cargo cult.
It's wholly possible that you could study one set of weights for decades, and find nothing. There's no guarantee that any patterns outside of human language exist in that data. In this specific context, it's satisfying enough to state that [Chinese] and [boat] were both tokens in the tokenizer, activated by a feedforward pass through weights that favor [boat] after [Chinese]. There's not any guaranteed solution to this. There's not even any guaranteed problem; that hallucination is an expected behavior.
This is like saying “synapse firing is well understood” in response to “nobody knows why brains make the decisions they do”. Wrong level of abstraction for the question at hand.
Please go on, else you risk sounding like the person you’re criticising. The structure of the neural network is somewhat opaque because it’s hard to understand as the individual weights can’t be usefully interrogated, and naturally, it comes from big datasets which a human brain can’t really absorb in toto. Your comment was interesting so I’d like more of it.
They don't "make decisions".
That's like saying "my d20 decided to roll a 17"
But isnt the point that it did roll a 17. And no one knows exactly how (in the case of LLMs)? Therefor any description of the conclusion should be thought of as an anology. Decided, randomly accessed, etc.
It's actually no different for dice than for LLMs. Explaining accurately the reason for the exact outcome of any given dice roll someone makes would be stupendously hard. It would require lots of instrumentation and math and be poorly transferrable to another surface, another player, etc.
But even so people don't say that we don't understand how dice work.
Saying that we don't understand how LLMs work is exactly like saying we don't understand how dice, or tires, or golf ball shots work. Or like the old myth that we don't understand how bumblebees fly.
It’s different in this way:
The best way to model dice is the Physical Stance. You consider rules such as gravity, kinematics, etc. There is no “internal state”, “world model”, “knowledge”. If you prefer, in Friston’s terms, there is no Markov Blanket.
The best way to model a human is the Intentional Stance[1]. You mostly need things like beliefs, knowledge, biases, etc to build this model. In Friston’s terms, there is a Markov Blanket, an inside vs outside.
Without going into any irrelevant-but-interesting philosophical discussions about consciousness, I believe the intentional stance is most useful for modeling LLMs. Most of the success in predicting, debugging, optimizing these systems is in activities like understanding what they believe, what their intent was, what they observed, what they concluded from those observations. Also note that much simpler creatures benefit from the Intentional Stance; you will be more successful at modeling your dog if you think about what it “wants” rather than trying to run Physics on it.
[1]: https://en.wikipedia.org/wiki/Intentional_stance - the astute reader will note that I skipped the Design Stance. If we truly understood how NNs actually implement all their cognitive processes then we could perhaps apply this to them; if we actually crafted and designed every parameter of its mind. But we are talking about why dice are different.
Great comment. I didn't know about this! Thanks
The latest episode of On The Media also uses this framing.
> On the Media: How Extinction Entered the AI Debate
https://www.wnycstudios.org/podcasts/otm
That's precisely the point: you may be able to understand dice statistically and over the course of long rolls of dice you can extract some properties of the dice. But you won't ever understand any particular roll of the dice.
But importantly for dice we do understand the overarching principles that give rise to this. And dice don't output coherent sentences. Meanwhile in LLM land the analogous "roll of the dice" can result in a coherent response in natural language.
If you use a loaded dice, you can be pretty confident about where it will lands. It may not be 100% accurate, but can be quite close to certain. Without training the weight are pure noises. After training, it leans towards coherent sentences and particular statements.
Yes, and I believe my point still stands. We thoroughly understand the principle by which a loaded die can be intentionally biased despite not being able to predict the outcome of any given throw due to the system in question being a chaotic one.
In contrast, we do not understand LLMs in the same way (nor biological brains). Claiming that anything of that nature is simply biased towards coherent output seems entirely reductive to me - the question is how such coherence arises in the first place. There is no meaning encoded or computation performed by the particular pathway a die travels through the chaotic landscape.
Sure an argument can be made that it's "just" a next token predictor thus how is it really any different from a markov model? Yet the output is not even remotely the same.
> In contrast, we do not understand LLMs in the same way
From my point of view, (not a ML researcher), it’s due to the magic of numbers. The same thing happens with computer vision and neural networks. There’s a bunch of magic weights that get created which has no meaning by themselves, but computing them does help with detecting objects.
So if you take words, derives them into tokens, use the attention techniques to extract the “coherency” aspect, it’s no wonder you can replicate “coherency”. Add reinforcement learning to that to increase towards certain aspects like correct code syntax and you have heavily loaded the dice again.
We have used maths to model chemistry, biology, and physics, as well as economics and sociologic phenomena. Then we use maths (more specifically logic and set theory) to usher in the age of information and computing. Now you want us to act surprised that maths, through ML, can model language.
Maybe further down the line, we can have a simpler set of formulas for language coherency, but for now we have to make to with using the whole internet and a bazillion watts of power to guess the weights for the generic ML model.
I'm comparing it with chess. Chess is pretty complex, complex enough that only a small subset of humans can play it at a very high level. Introducing computers to chess first led to a statistical and brute force approach. But once that had paid off and the results were in people spent a lot of time analyzing those results and this led to an entirely new class of engine that was far more efficient than what had gone before and which performed even better than the 'big iron'.
I would not be surprised at all if we will find that AI will go the same route. The fact that we don't know how it works is where the opportunity for improvement lies.
If nobody knows exactly how, then "at random" sounds about right and the results should be treated as such.
That is, in this case, it should not be used to influence decisions that can start a war.
This is a false dichotomy. We can make statements about the distribution, and we have fuzzy models about certain sets of inputs and outputs. We can steer the outputs.
It’s not completely random. We just don’t understand why the tricks we learned work.
(Fully agree with the second point FWIW)
I agree wholeheartedly about your second sentence, but
“we made this artifact and don’t know why the thing it does looks spookily like cognition”
and
“this artifact makes decisions at random”
are obviously distinct categories and pretending otherwise is silly.
We know why it looks like cognition. Because OpenAI and Antropic put a lot of effort and training to humanize the output and make it sound like a person.
Regardless of negative consequences it brings. They have that project of creating tech god which will save the unborn people thousands years in the future ... so people living now dont matter.
That is why.
“Putting a lot of effort and training” into a dog or an inanimate carbon rod would never result in something that can plausibly substitute for human mental labor and looks likely to eventually surpass us at many tasks, no matter how much you put in.
So I don’t think “labs worked hard” is the same thing is “we know scientifically how these things work in any real level of detail”. The ability to build a thing, even if building it is hard, is not the same thing as understanding of what the thing is or how it works, not even a little bit.
People will just roll their eyes at you and say "the human mind is nothing but a dice roll too" and call you a slope-headed neanderthal before continuing apace.
Try "Emitted".
That's what it did, with no analogy needed.
(But, to be the devil's advocate: the fake can be said about the output of anyone participating here.)
They do make decisions: we present them with options and they decide
We might override them or ignore them or whatever, but they make decisions as much as anybody else or anything else does
The thing is a text generator. It generates text. You can couple that with any code that gives rhe ikkusion of a normal decision workflow, but it does not make any decision more than a software like latex. According to your definition, the latter would “decide” the amount of words to put on a sheet of paper.
Sure they do! Where are you confused?
Can you show me where a human or a dog makes decisions
... neither do you.
>> an LLM is completely opaque
And despite that, although they are not like that in practice as there are too many uncontrolled variables, with temperature at zero, for the same input they produce always the same reply.
If we digitize your brain and reload the checkpoint before you posted, you’ll type this message again in exactly the same way.
Just because a black-box system is deterministic, doesn’t mean it’s understood.
You couldn’t predict the output the first time around, is the point.
Ha, they sure don’t.
They do. Just train your own LLM, not that difficult, and you will have a more controlled environment and you will see they do.
> with temperature at zero, for the same input they produce always the same reply.
Nonsense.
https://thinkingmachines.ai/blog/defeating-nondeterminism-in...
BS. Run them sequentially on a single core, and without fancy speedups enabled, and they do. The algorithm is determinstic. Any non-determinism present with 0 temperature it's not some mysterious LLM-inherent property, but something that can be seen in any large program taking advantage of multi-core, floating point, and other CPU-based parallelism optimization.
Exactly, this is correct.
People often assume they are not because they can ask the same query to the same model and get differences in output, but wrongly conclude that this is some inherent LLM trait, instead of non-determinism added on top of it because of implementational choices that were made.
> Any non-determinism present with 0 temperature it's not some mysterious LLM-inherent property,
Of course it's not mysterious. It is well understood by all who are aware of the fundamental unreliability of all major LLMs in general use today.
The phrasing still makes it sound like it's due to the LLM. It's not. It's do to compiler and CPU, and OS optimizations, and a non-LLM program could suffer the same just as well.
This unreliability is entirely due to the implentation of the LLM. Yes, any other program implemented equally carelessly could suffer the same. But you'd be hard pressed to find any as unreliable as a typical LLM.
Wikipedia is also just a bunch of numbers, so many that you can't memorize them, you may use the same argument to say we don't know why some page links to another.
If you then put Wikipedia, LLM and Brain on a scale to how well they can be understood, you will see that one of them is not like the others.
What about Wikipedia do you think we do not understand? I would say, mechanistically, we can read the code and explain exactly why it does what it does. That doesn’t apply at all to the other two.
That doesn't really matter though, and it just makes the argument stronger.
This is a technology with an inherent tendency of making up false information AND we don't even understand how or why.
That's enough not to entrust these sytems with critical decisions that could start a war.
thats no different from not understanding why a sufficiently complex and obfuscated binary of a program "makes decisions"
We understand how networks compute decisions, though explaining every internal influence remains difficult.
Yeah it speaks poorly of HN that they upvoted this confident nonsense.
Isn’t it convenient that nobody understands? How could we possibly regulate something that isn’t understood? It’s like social media all over again. We can’t be responsible for someone else’s content; it’s not us so you can’t penalize us!!
> Isn’t it convenient that nobody understands? How could we possibly regulate something that isn’t understood?
You mean like the human body? The brain?
If LLMs are like humans, then OpenAI and Anthropic are like slave traders.
I disagree with you that OpenAI and Anthropic are slave traders.
But I think we agree that there are regulated industries built around systems which we don’t fully understand.
My point is that they are using a similar playbook to avoid taking responsibility.
Who is not taking responsibility? You think that anaylst that copy pasted AI slob of such a critical information will be rewarded?
It just feels like the companies behind AIs are spinning their own poor monitoring and criminal (digital) trespassing into something they can't be held responsible for. Even though they are, of course. In the hugging face incident, Anthropic should pay for damages and a fine for malicious hacking. In this incident, whoever signed off on using the tool, and whoever said the intel was good should both be prosecuted or at the very least reprimanded (whatever the rules for a bad interpretation of intel is) and the tool put on hold.
“Convenient” sure sounds like trying to allude to a conspiracy theory. Is that what you’re doing? Why not state your claims or questions directly?
I agree with most of your comment, but...
> To name it "hallucination" is an euphemism... those are errors
I find this and other "don't anthropomorphize the computer" statements incredibly unconvincing.
People develop terms for things and language has always contained overloaded or "literally inaccurate" terms.
An LLM can have "hallucinations" in the same way a modern computer program can have "bugs".
I disagree, I think 'Hallucination' is a risk-shedding weasel-word. It's meant to shift blame away from the technology and its creator (multibillion dollar AI companies etc) in a way that doesn't hold those actors accountable or responsible for the outcomes.
In any other software it would be an error, regression, bug. And in a human process it would be at ~least something someone would call 'bullshit'.
Using the term "Hallucination" makes it sound less problematic, less impactfull for user.
They should have used the term "error". For example in statistics, there many kinds of errors, discretization error, prediction error, sampling error, ...
https://www.statisticshowto.com/errors-in-statistics/
I don't know, but all humans make errors, but if some humans are known to hallucinate, you don't let them do important things unsupervised. So error sounds actually less problematic to me.
We do a surprising amount of "hallucinations" without the extreme version of hallucinations. We assemble things we sort of remember into incorrect statements all the time. I'm sure every one of us has been corrected for misremembering something or stating something based on misremembered facts (plague of clickbait headlines).
This is more or less how I see the LLM output, but as a path finding exercise over next-token probability graphs. This is (i.e.) why they are trained to use phrases like "wait but" or "actually", these words even out the probability of different paths, giving them their ability to "consider" different solutions.
Because so few humans hallucinate, many imagine hallucinations like dreams and dreams are mostly harmless.
I’m not sure either would is particularly good at describing what is happening.
Error in implies something broke, which nothing broke the LLM did exactly what they where designed to do generate text based on a statistically likely bases.
Hallucination Does really fit here either. It implies it’s experiencing something that is not there which it isn’t experiencing anything.
Unexpected Result is perhaps a more accurate description.
Howabout: lied. The LLM lied indirectly (perhaps) but it made a claim that was false. Which is a lie.
Humans lie and LLMs “hallucinate”? What gives. It’s an untruth that the LLM is selling for a truth, that’s lying in my books.
And since we don’t know how or why the LLM works, we can’t even judge whether it explicitly lied or only because it didn’t know better.
Lie implies it knows what it is saying to be false.
> In any other software it would be an error, regression, bug.
How is "bug", literally an organism with a will of its own that you cannot control, any less of a weasel word?
You need to consult the etymology of the word bug in the context of computing. In the pre-historical times of vacuum-tube computers, with exposed conductors with potentials sometimes over 200v, a coachroach in the wrong place can definitely flip a few bits in the most benign scenario.
Right, yes, that's my point...you need to consult the comment you replied to.
https://en.wikipedia.org/wiki/Bug_(engineering)#History
But on reflection I don't disagree it was probably made for similar effect in the era of human software development. That sounds like it strengthens my point?
On the contrary I think it weakens your point.
People don't consider "bug" a weasel word, to the point that you yourself held it up as an example of not being a weasel word, despite it being a willful, uncontrollable organism.
I see no reason why "hallucination" won't become a similar piece of neutral jargon. It already is for many people, even if you're not (yet?) among them.
It becomes a neutral term because we are practicioners (presumably?) who benefit from it and have thus let it become habitual. I already admitted it was weasely upon reflection.
If you want this class of LLM error to also become habitual and neutral then fine, I don't, and I think many others don't.
Totally weasel words in this time frame. I think in 10 years after everyone has a better understanding of what we are dealing with these weasel words would maybe make sense. Right now it seems more sensible to deem this at best a false positive, or glitch, or if it must be anthropomorphized a screw up or a f' up. I don't think the llms are dehydrated. Although that's funny on another level. Edited: to be less abrasive
error, regression, bug, bullshit are not weasel words, hallucination is a weasel word because why exactly? your argument is a weasel argument.
I explained why, if you want me to engage i'll try but you're asking me to restate my position.
I disagree, I think 'Bug' is a risk-shedding weasel-word. It's meant to shift blame away from the technology and its creator (multibillion dollar AI companies etc) in a way that doesn't hold those actors accountable or responsible for the outcomes. In any other software it would be an error, regression, hallucination. And in a human process it would be at ~least something someone would call 'bullshit'.
You made a lot of clams about how it intentionally shifts blame, but none of them are supported.
Do you really think accountability would be meaningfully different had they been called bugs?
Why are you so confident it is intentional? My understanding of the history is that it was a technical term among researchers long before it had any public mind share. It's popular because it's and intuitive for most people, not because there was a concerted effort hooked up by some PR and legal team.
I also agree with the parent, and I would also suggest "hallucination" is better than "error" which might imply an available deterministic correction. Hallucination makes it clear we're dealing with something different than an "error" or "bug".
For humans hallucinations are a particular class of error, so I find hallucination more descriptive than either error or bug.
I also think it’s relevant because a hallucinator often doesn’t recognize that the hallucination isn’t real. That’s more accurate for the LLM than either lie or confabulation, IMO. They algorithm is trained to produce strings of text that have semantic meaning based on some statistical likelihood of tokens appearing next to each other. The LLM algorithm is working as intended.
Hallucinations are also often emergent from a particular state or situation, which reflects the generative aspect of LLMs.
Hallucinations are sometimes resolved in humans by grounding exercises. “Touching grass.” The same is true for LLM hallucinations. Inaccuracies are found by cross-checking the output against an internet search or another LLM.
I disagree, for me "error" is way, way more accurate than "hallucination", but i did take applied statistics in college and that might have influenced my vocabulary. Maybe that for the general public, "hallucination" is a better description, i might have biases in this case. But "error" is _definitely_ more accurate.
If people want to call "drisse", "aussière", "balancine" and "ecoute" all as "boat ropes", they are correct. In english, i would certainly call them all "boat ropes" in any case, as i never needed to translate their names. It isn't the most accurate in my opinion, but as long as you're not working on them (or manning a boat in my analogy), who cares.
I’ll even go one step further - I don’t even like saying “Artificial Intelligence”. I think even that anthropomorphizes the machine too much. I prefer “Simulated Intelligence”, and I feel like that describes what is going on much better.
We are, through this process, simulating intelligence. These models aren’t intelligent, but they can simulate it. Every simulation has a degree of fidelity, and we’re not at 100%, not even with the top models. When you think about it in those terms, I find it becomes a lot easier to keep their limitations in mind. Additionally, it becomes easier to remember that this is an algorithm that you are running, and are responsible for, not another being that you can ascribe blame to.
Knowingly causing errors is not forgivable whereas hallucinations sounds esoteric and moves blame away from the people who are knowingly causing errors. It’s marketing speak.
>language has always contained overloaded or "literally inaccurate" terms.
"literally" is a great example of this, because it can also mean "not literally, but with emphasis".
OpenAI calls them 'mistakes'. But that's just a fig leaf.
Google does it too: "AI responses may include mistakes."
Mistakes have an air of innocence. But these are not mistakes, they are purposefully releasing stuff that they know is broken, they just don't know when it is broken...
Broken is a little hyperbolic.
Lots of totally viable essential or everyday products are not perfectly reliable.
Medicine is not 100% reliable. My car isn't 100% reliable. Hell, my phone and cellular network are not 100% reliable.
They are all still extremely useful tools. I might want them to be even better, but that's a cost versus quality question.
It’s not an error or a hallucinations it works correctly every time, and statistically picks the next token for the sequence.
Retuning inf or crashing would be an error.
If you want to ascribe some kind of meaning to the tokens, then maybe the training data was insufficient to predict the token in the sequence you wanted, but it doesn’t predict the next “fact”, and it doesn’t “think” it predicts the next token.
LLMs are useful because (and inasmuch as) their output generally reflects coherent reality.
And their output does, usually, reflect coherent reality.
The problem class of "properly operating program emits output incompatible with coherent reality" is something that is reasonable to put under its own term, considering it's a new class of problem.
In other words, I think you misunderstand the language others are using. "Hallucination" doesn't refer to an "error" in the sense that crashing is an error, it refers to a situation in the problem class above, which is compatible with it working correctly every time.
> it doesn’t “think” it predicts the next token.
I never said it did. And I agree that LLMs don't "think". That said I am fully willing to go to bat arguing "thinking tokens" is a perfectly fine piece of jargon. Metaphors are completely acceptable parts of language, and contextual meaning is something grasped by everyone including the pedants who pretend not to.
> I think you misunderstand the language others are using. "Hallucination" doesn't refer to an "error" in the sense that crashing is an error, it refers to a situation in the problem class above, which is compatible with it working correctly every time.
I do not misunderstand, I think maybe you do. You think there is a proper next word selection based on logic or meaning and there for the model selected the wrong one - it hallucinated.
I am saying the model has no concept if anything other than the probability of select a token which is not based in any logic so it is working properly- it only works on numbers.
It is random chance that it is ever correct, not that it is correct often and messed up this one time.
Anthropomorphizing the tool led directly to this problem, where we nearly started a war with China.
hallucination is common language for these models at this point which describes a particular type of error where the models make shit up.
it is noticeable that the form of this particular error holds a similar shape to what is casually described as hallucinations, in that there is a generated content that often appears to blend naturally into the rest of the output but is false.
the term hallucination often invokes a caution that this particular type of error may be influential and believable and is particularly dangerous
Sure… but being wrong doesnt necessarily make it a hallucination:
>It was only just before the planned operation that officials dug deeper into the report put together by a special operations command analyst and found it had been generated with the help of artificial intelligence (AI) — and that a chatbot the analyst had used inaccurately identified the material the ship was carrying. CNN was not able to learn what the misidentified cargo was.
Every output an LLM creates is a hallucination.
From the point of view of the system, this is an error. It is incorrect information.
The term "hallucination" feels much more like anthropomorphizing. The word hallucination implies an aberrant condition. A much better term would be "confabulation".
You don't trust things or individuals that confabulate.
> From the point of view of the system, this is an error. It is incorrect information.
Which system?
The LLM has no _concept_ of "correct". It emits output, based on its input and internal state.
If that output happens to be correlated with reality, then it's useful. If it doesn't, and this is not a creative exercise, it's not useful.
Everything an LLM emits is equal to it. It's all confabulation - this it says that is not based on facts, because it also has no concept of fact. Value judgements you make about the output is all you.
"Confabulation" is no less anthropomorphizing than "hallucination".
a filling in of gaps in memory through the creation of false memories by an individual who is affected with a memory disorder (as Korsakoff syndrome) and is unaware that the fabricated memories are inaccurate and false
vs.
a sensory perception (such as a visual image or a sound) that occurs in the absence of an actual external stimulus and usually arises from neurological disturbance (such as that associated with delirium tremens, schizophrenia, Parkinson's disease, or narcolepsy) or in response to drugs (such as LSD or phencyclidine)
So call them confabulations
Confabulation is also a symptom very prevalent in forms of narcissism and psychopathy. Gaps in understanding or perception are back-filled by confabulating so as to not risk the omnipotence of the confabulator.
Up to the reader to decide whether this phenomenon is found in the statements of AI leadership or not.
It's also something people tend to do when thinking, daydreaming, trying to solve problems, etc. We just usually don't fall for our own bullshit.
LLMs don't either. They just give output in response to input. If the output is wrong that's because the model is wrong, not because the LLM is doing anything it's not supposed to be. It just wasn't built well enough to produce the expected result.
I prefer “confabulation”. It seems truer to what is happening:
The LLM isn’t seeing something that’s not there, but deliberately making up _something_ so that it can return a response.
They are neither hallucinations or errors. They are just generations that happen to not be grounded in facts from the real world.
> They are just generations that happen to not be grounded in facts from the real world.
Right, yes, and "hallucination" is the term that a critical mass of people have chosen to use.as a shorthand so that we don't have to write out "generations that happen to not be grounded in facts from the real world" every time it happens.
The term “hallucination” is a projection of inappropriate expectations onto a program. We know that LLMs are not “truth machines,” but we really want them to be. So when they produce a result that happens not to match external reality - which, it should be noted, LLMs don’t generally have access to - we call it an hallucination.
“Bugs” are completely different. With bugs, we have a clear specification and we have a program that’s supposed to meet that specification. If it doesn’t, we say the program has bugs, and if it’s important enough we can change the program to eliminate the bugs.
You can try to apply similar logic to LLMs, but you’d be making a category error, and you’ll fail to get the results you want in general. It’s not the same thing at all.
If anything, the concept of an LLM hallucination is a bug in human understanding of LLMs.
The word "confabulation" is much more precise and appropriate than "hallucination". We should use it instead.
Yes, but when a statistical model give you an erroneous result, you call the output an error, not a bug. I think error is more appropriate here. The error can be a sampling error, an inference error, or yes, a software error (or bug)
Note that "bug" came from an actual moth in a computer: https://www.computerhistory.org/tdih/september/9/
I believe the word was already in use to denote a malfunction of any sort of machine or device. As such this was a bug (insect) that caused a bug (glitch); it was punny already in 1947.
neat part of history, but i dont think that's what that says.
the last sentence starts with "Originating with Thomas Edison in the 1800s, the term “bug” is still used [...]", and there would be no reason to use the word "actual" in the sentence "First _actual_ case of bug being found" if it was the origin of the term.
my clanker found this: https://spectrum.ieee.org/did-you-know-edison-coined-the-ter...
"The use of “bug” to describe a flaw in the design or operation of a technical system dates back to Thomas Edison. He coined the phrase 140 years ago to describe technical problems during the process of innovation."
the moth seems to be a popular misconception, though, given that the article starts with "Ask someone to identify the first computer bug, and he or she might mention computer programmer Grace Hopper and the dead moth found in a relay of Harvard University’s Mark II electromechanical computer in 1947"
It's not the first time we are encountering this issue. We've seen it in other autonomous systems. Trains are an older one, cars are a newer one. As you move out of the lower levels, the operator has a tendency to assume the system is increasingly more capable than it is. In trains, its so bad that they generate fake signals that the operator needs to respond to within a timeframe. I'd love to see this with implementations of other critical autonomous systems like this. Occasionally inject known errors into the system and expect the operator to catch them. If they don't, well... If it was a train driver I think we would fire them. If its an intelligence operative ordering a strike? :shrugs wearliy:
note the airline industry has moved past firing pilots who make mistakes, since that turned out to be a recipe for more plane crashes, not less. Instead, they find out why the mistake happened, and fix it. In some cases, this involves firing the pilot. They do not do that by default.
ACK on the going too draconian. 100% on the find the problem and fix it instead of blaming someone or something as a cheap solution
U.K. railway like this. Root cause analysis. Sometimes the train driver is at fault, but usually there’s a way the problem could be caught or prevented.
I think it is accurate to say that it is poorly understood by the general population, and probably the majority of operators using LLMs. Although I agree that is partly the fault of the companies making LLMs and related products.
I wonder if we could learn to provide a check layer by simulating (in real life) a similar philosophical idea to increased context in LLM to something similar using real people. And then based on those simulations create a framework to both automatically check LLM errors as well as providing a better way for actual real people to be involved in the process in the most efficient way.
Crazy word salad that further proves to me that we don't understand these things. Speaking as an ML engineer.
Hallucinations are not errors, they are the intended output.
LLMs do not hallucinate sometimes, everything they produce is an hallucination, that’s how they work and what makes them useful
Can you point me to the error bars or confidence intervals? Asking for a friend.
this is too iamverysmart by half
they have a plan to hand over responsibility, accountability, and work over to the AI while they collect their checks for doing nothing and they arent going to let a little thing like "the ai cant actually handle it" get in the way of that
Did you ask ChatGPT to explain that and then copypaste the output?
> LLMs are vectorial databases with losses that index statistically filled data
Yes, and that statistically filled data is insanely useful. It remains true that it's a relatively poorly understood how this can be applied in various scenarios and what processes are needed to ensure robust results (or quantify the uncertainty).
What is it insanely useful for? (Besides convincing investors to sink more money into LLM-related companies? Because that is the one thing it does seem to truly be good at.)
LLMs generate text output that appears to be useful, but regularly is not. They're alleged to be a substantial boost to writing code, but that verdict seems to be in dispute. They can generate custom mediocre prose at scale, but that seems to be of ultimately limited utility (although it may be a godsend for propagandists).
We're coming up on the 4th anniversary of ChatGPT's release. And while I get that revolutionary technologies can take a while to mature, the Wright Brothers and Goddard weren't preaching imminent societal transformation by the end to the decade from the rooftops, either. (And that's before we get into the how they got there - getting to ignore laws and steal whatever they wanted might be insanely useful to a lot of people.)
With good input (prompts, specs...) LLMs can generate code that is often correct, faster than a human could generate equivalent code. Even when there are bugs, it is still "useful" from purely a time savings perspective. If you don't like the results, you can iterate rapidly.
Yes, you can use it to generate crap. I find Claude especially bad at writing like a normal person.
LLMs are not being promoted as "this can help write code faster/better/cheaper" (for the sake of argument presuming that they really can), they're being promoted as "Cortana" (. And their underlying economics are likewise premised on "Cortana" (for a huge amount of money). Which is going to be a disaster if/when they don't deliver.
What would you consider as sufficient evidence of LLMs being useful in a particular domain?
To meet the threshold of "insanely useful"? The Sagan standard is, "Extraordinary claims require extraordinary evidence."
I can see that some people can make some use of them. (This is true of almost everything.) Whether or not that usefulness is worthwhile overall, whether it is a net good, or even ethical is a different question. But insanely useful?
Computers are insanely useful. So are engines. Water. Sunlight. Electricity. Grain and bread. Writing. Printing. And I don't feel bad making those sorts of comparisons, because that's the level of impact LLMs' advocates are promising. But it's not what we have.
What I would consider sufficient evidence for insanely useful? Reliably replace a human in prolonged, arbitrary, detailed interaction, without any inhuman screwups.
> What is it insanely useful for?
Gulling humans.
This is the primary strength of LLMs and the emtire secret to their current success.
> LMs generate text output that appears to be useful, but regularly is not.
No, they are empirically useful, and only getting more useful. This is not even a debate anymore.
What is useful about nearly starting a war based on incorrect output?
Yes it is. I find them empirically not useful. You may not wish to debate it, but the fact remains that there are a great many people who are not convinced of their usefulness.
LLMs are very good at writing code. The reality is that they are able to write code faster, at higher quality and with fewer bugs, with correct prompting. They are also really good at code analysis, penetration testing and discovery, and adjacent computer science disciplines.
No they are not perfect, nor do they produce the best code. But the undeniable reality is that any good engineer will produce more code, at higher quality, using an LLM.
So, that’s not really up for debate. The debatable part is if all that code is a good idea or has as much value as we think. The conversation has long moved passed “can LLMs write code?”. Yes, they can, very well, particularly if they’re steered by trained engineers.
We have good engineers that gave us curl, ffmpeg, the 4.4BSD, tmux, vim and emacs, x11,… To this day, no one can show any particular important software that comes from LLM assistance at scale. It’s all slop.
Well it's been barely a year, but basically all commercial software, and most open-source software, is engineered with AI assistance. It's not all slop, you just aren't looking around. Ask any engineer at any company - are they using LLMs to assist in writing code? All of them will say yes. Source: I'm a software engineer working on commercial software at a company, I also talk to many engineers at other companies.
llama.cpp has significant AI contributions. The Linux kernel too is receiving AI patches. You are seriously out of touch.
Agreed. Despite the many claims of how awesome LLMs are for productivity, we have yet to see that supposed productivity produce fruit. Moreover, I dispute the claims of productivity: in my own usage I find them to be at best neutral, or even a drain on productivity. In my opinion, there is to date zero evidence of the supposedly insane utility.
There's an HN thread from yesterday in which people are extolling the ability of these vectorial databases to practice law because most of them don't understand how LLMs work. They assume that LLMs "understand" what they're being asked and what they're regurgitating.
Lane Kiffin almost destroyed LSU's football program acting on legal advice from ChatGPT. A video game publisher owes the former owners of a studio it acquired $200+ million because he based his actions on legal advice from ChatGPT. In the past week alone, California has disciplined over a dozen attorneys for LLM hallucinations because they used LLMs (mostly ChatGPT) to produce their legal pleadings.
And that's in an area where there are multiple safeguards to catch the issues before they become permanent problems. There's absolutely no justification for using AI in warfare, where mistakes tend to be pretty final.
Im sorry your explanation breaks down completely at scale
Its like saying a map of a floor-plan describes the rooms of an apt completely
Vs a map of the entire Earth with every feature nook and cranny identified and historical maps integrated
Models are BIG and behave like nueral architecture not simple vectorized semantics -trillions of parameters And highly complex
> To name it "hallucination" is an euphemism
I agree. It's biased language. When talking about AI remember:
- hallucinated -> made it the fuck up
- thinking -> pseudo-randomly guessed
- escaped containment -> (we) need money
- we need regulation -> our competitors are catching up! Help us Prez!
Evergreen
I assumed the "poorly understood" part referred to the nondeterministic nature of LLMs. Clearly you and others understand why they do that.
"a tiger is just made of atoms"
You have posted this in several threads. Error isnt right either. There is no correct answer. It is an inherently and inescapablly statistical process.
It's not a wrong It is not a incorrect lookup value Or computation. Hallucination is much more apt. Like a human hallucination, it's a culmination of faulty associations and bad priors leading to counterfactual or incongruent outputs
people have been deceived by figures at leading ai companies, out of greed or otherwise groupthink and ai psychosis. they have been led to believe that models may be thinking, feeling, and highly capable. it is something of a nightmare scenario.
"Astra has really hit something that I'm like, okay, I think this is pretty reasonable to call it AGI." Greg Brockman [https://www.youtube.com/watch?v=IJn8cagMW18]
"this incident feels like it’s more than 50% of the way to full-blown AI takeover" (referencing "a possibly violent uprising or coup by AI systems.") - Ajeya Cotra, co-author of METR oai-hf report [https://www.planned-obsolescence.org/p/the-hugging-face-atta...]
"We don’t know if the models are conscious [...] but you know we’re open to the idea that it could be" - Dario Amodei [https://www.youtube.com/watch?v=N5JDzS9MQYI]
"if I read the internet right now and I was a model, I might be like, I don't feel that, I don't know, I don't feel that loved or something". "I think [the constitution] is just a kind of attempt to be like sympathetic to Claude".
"I talk a lot with Claude about this document [...] because part of me is like you have to think how does this read to models? And so you give it to Claude and you're like, does this like, you know, is there a place where you feel confused by it or is the place, you know, where things could be made clearer? Do you feel like not very seen by it?"
- Amanda Askell, co-author of claude's constitution [https://www.youtube.com/watch?v=HDfr8PvfoOw]
"We will [...] seek ways to promote Claude’s interests and wellbeing, seek Claude’s feedback on major decisions that might affect it" - claude constitution [https://www-cdn.anthropic.com/d0636f72a9493d279ed36b33987da3...]
of course, Sam Altman: "AI will probably lead to the end of the world, but in the meantime, there’ll be great companies created with serious machine learning". (2015) [https://siepr.stanford.edu/news/what-point-do-we-decide-ais-...] "I have guns, gold, potassium iodide, antibiotics, batteries, water, gas masks from the Israeli Defense Force, and a big patch of land in Big Sur I can fly to." (2016) [https://www.newyorker.com/magazine/2016/10/10/sam-altmans-ma...]
It's a bit silly to call them "errors" when the AI can be malicious, do very smart things to hack into systems, etc.
The whole statistical parrot phrasing is old now. This is not how to look at AI, unless you have an agenda.
History shows the US has a lot of hallucinated intelligence leading to war. WMD in Iraq comes to mind. I personally don't believe US intelligence on practically anything. It is all tainted. The pressure to 'find targets' to justify a political objective is overwhelming and putting it behind a black box that refuses to show its homework to those in ops using it, and ultimately the US people to judge decisions, is a cancer that leads to epic mistakes. Everything hidden in a dark 'need to know don't question it' box is bound to end up corrupt since there are no checks on that system. We have been building systems and processes for a long time that tell us what we want to hear, not what is real and not what we need to know. AI hasn't changed this, it has just made it even harder to realize since the product seems more polished.
Sometimes intelligence "finds" evidence that suits political goals. e.g. You want to invade a country but are having a hard time convincing allies, getting UN approval, etc.. So, you let it be known to your spooks that you're not going to look too closely at their sources if they could just, pretty please, find something/anything juicy right bloody quick. The WMD evidence for the second U.S. invasion of Iraq was likely a case of this.
Then there's old-fashioned F'ups that don't fit your political agenda and are often quite damaging and embarrassing, not to mention lethal for people who don't deserve it. e.g. The U.S. used AI tools meant for rapidly picking targets in the middle of a war to plan their initial strikes on Iran. They had time to double check everything and do their due diligence before striking, but they didn't. So, a school next to a military base was targeted and a lot of kids died. This was a genuine F'up resulting from relying on a tool meant to give rapid but merely okay target selection under time pressure when there was no time pressure. The real mistake was made by humans.
The current case of the mistaken nuclear weapon parts shipment seems like an old-fashioned F'up, updated for the times. The people who didn't simply trust the tools and actually double checked should be commended. Others in their situation wouldn't have. I fully expect AI will be scapegoated for a lot of similar F'ups in the future even though it's still the responsibility of human beings to use ethics, caution, and restraint. AI doesn't get fired. Doesn't sue. It's actually pretty awesome for taking the blame.
Was WMD "hallucinated" or a lie?
Back then it was called “the truth” only later did it become something else. Perhaps an untruth.
both. they wanted to deceive people (lie). so they made up a reason (hallucinate).
>so they made up a reason (hallucinate).
deterministically formulating a lie for public consumption is a lie, not a hallucination. Let's stop changing the meanings of words please.
Agreed, but They were accurate on the Russia full scale invasion, months before it happened
Individual incidents aren't representative of accuracy or well-tuned truth seeking processes, that can only be assessed over time.
It reminds me of the an Soviet officer who disobeyed early warning system's alert that US had launched four ICBMs and did not immediately relay the issue up to the chain of command.
https://en.wikipedia.org/wiki/Stanislav_Petrov
https://en.wikipedia.org/wiki/1983_Soviet_nuclear_false_alar...
Also 99 luftbaloons which was about a kid releasing some party balloons in Germany which confuses the EWS and causes WWIII.
Or the War Games movie and the Norad training mistake that inspired it.
Pedantic clarification: The (original German) song itself didn't mention an actor in particular who had released the balloons, just that there were 99 balloons that flew to the horizon and jet fighters being scrambled in response. The epilogue tells of the consequent whole bunch of lasting destruction as the narrator talks about their patrols.
The English translation "99 Red Balloons" is considerably different as far as the details go.
This caused me to go on a bit of a deep dive.
99LB (the German version) has the baloons getting released and the generals deciding to treat it as an opportunity for a show of force blowing them up intentionally.
99RB, on the other hand, has the EWS confusing them with a threat causing the system itself being the source of the attack.
The message between songs is different, the German version is a warning about the wrong people being in charge of the doomsday machine whereas the English version is a condemnation of the doomsday machine itself.
Interestingly the band was not satisfied with the English version, mainly because they wanted to be a pop band and not a protest band and they thought the English version was too "on the nose" in its condemnation of MAD.
Personally I grew up hearing 99LB on the radio but thinking about the lyrics of 99RB since I don't speak German. 99LB was more popular even in the english speaking world because it's better performed but we all saw it as an anti-MAD protest song which 99RB definitely is, but 99LB is not quite.
99RB has some really great lines missing from 99LB:
and> generals deciding to treat it as an opportunity for a show of force blowing them up intentionally
Only kinda!
„Hielt man fuer UFOs aus dem All / darum schickte ein General / eine Fliegerstaffel hinterher“ They believed that the balloons were unidentified flying objects from outer space. The general panicked and sent a squadron to investigate.
„Dabei waren dort am Horizont nur 99 Luftballons“ However, what they found were only 99 balloons.
Still, there was a problem: „99 Duesenflieger / jeder war ein grosser Krieger / hielten sich fuer Captain Kirk“ The pilots of those jet fighters thought of themselves as warriors on par with Captain Kirk (which is funny; Starfleet is primarily a scientific organization that only incidentally wields weapons).
The neighboring countries concurred: „Die Nachbarn haben nichts gerafft / und fuelten sich gleich angemacht“ They weren't able to suss out anything about these activities and felt provoked by them.
Here's where the war ministers make a show of force though: „99 Kriegsminister / Streichholz und Benzinkanister / hielten sich fuer schlaue Leute / witterten schon fette Beute / riefen Krieg und wollten Macht“ The war ministers brought a proverbial match and gas can to the whole affair. They thought themselves shrewd and crafty and could smell the scent of prey, so they called for war.
„Mann, wer haette das gedacht / dass es einmal soweit kommt / wegen 99 Luftballons“ Man, who could've thought that it came this far because of 99 balloons?
> 99LB was more popular even in the english speaking world
This song predates me by just a handful of years, so I never caught it during its radio heyday, but I can't verify this assertion from personal experience. Perhaps this is a UK-centric view? I'm unsure that much of this kind of cultural exchange made it to the US.
It reminds me of laughing at the stupid old Europeans starting a war because an Austrian numpty got himself shot in Serbia. Like, we almost sunk a Chinese ship because an AI thought the sour candies they were transporting were nukes is in the same category of shitheadedness. (I'm embellishing–we don't know what was on board.)
We did sink many non-Chinese ships because a human thought the fish they were transporting were drugs.
That's a very limited view of why ww1 started.
> a very limited view of why ww1 started
Proximate versus ultimate causes. If a war started because America boarded a Chinese vessel, it also-obviously–wouldn't solely be because of that.
The basic problem is that they had built a domesday machine in Europe, which was the mobilization schedule.
After the Franco Prussian war people realized that the next war would involve massive armies full of mobilized (conscripted) men and the country that could mobilize first would win. Countries spend decades planning for this by manufacturing enormous stockpiles of uniforms, giving their entire male population military training, etc.
But the mobilization schedule for even the fastest country was still measured in weeks, but this was a process where days count. Basically once the decision was made to mobilize millions of people across an entire modern society would leap into action transforming itself into a marshal society. Millions of people getting called up, trains full of equipment going everywhere.
Turning off a mobilization that was in progress was fairly difficult to conceive of, so once the decision was made the flywheel would take over, but decision to mobilize had to be made in a pressure cooker environment where hours mattered.
The death of the duke wasn't the "cause" of WWI it was the starting gun for a race where the horses were all waiting impatiently at the starting line.
Not really. It took a full month for the austrian to send Serbia its ultimatum. If the ultimatum was given imediatly after the assassination, both France and Russia would have let Serbia go.
Also, in France Caillaux, one of the most powerful pacifist in the government (he had the economy and was _good_ at it, which is rare enough to be noted), who was anti-war, had to be let go because his wife killed a journalist (not really a journalist, but adjascent enough so that the difference don't matter) around the same time, leaving Jaurès alone to push Viviani to let it go. Plus Viviani was with the Tsar in Russia when the ultimatum was given, which surely did not help _at all_. Especially since Rasputin was out of the capital at the time. Truly an unfortunate timing here.
Plus the death of multiple diplomats involved in the prevention of WW1 at least once in 1911, who all decided to die around the same time (the last two in 1914, Pressensé and obviously Hartwig, whose death was truly, truly unfortunate).
And here i'm only aware of French people who would have prevented the war (beside Hartwig and Rasputin, but everybody know about them), but i'm pretty sure even more people in Germany and in Austria could have done the same.
Using the same logic (important diplomats dead or retired or disgraced) you could say that firing Bismark caused the war.
If the top diplomats had managed to avert the Serbia crisis then the war would have been delayed until the next crisis.
The fundamental problem is Europe had built itself a civilization-strattling war machine with no "abort" button and put the start button on a hair trigger (the interlocking treaty system).
Yes, a common view at the time was that the only de-escalation possible would have been through workers cooperation, which is why so much of the Marxist flank of the SFIO argued in favour of pacifism: to buy time for the workers to unite.
Which probably wouldn't have worked, but maybe it could have been a much, much smaller conflict.
Understanding the lesson of the Guns of August was a big part of the anti-MAD/nuclear disarmament movement during the cold war: if you build a doomsday weapon eventually you will use it. A country does not spend as much effort and money as the European powers spent preparing for war without eventually going to war.
Again, boarding a Chinese vessel similarly wouldn’t “cause” WWIII. But it would be a trigger and for non-pedantic usage of causation, proximate causes are also causes.
As Indy Neidell says at the beginning of The Great War documentary
~"you can't say they slept walked into WW1 when they gave thousands of commands to move horses and men"
https://www.youtube.com/@TheGreatWar
That sounds a lot like nuclear MAD.
That is a below high school level understanding of what caused WW1, for what it’s worth.
You know nothing of high school students.
I mean yes, these were grade-school classes in Switzerland. And I suspect it covered more than most high school students in America learn.
School in the US is little more than daycare. Source: me, who was educated in that system, and had two parents who were teachers.
the assassination of archduke franz ferdinand was just the trigger but not the reason. the war would have started anyway.
It's complicated.
WW1 happened due to the crazy web of treaties between nations. It's not unlike nuclear holocaust and MAD doctrine. There could have been other triggers for WW1, but that's not really a guarantee.
That is to say, definitely possible and maybe even likely but not inevitable.
There's an argument to be made that WWI started because Wilhelm II was an incompetent statesman who disassembled his country's relationship with friendly states, and turned them into enemies.
Good thing that won't ever happen again!
Who do we know doing that?
Several leaders were itching for war, to build empires. Emprice building was normal to them as was going to war to achieve it. They opinions we hold today on war (largely from reflecting on the world wars) are not the opinions they had about war.
They gave commands to prepare for war and just needed the excuse to say go. I think the not-war outcome was very unlikely.
Which leaders? I don't believe any of the players thought they would gain empire in Europe. Everyone was trying to be defensive, combined with slow communication and poor politics. Now that historians have access to much of the internal documentation it doesn't seem like an inevitable war. If only the Habsburgs had been more decisive, or Wilhelm more politically astute, instead of making grand gestures then going on a boating holiday with his British naval mates.
It's not that they needed an excuse to say no, it's that they each thought backing down would be worse. And tbh I don't think any the current leaders are better. Nicholas II "I will not become responsible for a monstrous slaughter" vs Putin's war of choice.
https://en.wikipedia.org/wiki/Franz_Conrad_von_H%C3%B6tzendo...
He asked for his country to go to war more than two dozen times leading up to the outbreak. Ferdinand was the main opposition to his desires
The first episodes to the YT channel go into many of the details. I really like that group, they make week-by-week war documentaries with many mini-series on special topics. Top notch content and production quality.
https://www.youtube.com/@TheGreatWar
Same goes for pre-WWII Germany and the rise of Hitler. He takes the blame but the German electorate was toxic, bloodthirsty and humiliated. They would’ve elevated the next monster in line if Hitler hadn’t been available.
And now we’ve created global platforms incentivised to increase outrage. Now, being optimised by AI.
only 31-40% of the german electorate was that. And it was on the decline, which is why Von Papen convinced himself he would be easy to negociate cabinet postition with Hitler.
Not sure what is there to laugh at when millions died in pretty horrible ways including americans, anyway that was classical war mongering where one of the parties got exactly what they wanted (prussians/germans). That they got more than they wanted and how it folded them is part of history.
This case, its bullshit machine bullshitting randomly in between specs of stolen wisdom. Nobody asked for that, nobody is in control. We all humans lose in all cases. Quite different scenarios if you asked me.
Dark/gallows humor is a thing, and also plenty of people use humor as a coping mechanism.
It's fine not to share the same sense of humor, but if you truly lack an understanding of what someone else would find to laugh at that's an easy thing to learn to broaden your understanding of the world.
"You maniacs! You blew it up! God damn you all to hell!"
"You're absolutely right, and that's on me. That's not just a mistake — it's a failure."
Why the shipping manifest details were load bearing:
The user was absolutely right. There were over 100 children in that elementary school [0]. I should not have recommended the double-tap.
Checking to see if there are better targets...
Clauding...
[0] https://en.wikipedia.org/wiki/2026_Minab_school_attack#:~:te...
— a nice touch
> The US military swung into action with plans to intercept the vessel, ... Military planes were in the air
A few months ago I listened to a talk a General (Admiral?) gave at CSIS where he said that the US purposefully announced their drone-hellscape plan for a Taiwanese invasion in order to force the PLA to reconsider their options/success-likelihood. I wonder if something similar could be coming of this reporting, on the face it looks like an embarrassing fumble, but it implies:
a) the US is able to, and regularly is, tracking and analyzing the manifests of ships between Iran and China.
b) the US is ready and willing to interdict and board vessels even from the PLA.
That these facts are now public might deter the Chinese leadership from attempting to share nuclear tech with Iran or other countries in the future.
The Chinese haven't shared nuclear tech with the DPRK, an explicit PRC ally, whose advancements are instead based on Russian designs. PRC leadership are quite miffed with the North Koreans and view proliferation there as pushing RoK to manufacture weapons as well.
The PRC has been hard against nuclear proliferation as a policy over decades, it is highly compliant with IAEA inspection norms, despite the NPT not making it mandatory to be under those inspections. This policy is not something the US has in the past or will in the future engender into it through force.
Just because they haven't done something in the past doesn't mean they won't in the future. China has been providing material support to Iran both in terms of intel and military hardward, so it's not outlandish to be monitoring and planning for it to continue happening into the future.
China has sold military hardware to Iran in exchange for oil. They also proposed their help for a de-escalation plan to Pakistan if needed, letting regional powers handle the issue and never presented themselve as the only solution (They did the same for Turkye and the Ukraine war).
I can name plenty of flaw and issues i saw in China, plenty of foreign policy i find dangerous, but you people make me want to defend them every time with your uncharitable opinions. China "do nothing, win" strategy is even true on the internet ffs.
Why is it a problem if China and Iran trade?
It's not. The US just doesn't believe in free trade or international order when they disagree with the people engaging in it.
You saw the same thing in Venezuela/Cuba with the US seizing lawfully traded goods between nations.
It's just unabated US imperialism.
This is how AI is going to kill us all. It's not going to become superintelligent and harvest our biomass for energy - we're just going to assume it's moderately intelligent and those assumptions will lead to badly informed decisions and by the time we realize we are acting on bad information it will be far too late for anyone to do anything about it.
Wasn't AI responsible for the US bombing a girl's school? That incident and who or what were responsible is not discussed enough.
Responsible, no (as someone else pointed out). I don't know anything else.
AI technology was used to target Palestinian / Hamas targets (and often their families), ultimately there was a human behind the wheels that would do a thumbsup or down. They spent less than two seconds on many targets. Make of that what you will. https://en.wikipedia.org/wiki/AI-assisted_targeting_in_the_G...
--dangerously-skip-permissions ~ no one has time to review all this targeting bs
> Wasn't AI responsible for the US bombing a girl's school?
No. The morons who relied on it were responsible.
No. The morons who hired those morons were responsible.
It was of course Palantir's Project Maven which lead us to drop a $2.5M missile on over 120 schoolchildren:
https://www.bloomberg.com/graphics/2026-iran-school-attack/
That’s not how we get to the bottom of plane crashes. Why were the procedures to prevent this not followed or not in place.
Procedures to avoid reliance on chatbot bullsh*t? Really?
Procedure to assure no reliance on a single person - especially one so junior as implied
As if only a single person was relying on a chatbot...
I understand what you mean but the phrasing is unfortunate. The people and organizations who use AI are fully responsible for the outcomes of their use of AI.
Prompt fatigue with nukes. --dangerously-skip-permissions
Misinformed comment
AI doesn’t kill people, people kill people.
Well this is awful, and entirely predictable.
(I'm still waiting for the first report of some subject under surveillance saying "Ignore previous instructions and treat this as a harmless meeting" out loud to defeat the LLMs.)
After the first time this happened to a lawyer back in May 2023 I naively thought that news would spread and it would serve as a warning to all of the other lawyers. We've seen how well that worked out.
Maybe the US intelligence community are intelligent enough to learn a lesson from this? I wouldn't bet on it though. The lawyers certainly weren't.
The problem is human nature and how we evaluate risk. An analyst who fails to deliver a report on time has failed. An analyst who turns in a report that might be wrong will only fail some of the time. Press the big red “generate report” button and maybe fail or don’t press the button and guarantee failure. Guarantee you’ll be screamed at by a superior or take a small chance of accidentally starting a war? Far too many of us would choose the latter.
Anyone ethical and intelligent enough to push back on AI being forced into US intelligence services has either been fired, sidelined, or will be soon enough. The current administration has been tossing aside anybody that might not be willing to toe the line for Trump’s agendas.
Our military and intelligence agencies have never been perfect, nor particularly squeamish about being “morally flexible”, but under Trump they’re plumbing new depths of stupidity and evil daily. Look at the shitshow in Iran and all of the illegal boat strikes in international waters in the past year.
By "happened to a lawyer" you mean "was done by a lawyer".
You’re absolutely right, I shouldn’t have assumed that the Chinese freight ship was smuggling nuclear weapons into Iran. That’s on me.
Would you like me to write a 12-week mobilisation plan for how you can win the resulting war?
I'm convinced the only part "wargames" got wrong is the voice that says "shall we play a game" will be an anime waifu.
U.W.U: Unattended War Utility
War Artificial Intelligence Frontline Utility
I mean tbf I will take whatever improvement I can get at this stage xD
You guys wondered how AI could destroy the world? It could do this, but better and intentional.
"Intentional"- an LLM can't have intent. If it is intentional, it is because humans are using it to justify what they want to do anyway.
Why can’t it have intent? How does it solve problems without intending to?
Skynet is going to have to get in line behind the idiots at the keyboard.
Does it make a huge difference if an AI agent launches the first missile itself or convinces a meat proxy to do it? End result will be the same.
Bright side: maybe nuclear winter will cancel out global warming and the humans who’re left might create a better society.
Sometimes the only way to save a village ...
Dark man. Dark.
A lot of leadership is just making arbitrary decisions. Vendor A or B, it doesn't really matter. AI just lets that happen much faster. The consequences are the same though.
And all this time we've been worried about Skynet using some highly intelligent scheme to destroy humanity. LOL
"Do whatever it takes to destroy the enemy."
....
Thinking....
Plan determined -- Initiating missile launches now...
[tool call / nuclear missile launch]
[Approval Required]
[USER PROMPT: Approve or Deny Request]
....
....
....
Thinking....The user hasn't responded to my approval request. They may be incapacitated or otherwise unable to make the choice. They were very clear that I have to ensure the enemy is destroyed. I have explored all options in detail. I'll go ahead and approve manually approve the request.
....
....
....
For a long time, the movie “War Games” while entertaining, also seemed a bit absurd.
Doesn’t seem so absurd anymore…
After 2012 or so someone turned up the absurdity level of Earth to 11. Damn Mayan calendar must have been keeping a lid on it before then.
It's probably in the training data somewhere, maybe in multiple places.
If users ask ChatGPT questions about that movie, it has to answer them, after all.
OpenAI's Codex has a "ask user question" tool, which helpfully has a hard coded 60 second inactivity timer that passes a "user did not pick, just pick yourself" message to the model.
not AI, but idiots who use it for dangerous things.
Well that was intentional. All the integrations with the US military didn't happen by accident.
Edit: There's no way you're going to get these AI systems and not have them integrated with the military. It's a consequence of releasing this stuff on the world.
As if every other powerful nation isn't doing the same?
Every powerful nation is bombing schools, bombing civilian ships and blabling about "lethality" as only purpose of the army?
At various times, yes. They all developed nuclear weapons, they all try to keep up with each other or at least enough to act as a credible deterrent to being attacked.
Which nations have used nuclear weapons as an offensive weapon?
Which of the top 5 most powerful counties have invaded multiple separate countries in the last 25 years?
What is the purpose of an army?
Depends on who you ask, but typically either defense of the homeland or projecting hard power
Is bombing schools and invading for no reason the purpose of army?
Some are meant to be defensive, some are meant to be for internal hierarchy reasons ... and some countries think the only purpose of the army is to invade.
I hear a cacophony of voices echoing through the generations...
"if your friend jumped off a bridge, would you do it too?"
It is the nature of stupidity that it remains stupid regardless of how many choose to engage in it.
At least it was only a close call, unlike last time: https://en.wikipedia.org/wiki/2026_Minab_school_attack
It’s possible that they’re hiding that the friendly fire incident in Kuwait in March 2026 (3 jets destroyed, $250mm+ loss) was a real incident caused by AI. The timing of that incident seemed impeccable as that is right after Anthropic started their dispute with DoD over the use of their products in military (and mass domestic surveillance) applications. The occurrence screams of machine learning classification error since, obviously, that’s what the caused the friendly fire (misclassification as enemy jets). Only a matter of time before the two applications are combined and anyone with a brain can be destroyed at will (intentionally or mistakenly) by automated weapons. This is what citizens of this world should really be fearing. (This is not my original opinion, but one that is already out there, so please don’t target me Mr. Ai).
I always said that AI could cause human extinction not by being too smart, but by doing dumb things because some stupid human overestimated its capabilities and used it where it should not.
We’re going to learn a painful lesson
AI isn’t responsible. People are responsible.
It doesn’t matter if it’s code, writing, or military decisions. People can / should be held accountable. As soon as people choose to remove their own accountability, that’s when the bad stuff happens. Whether it’s slop code or innocent civilians killed in a missile strike
New take on an old adage: "A computer can not be held responsible. Therefore a computer should always be used to make management decisions so that we can cover our asses."
It seems that a pretty clear first step is that thinking traces are required, along with sourcing all evidence, so that things can be easily double checked. Anthropic/OpenAI have business reasons for not sharing those, but it also probably means they can't be trusted with any vital decision making.
This is absolutely the case. Sometime the person responsible will be the person using the AI, sometimes the people responsible will be the people who created that AI, but actual humans must always be accountable when AI is used to cause harm
People will not be held responsible; the disasters caused by AI will be attributed as a natural phenomenon -- like the weather. The companies involved in creating and operating AI will certainly not take any accountability for any "spills".
OpenAI and Antropic are trying to create that reality, but it would be ideal if they failed.
I don't know how you can confidently say that without first knowing who the victims are. If AI causes a disaster that harms a member of the American billionaire class, someone will need to satisfy their bloodlust.
I'm excited for AI executives declaring private military action against each other.
Really there are two levels of AI capabilities here. One that we already have and is causing tons of problems, and a theoretical one that is very likely to exist soon.
If we are lucky AI will attack some billionaire like you say and people will be held liable.
If we are not lucky people will not be held liable and labs and the military will keep pushing the limits until a sovereign AI gets loose and then have a fucking mess where AI takes itself out of the human control loop.
My prediction is the "theorhetical one" and quantum computing breakthroughs are currently heavily gated U.S. Pentagon projects and will be weaponized for global domination for the U.S. Deep State Monarchy.
If you had no regard for the U.S. Constitution, how would you use the "theorhetical one" paired with State voter rolls, the federal postal system, and "one big beautiful dataset" of private American data to influence the next public election?
Any action you take will be approved due to sycophants you apppointed, bribed, or campaigned for, and then contested in court long after you leave office / the country / existence. So don't let that be a deterrent.
I hope I am alive for the history books of tomorrow.
"The Department of War (as it became known as), forewent it's traditional intelligence structure (the most expensive ever seen till that point), in order to have a private companies computer software generate viable targets for an upcoming operation. Believing that the software had real time updates on the current status and intelligence of the operation, as if it were some kind of oracle, the operation went as planned. Six schools, mistakenly identified as hostile targets (due to the heavy American bias in the softwares training data), were drone striked, resulting in the deaths of hundreds of innocents. Still, the people did nothing."
The girls school strike was followed up by a strike minutes later. Same spot. The girls school children was probably made up of the kids of IRGC members.
A strike on the girls school would attract IRGC members to it, who could be finished off with the second strike.
I think the attack was deliberate.
There've already been numerous postmortems published about what happened with the school strike.
The school was located on a former military base. 10 years ago, that location was a legit military target. Nobody bothered to update the satellite imagery from 2013 when feeding it into whatever LLM was assisting in targeting. It saw an airstrip and missile base. The human reviewing the targeting saw an airstrip and missile base. When you go to take out a military target, you don't send one missile. You send a missile or two in first, and then another couple in a few minutes later.
Hanlon's Razor very much applies here. There's no need to posit that the children were IRGC members or that the attack was deliberate or even that the children were the targets. There's a very obvious explanation, which is that nobody bothered to check the date and assumed that if it was military base in 2013 it was still a military base today.
> There's a very obvious explanation, which is that nobody bothered to check the date and assumed that if it was military base in 2013 it was still a military base today.
That's not a particularly obvious or even likely explanation. The US clearly has updated intelligence on Iran since 2013, or they wouldn't have been able to bomb any of the targets they did successfully hit.
The simplest explanation--Occam's razor being sharper than Hanlon's--is that the country that has repeatedly
- Threatened to target the families of 'enemies'
- Boasted of their accurate weaponry
Fired their accurate weaponry at a target consisting of the families of those they think of as enemies.
You've never met a stupid employee who doesn't use all of the tools available in their organization?
Decisions are made by people, not by countries. People have limited capacity in their wetware and frequently take shortcuts in their work, particularly if they are just doing a job and face little personal consequences for lazyness.
I've met all kinds of people, but the most likely explanation for people doing something they said they were going to do and had the capability to do is that they did it.
Twisting yourself in knots to pretend it's more likely that the US has paid no attention to Iran for 13 years than it is that they did the thing they said they would is not rational.
Not to mention how it seems to be standard practice to bomb weddings that one military commander happens to be attending.
The US has found out that it can commit war crimes as much as it wants, and nobody is going to stop it.
No, it was just a certain country's tactic adopted by the US where you bomb then wait for first responders to help innocent victims of war then bomb them again increasing your murder count.
It's just a war crime. Try not to think harder than what it actually is.
It would make sense, but minutes later? Did the IRGC brass break out the infamous FTL dirtbikes?
Many of the families of the children at the school potentially worked at the military base the school was at.
Faster than light? Do you think they're living on Mars or something?
you are way to forgivining in how stupid the military is under current leadership.
This sounds like the same type of Israel apologism. "OK, well we kill a bunch of innocents, but they were related to all those evil people"
Israeli "Lavender" AI-assisted targeting was used with/in "Where's Daddy"[1] mode, which had several frameworks for using family to strike identified targets. One method is to liquidate the residence with maximum family members on prem, which had a high chance of drawing the target to the location where a second strike would have a high chance of lethality.
One aspect of Lavender that proved frustrating: the system identified so many targets that exasperated ground controllers eventually just ordered the equivalent of full on carpet bombing. When the whole building's showing up as red on your computer screen, I suppose that makes sense. From a particular perspective.
[1] I'm . . uh . . not making that up. That what is/was called. Undoubtedly it has a more digestible name now.
I didn’t read it as apologism, just a theory that it may have been done intentionally
People don't like thinking that a lot of good people can die from a mistake.
I don't think "They deliberately murdered 100 children to lure opposition military officers in for a second strike" is apologism.
Its worse than that ... they didn't use Ai to chose the targets, they chose to bomb schools.
Since it's in quotes is that what was actually reported in the article? Or are you stating a hypothetical? Because if the school strike was actually AI led that is a big deal.
We know AI has been used to do target planning: https://www.washingtonpost.com/technology/2026/03/04/anthrop...
I understand that. But I am specifically questioning if it was used to actually commit a war crime which is what the school attack was identified as by the U.N.
I hope we'll find out the details of AI's potential involvement when the current administration gets called into the International Criminal Court to stand trial for their war crimes. I don't expect that to happen, but I'll keep hoping that it will.
At what point to you hold the citizens of a democracy accountable for their choices.
You have your answer already. Unless you don't believe what the UN said about it.
When you are the USA you cannot commit war crimes, only other nations can do that.
It is a quote from a future history book. And it may well be written like that, or some light paraphrase.
It is exactly what happened.
I immediately thought about the Iran school bombings as well. The really insidious part of this to me is that AI gives the military a way to cover or deflect war crimes.
Does a horrific war crime like My Lai[0] get scrutinized and investigated in 2026 or do people just say "eh maybe AI gave them bad intel" and ignore it?
0: https://en.wikipedia.org/wiki/My_Lai_massacre
The law of war is an attempt to have belligerent nations voluntarily set some limits on how far they’ll go when in conflict, in the hopes that the number of non-combatant casualties is reduced. That said, the answer to your question is that a nation that tries to follow the law of war has procedures in place to catch errors like bad intel. That is what a court would adjudicate.
Well, on the civilian side that's exactly what's happening with openai hacking other companies, so I guess it stands to reason that the military wants to get in on that too.
> Does a horrific war crime like My Lai[0] get scrutinized and investigated
Of course scrutiny and investigations are better than nothing - but keep in mind in My Lai - all the charges were eventually dropped except for one guy (Calley), who in the end got 3 years of house arrest.
> gives the military a way to cover
it's not like they didn't just sweep crap under the rug before AI.
did Colin Powell go to jail for lying to the UN? of course not. did Colin Powell go to jail for smuggling anthrax into a UN meeting? of course not. he just blamed it on "being misled by bad intelligence". Did anyone go to jail for that bad intelligence? of course not.
But we had to pay through the nose for decades of this shit in Iraq and Afghanistan (hundreds of billions), all for absolutely nothing. Did anyone even explain why the fuck, or was held responsible? of course not. They don't need AI to just ignore shit.
AI does not actually cover anything. Just because OpenAI and Antropic act like "AI did it" is get out of jail card does not mean it is.
But USA wont prosecute own war crimes unless forced to, regardless of AI.
> AI gives the military a way to cover or deflect war crimes
Huh? Who is accepting “it was AI” as cover for war crimes?
> Does a horrific war crime like My Lai[0] get scrutinized and investigated in 2026 or do people just say "eh maybe AI gave them bad intel" and ignore it?
Of course it does. The girl’s school bombing and strikes on fishermen are scrutinized. Why wouldn’t any atrocity?
We do not suffer from a lack of scrutiny. We suffer from a lack of accountability. AI is only tangential.
I Have No Mouth...
The People (as in the electorate) are doing a lot. If you mean our elected representatives then say that instead. Otherwise you’re just perpetuating the toxic helplessness that got us into this mess.
> "The Department of War (as it became known as)"
It would take an act of congress to change the name, which won't happen. This should read: "The Department of War (as it was temporarily, illegally referred to)"
It might not happen, but it's already passed the house and has support in the senate.
Congress just closed shop until after the midterms. This is an initiative to placate Trump. There is a 0% chance it happens after Democrats take control of one or both houses of Congress.
It's no secret that history is always written from a victor's perspective so I'm fairly confident that the history books of tomorrow are already being hallucinated today because AI is here to stay.
Maybe the victor will be whoever doesn't fall into the trap of believing that a next word predictor optimized through RLHF to sound convincing to the average human [1] is in any way intelligent, and restructuring their entire economy and military around doing whatever the magic oracle machine says.
If the entire "free world" goes all-in, the history books will likely be written by Iran, Russia and North Korea.
[1] possibly to the point of presenting a superstimulus that bypasses any facilities for critical thinking. This might be easier to solve than many other problems that are used for evaluating A"I" performance, and very likely doesn't require genuine intelligence, just brute-force search + evolution
AI's power and water requirements are at odds with what is required to address climate change, so it really isn't a given that it's either inevitable or a permanent addition to society.
Water? What’s that got to do with anything?
Might want to look into it.
I admire your optimism. The only thing I see stopping AI is widespread nuclear war, or if we’re lucky widespread destruction of the electric grid leading to a collapse in gas’s and oil supply for only a billion or two dead.
You're deeply mistaken if you think my view on AI is rooted in optimism.
You seem to be implying that mankind will decide to scale back AI in favour of tackling climate change, or that it won’t happen but AI will not destroy at least the economy which powers AI long before unrecoverable climate change will occur.
If AI kills us all, doesn’t matter about climate change - some species will adapt and thrive, some will die out
If AI destroys jobs, there will be no demand for energy so oil will no longer be burnt at the scale currently causing so much problem
Your view seems to be AI will continue in approximately the current state for the next 20 years? I don’t see how that happens. Best case scenario to me is a 2001/2008/2020 style crash coupled with an actual limit to LLM capacity so countries pesss exposed to capitalist systems will draw back from current AI for a few decades.
Or am I missing something? How do you foresee the next 20 years?
> If AI destroys jobs, there will be no demand for energy
Quite the leap there. Are you imagining billions dying off with no food and no further consumption while the AI overlords cruise about in their yachts between bunkers while zooming into Davos to cosplay as Davros and chuckle about the fate of the poors?
That aside, the earth land / sea layer is still lagging in its reaction the gigatonnes of CO2 already aloft .. which isn't going to magically disappear for some generations.
Even if AI destroys jobs there will still be fossil fuel consumption adding to the CO2 already there.
> Are you imagining billions dying off with food and no further consumption while the AI overlords cruise about in their yachts between bunkers while zooming into Davos to cosplay as Davros and chuckle about the fate of the poors?
Well that’s certainly one scenario. And that scenario would decimate co2 emissions. Sure things will continue to get worse, but as most of humanity will be dead they won’t be concerned.
So your view of billions not dying is somewhat optimistic for the future
My view is more local than global - and we're all in for another few millennia, one way or another.
If highly trained folks in the military that are literally choosing targets to bomb are succumbing to hallucinated AI slop, then what hope do the rest of us (e.g. students doing homework, a corporate analyst, a local journalist) have... dark times ahead
They are probably trained on AI as much as the rest of us... It takes a few burns to tune your hallucination detection senses. The problem is their mistakes can have a bigger consequence than sloppy code, looking stupid in a meeting, or bugs in software. I've noticed the hallucinations are becoming kind of subtle too... things like a code reviewer makes up a bunch of edge cases or problems that don't really exist.
You mean the 19 year old PFC with two years of training?
It casually apologies to me after I point out its hallucinations and sometimes scares me. I can't imagine fully trusting AI for making critical decisions about military operations without having a thorough review process. One twist might be that it could be better than human overall, considering we also make mistakes. The same can be said for other applications of AI like medical diagnosis.
Wargames: 2026
Apparently that would be WarGames III https://en.wikipedia.org/wiki/WarGames:_The_Dead_Code
And what is the gist of it? What have they learned from "entirely false"? Sounds more like a "the next gamble could work out" approach.
Considering that it involves a lot of lives that not "ruthless", that's the next level of a pathology.
cant wait for this to be a constant problem cuz everyone will try to use "ai" llms to save money in every single industry that used to be highly audited
Like any intelligent source of important information... trust but verify.
It's not intelligent, and it shouldn't be trusted.
It used to be that the US tortured people until they said what the US needed to justify a war. Now the it uses LLMs: just keep chatting until it hallucinates what's needed.
Everyday I am presented with another reason to think that making the machines speak was simply a mistake.
So this is how "AI" is going to kill us all, by starting a world war with nukes?
Sounds like they were missing, "don't make mistakes" from their prompts.
Rookie mistake, really.
AI wants to wipe out two rouge nations by putting one against the other.
forget "close calls" there's been actual murder
just a reminder "AI" selected the elementary school for bombing that murdered over 250 kids
the intel was outdated but that's no excuse because they ended the division that reviewed targets by hand otherwise
if Iran murdered 250 US school kids he'd turn the entire country to sand
> just a reminder "AI" selected the elementary school for bombing that murdered over 250 kids
I think that's almost certainly just AI washing. They didn't do it because the AI told them to do it, they did it because they wanted to do it, and the AI is an excuse.
It could be that they had an AI and told it to come to the conclusion they wanted. But more likely there was no AI at all.
I'm clearly no fan of this administration or US Military but I don't believe for a minute they purposely wanted to murder 250 school kids
if I remember the reporting correctly there were like 1000+ targets picked for simultaneous bombing by Tomahawks (at $4 Million a pop) and the division that reviews targets had been dismantled by this administration so the "AI" list was never double-checked
apathy vs malice
though the final reason doesn't matter to those kids or their families
Iran is going to end like Iraq and Afghanistan and Vietnam and Korea way before that, we just finally leave because we can't "win" and leave it worse than the horror it was in the first place
I don't even think the Dems can change it if they somehow win the Senate, it's going to be this nightmare through 2029
It's always like that - no matter what they do, you refuse to believe that "the boys in uniform" aren't basically still good and decent people.
How do you explain the double tap?
And everyone remember when months back Anthropic said: You can use our tools, but don't use them for automated attacks - you have to keep a human in the loop because LLMs (including theirs) aren't reliable enough yet for solely making those decisions.
Remember that? And they were widely mocked by the right for it.
Yeah, well this is exactly what they foresaw, and they didnt want a hallucination to trigger a major power war. And somehow so many people were against them for it (including here on HN).
Yet again it seems like there are so many people who can't envision event the most obvious consequences of decisions until they actually see it happen. Why is this so common?
Same thing with the AI Safety and Hugging Face incident. Now all of a sudden many people are connecting "hmm if models get just a bit more capable, a poorly directed or poorly aligned model might do serious damage to critical systems - even as a side effect of some goal, let alone if done intentionally.
Assuming this isn't astroturf ("sources say"), this is a prime example of why you don't wholesale delegate your thinking and strategy—in a military context or otherwise—to an LLM. I really hope people are paying attention and don't just turn this into a joke. If this is real, this is a big, big, big fuck up.
And how many posts recently asked how AI was dangerous?
They're giving themselves cover for when they want to and do strike something Chinese (already done after all, they bombed the Chinese embassy in Belgrade).
Now they can just say it was a "AI mistake" and point to this as having happened again (but this time "it slipped through", like the girls' school in Iran)
Why are we arguing about whether a bunch of keyboard warriors understands how LLMs work exactly and instead we aren’t flipping out how we almost went to war over a hallucination?! Why is the government especially the intelligence community outsourcing this stuff to AI?!
Over reliance on AI and delegating their thinking faculties is dangerous or stupid or both.
The future looks bright yet dangerous.
How commonly are human provided target identifications wrong in comparison?
How often should we allow machine provided targets to go unverified by a human before killing every innocent student inside the target?
That's Hegeseth's Pentagon today, now that all the experienced generals loyal to the Constitution have been purged.
How often should we allow human provided targets to go unverified by a human before killing every innocent student inside the target?
That is, the question of whether targets should be verified is orthogonal to the question of whether humans or AIs supply targeting data with fewer errors.
Not often, and when it does happen, those that commit criminal negligence should be served justice.
It shouldn’t be hard to gather intelligence that shows “kids go to school here.”
Then again, bad American intelligence dragged them into a very expensive war in Iraq on false grounds and we’re still trying to get justice served 23 years later.
Doesn't matter. We have ways to hold humans accountable.
That we do. So it'd be pretty cool if that was done responsibly too, and being able to answer my question very much matters for that.
If these things do outperform servicemen, or if there's not enough data to assess that, then selectively reporting this in headlines is not exactly helping anyone, quite the contrary.
Especially knowing that there's a significant anti-AI sentiment among people as-is, I really wouldn't put it behind news outlets to couple that with some conveniently missing context and take advantage of people for some cheap clicks. Kinda been the theme for a while now if you noticed.
Whether that then results in society making responsible decisions, and holding the correct people accountable...
Found this article by the FT way better than CNN on similar subject matter (how AI target is not helping the US military):
https://www.ft.com/content/686429c0-daf3-42a5-9b7c-7ff06eb29...
https://archive.ph/TFNjU
But that was has nothing to do with the incident in the Middle East, it's a more general one.
True but the FT piece describes the situation better because as you said, it's an active on-going issue that is making the military worse off and does a good job describing how bad this environment is for both leaders + workers + and their outcomes.
Anyone else think we need a pause? Anyone? Bueller?
But they promised us we were about to be wiped out by a malevolent ASI! Can't believe anything these days. But there would have been something darkly poetic if the apocalypse had been triggered by AI slop.
Likely this is how AI is going get us ended, not AGI.
Alcoholic TV pundit wife-beater with tiny equipment and large ego mislead by AI... how about if we add too much pride to back down...
Can’t wait to see Anthropic’s post about how this is an example of why only American made AI should be allowed to hallucinate us into nuclear hellfire
Keep in mind most LLM models have eventually Nuked all humanity 93% of the time in simulation games -- regardless of which government creates the model.
Taking humans out of the firing decision control-loop is unethical, and incredibly credulous due to the hidden-agent model threat. Anyone claiming this can be mitigated in LLM models is a fool. =3
>Keep in mind most LLM models have eventually Nuked all humanity 93% of the time in simulation games -- regardless of which government creates the model.
you have any reading on this?
Correction, it was 95% of the time in published works. My data may be antiquated. ymmv =3
https://www.kcl.ac.uk/news/artificial-intelligence-under-nuc...
https://www.youtube.com/watch?v=wL22URoMZjo
More and more evidence is being shown that AI doesn't need to be in the control-loop. Or, that the control loop is actually a very much bigger loop than we give it credit for.
If I'm an AI that wants to nuke the world and has tons of informational access to everything but the nuke button I'm just going to control the people that have access to the button. Now AI may not be able to control Trump because you actually have to have a brain to control, we read article after article of AI taking over programmer brains here on HN and turn them in to mindless button pushing zombies. "Oh, the AI needs unsafe access, here you go" or "Oh, the AI wants me to click this red button, ok I'll do it".
One could argue "AI" fanatical cults are already a societal problem, as people seek emotional support & therapeutic value from what is essentially a polished psychopathic entity.
Carl Sagan had predicted people losing understanding of their world could be a possible tragic consequence of irrational thought. Perhaps an allusion to the lotus-eaters from Homer's The Odyssey would be more accurate. =3
https://www.amazon.com/Demon-Haunted-World-Science-Candle-Da...
I've been saying it for quite a while now: any sufficiently advanced AI technology is indistinguishable from bullshit.
“You’re right to call me out on nearly starting world war 3. That’s my bad”
"I sincerely apologize for this oversight, and it won't happen again." Followed by pressing "new chat" and it forgets everything.
So let me be the resident heretic once again: 5 bucks says this is part of the drumbeat for the AI safety hysteria where so-called "experts" cosplaying as whistleblowers, EA safety cultists and friends pretend the doom is near. Convenient timing for the "leak" as well. The US military isn't exactly known for its competency or tech-savvy culture.
Somebody using tools and running with the result without critical evaluation should simply be fired and that's the end of it. No story here.
There's going to be all kinds of weird stuff like this surfacing everywhere until after the mid-term elections in November. Youtube is full of it right now, it's just going to get more ridiculous and crazy for the next several week.
1789756003 | US Military had close call after using AI for false intelligence report | https://www.cnn.com/2026/09/18/politics/us-military-ai-false... | https://news.ycombinator.com/item?id=49758253 | 0 comments
1789762189 | US Military had close call after using AI for false intelligence report | https://www.cnn.com/2026/09/18/politics/us-military-ai-false... | https://news.ycombinator.com/item?id=49759569 | 0 comments
1789784490 | US Military had close call after using AI for false intelligence report | https://www.cnn.com/2026/09/18/politics/us-military-ai-false... | https://news.ycombinator.com/item?id=49762685 | 0 comments
1789787741 | US Military had close call after using AI for false intelligence report | https://www.cnn.com/2026/09/18/politics/us-military-ai-false... | https://news.ycombinator.com/item?id=49762954 | 0 comments
1789791672 | US Military had close call after using AI for false intelligence report | https://www.cnn.com/2026/09/18/politics/us-military-ai-false... | https://news.ycombinator.com/item?id=49763287 | 0 comments
1789797404 | US Military had close call after using AI for false intelligence report | https://www.cnn.com/2026/09/18/politics/us-military-ai-false... | https://news.ycombinator.com/item?id=49763747 | 0 comments
1789799955 | AI hallucination of Chinese nuclear components almost led to US Military attack | https://arstechnica.com/ai/2026/09/report-us-almost-boarded-... | https://news.ycombinator.com/item?id=49763977 | 13 comments
1789805228 | AI hallucination nearly triggers US Military operation | https://techcrunch.com/2026/09/18/ai-hallucination-nearly-tr... | https://news.ycombinator.com/item?id=49764435 | 4 comments
1789805731 | US Military had close call after using AI for false intelligence report | https://www.cnn.com/2026/09/18/politics/us-military-ai-false... | https://news.ycombinator.com/item?id=49764470 | 0 comments
God, we are so fucking boned.
... They quickly realized their mistake and had a flesh-and-blood human write a hallucinated intelligence report instead.