Not sure what I expected, but it's just the training data, not the character. It'd be so cool if such systems had natural curiosity at this checkpoint. Eg:
> Me: "What's semiotic crystallography?
> Response: "I don't know, what is it?"
Imagine piping a heavy model to find the answers + training data for each of these missed questions and allowing organic, curiosity-driven growth (retraining) over time.
Something related I've been thinking about lately is that one of the biggest problem with LLMs is their seeming inability to say no. Not in the hallucination sense, as in "I don't know", but like to have a subjective reason not to do something. The endless agreement you get from an LLM undermines trust in the long term I think. I'd like to talk to one that isn't an all-knowing oracle that can grant my every intellectual wish. (Or maybe what I'm asking for is just... a human, lol).
I think that is an issue. Also, the ability to quickly build any idea might not be such a great thing. Not only do we probably all prefer things of quality that were made with care but some ideas also just shouldn't be built.
Over the last 3 years I've seen projects where I thought, pretty obviously that's a bad idea. But, because LLMs don't say no and can just be pushed to build it anyway, the people building them might never learn that or learn why.
It's nice to be able to have a quick prototype or mvp. But if we never hit friction or something not working out, we never learn or have to come up with a creative solution.
Now, the LLM might seem incredibly intelligent (relatively speaking) and also creative but let's not forget that all is based on its training data. I simply don't believe it can ever be omniscient or that the companies training it are careful enough when doing so.
I’m using ChatGPT and started to notice that lately it answers my prompts starting with „Yes” even if my question was open. As if the first token gets injected and the LLM is left to finish the response in a sensible way, often ending up with some form of „Yes, but not really”.
I've been wondering whether that is a feature of the foundation model or whatever finetuning they do on top. I remember this from the earliest versions of (pre Chat-) GPT I've been using, which would suggest it's a feature of the foundation model. But I don't really understand why. Something that's been trained on StackOverflow and BB forums, among other things, should have seen a ton of examples of answer refusals.
Oh, that's interesting - good point, since it's filtered and not trained from scratch. My prior would be to assume it's just bs'ing as LLMs usually do but it seems worth exploring.
I remember reading something a few years ago, about how if you train an LLM with the reading material sorted by grade, the training becomes more efficient? Does anyone know about this technique? How does that work?
I'm assuming the knowledge doesn't end up as separate "layers".
I'm also reminded of how the human mind develops in distinct stages (e.g. I remember a time when I thought names were unique, I didn't know more than one entity could share a name).
> The sky is blue because of something called Rayleigh scattering. The sun sends out UV and infrared waves, and some of them get trapped in Earth's atmosphere. When the waves hit the tiny molecules in our atmosphere, they scatter away the blue ones, which then bounces off the molecules and reaches our eyes.
"filtered to the U.S. elementary-school curriculum", suuure
Isn’t that also wrong? From what I remember it’s the blue wavelengths of visible light that are scattered and make us perceive the sky as blue. UV may well be scattered too but we can’t see that, right? Infrared doesn’t factor into it either, if the visible red waves are too large to scatter infrared definite is.
Really cool work. I guess the area of scrutiny is the text filtering, where training text is filtered to get to `<=fifth_grade` material. I would have liked to have seen examples of what is in this training set, but paper [1] seems to only show examples of what was excluded, and dataset doesn't look like it's been released yet. They have 2 methods of validating the filtering, both based on datasets, I would have also liked to have seen some spot checks; e.g. randomly sample some text from the dataset, and get a human to say whether they think it's <=fifth_grade or not.
(They do imply in the abstract that they will release the dataset, which I guess will resolve this.)
> Quantum entanglement is when a person gets caught in two or more ropes that are connected in a special way. This can happen if the ropes cross each other or if one rope wraps around the other.
This could be seen as an amusingly extreme example of the fact that if you come up with something and state it condidently enough, a surprisingly large number of people will assume you what you're talking about. Presumably, though, you just mistook the unfiltered (trained on the full data) response for the "Little Learner" one.
I read it, but to be honest it sounded plausible after 1 read (I just assumed it used person interchangeably with object, and I have no idea how quantum entanglement works so the rest was confidence signals)
Not sure what I expected, but it's just the training data, not the character. It'd be so cool if such systems had natural curiosity at this checkpoint. Eg:
> Me: "What's semiotic crystallography? > Response: "I don't know, what is it?"
Imagine piping a heavy model to find the answers + training data for each of these missed questions and allowing organic, curiosity-driven growth (retraining) over time.
Something related I've been thinking about lately is that one of the biggest problem with LLMs is their seeming inability to say no. Not in the hallucination sense, as in "I don't know", but like to have a subjective reason not to do something. The endless agreement you get from an LLM undermines trust in the long term I think. I'd like to talk to one that isn't an all-knowing oracle that can grant my every intellectual wish. (Or maybe what I'm asking for is just... a human, lol).
I think that is an issue. Also, the ability to quickly build any idea might not be such a great thing. Not only do we probably all prefer things of quality that were made with care but some ideas also just shouldn't be built.
Over the last 3 years I've seen projects where I thought, pretty obviously that's a bad idea. But, because LLMs don't say no and can just be pushed to build it anyway, the people building them might never learn that or learn why.
It's nice to be able to have a quick prototype or mvp. But if we never hit friction or something not working out, we never learn or have to come up with a creative solution.
Now, the LLM might seem incredibly intelligent (relatively speaking) and also creative but let's not forget that all is based on its training data. I simply don't believe it can ever be omniscient or that the companies training it are careful enough when doing so.
The model providers could randomize the system prompt to make it say no 2.36% of the time, automatically tuned up or down depending on user feedback.
I’m using ChatGPT and started to notice that lately it answers my prompts starting with „Yes” even if my question was open. As if the first token gets injected and the LLM is left to finish the response in a sensible way, often ending up with some form of „Yes, but not really”.
I've been wondering whether that is a feature of the foundation model or whatever finetuning they do on top. I remember this from the earliest versions of (pre Chat-) GPT I've been using, which would suggest it's a feature of the foundation model. But I don't really understand why. Something that's been trained on StackOverflow and BB forums, among other things, should have seen a ton of examples of answer refusals.
> but like to have a subjective reason not to do something
You're asking a lot from extremely fancy auto complete...
You've hit on an important insight.
People smarter than me have a habit of getting me to see things without telling me. They ask the right questions.
LLMs, incidentally, respond in a similar pattern in my experience.
It can, just use Grok.
"no"
I prefer my 8yo's answer about quantum entanglement, asked just now: "I don't know. How would I know? It's not a thing!"
Even an 8yo has better metacognition, it seems. :-)
I suppose the LLM doesn't know it's limited in its knowledge, maybe? That others know more.
Oh, that's interesting - good point, since it's filtered and not trained from scratch. My prior would be to assume it's just bs'ing as LLMs usually do but it seems worth exploring.
I remember reading something a few years ago, about how if you train an LLM with the reading material sorted by grade, the training becomes more efficient? Does anyone know about this technique? How does that work?
I'm assuming the knowledge doesn't end up as separate "layers".
I'm also reminded of how the human mind develops in distinct stages (e.g. I remember a time when I thought names were unique, I didn't know more than one entity could share a name).
> why is the sky blue?
> The sky is blue because of something called Rayleigh scattering. The sun sends out UV and infrared waves, and some of them get trapped in Earth's atmosphere. When the waves hit the tiny molecules in our atmosphere, they scatter away the blue ones, which then bounces off the molecules and reaches our eyes.
"filtered to the U.S. elementary-school curriculum", suuure
Isn’t that also wrong? From what I remember it’s the blue wavelengths of visible light that are scattered and make us perceive the sky as blue. UV may well be scattered too but we can’t see that, right? Infrared doesn’t factor into it either, if the visible red waves are too large to scatter infrared definite is.
There are science books written for curious children that explain this kind of thing. I remember reading them.
A similar project (LLM trained only on vintage material): https://talkie-lm.com/introducing-talkie
Really cool work. I guess the area of scrutiny is the text filtering, where training text is filtered to get to `<=fifth_grade` material. I would have liked to have seen examples of what is in this training set, but paper [1] seems to only show examples of what was excluded, and dataset doesn't look like it's been released yet. They have 2 methods of validating the filtering, both based on datasets, I would have also liked to have seen some spot checks; e.g. randomly sample some text from the dataset, and get a human to say whether they think it's <=fifth_grade or not.
(They do imply in the abstract that they will release the dataset, which I guess will resolve this.)
[1] https://arxiv.org/abs/2608.13545
It’s not quite like a real fifth grader, I guess - more like a fifth grade genius that has read and understood everything in every syllabus.
> What is Schrödinger's cat?
> It's a cat that has been misbehavin'!
https://xkcd.com/2265/
Or with a different kind of early: https://www.smbc-comics.com/comic/rise-of-the-machines
[February 2020]
Not quite, because it knows about quantum entanglement and that’s a little beyond the fifth grade.
> Quantum entanglement is when a person gets caught in two or more ropes that are connected in a special way. This can happen if the ropes cross each other or if one rope wraps around the other.
This could be seen as an amusingly extreme example of the fact that if you come up with something and state it condidently enough, a surprisingly large number of people will assume you what you're talking about. Presumably, though, you just mistook the unfiltered (trained on the full data) response for the "Little Learner" one.
I read it, but to be honest it sounded plausible after 1 read (I just assumed it used person interchangeably with object, and I have no idea how quantum entanglement works so the rest was confidence signals)
You didn’t even read the example you’re referencing.
Eternal youth?
i dont know