I'm growing weary of all the AI hype being shoved down my throat. Every time I dig into examples of what it can do, the result seems shittier than what a talented human would achieve instead. I'm forming an impression it's a kludge for the mediocre.
I don't want to be a crotchety old grump about this - I'd love to hear thoughts on truly notable examples where quality far exceeds our existing best in class work.
I think your only problem is too-high expectations.
>examples where quality far exceeds our existing best in class work
I'm no expert, but what percentage of the economy needs/is best in class work?
I agree running up against the limits of these models after reading the marketing material can give the impression they're "a kludge for the mediocre" and they certainly are in part that, but they're also so much more.[0]
After the recent spate of announcements for coding LLMs, I dug up a problem from my early days in programming contests.
A simple, naive solution from these LLMs runs 1000-2000x slower than the winning solution depending on the algorithm the LLM uses.
Asking the LLM to "make it faster" can knock the runtime by 10x. Close but no cigar.
Further prodding for even faster solutions sometimes shows glimmers of bridging the mental gap - but inevitably, the LLM doesn't know what to do with the newfound knowledge and wallows in its local minimum. Typically ending with "this is the fastest code to solve the problem".
Oh, grasshopper. You fail.
It wouldn't be so bad if this problem solving session was consistently reproducible. But every time I create a new session and give the problem, often times the naive solution appears. Sometimes the LLM can't even produce a working naive solution - even though the comments in the proposed source code shows the LLM knows what the correct answer is - the actual source code can't produce that answer.
Once, I decided to give some hints to the LLM to see if I could get it to jump the chasm. The LLM decided to limit its search space to a certain range - which I knew was wrong, and resulted in incorrect results.
I prompted the LLM to try and expand the search range, which it cheerfully tried and deduced an expanded search range - which was still too limited. Further prodding resulted in the LLM reverting back to the original search range. No amount of prodding would get the LLM to expand its search range. In a fit of rage, I told the LLM to use an arbitrary large search range which would cover the necessary search space. The LLM refused!
Dang - I was chatting with HAL from 2001 - A Space Odyssey.
I have been unable to repro that particular chat situation.
I guess we should be happy that the LLM can even produce a working solution - probably 5-10 years ago, we'd be gobsmacked. Even call it magic... But it's not remotely consistent.
but i agree with the others - AI currently shines not at being better than best-in-class humans, it's - sometimes - useful at doing intellectual chores.
> examples where quality far exceeds our existing best in class work.
You would be seeing far more than hype if we were at this point. This level of AI would be world changing.
I feel your expectations are too high. Realise what LLM’s are, what they can do and to what standard, and leverage them accordingly. It takes time to determine the scopes in which they increase your productivity. Trying it a few times and then disregarding it seems extremely foolish when so many developers are telling you it has enhanced their productivity to some degree.
An AI doesn't need to think in the same way humans think. It just needs to achieve results (that are better, or at least equal to humans).
The same question has been asked of chess "ai" in the past - that chess ai isn't thinking, it's "just" searching through all possibilities etc. And yet, the result is that no humans can beat chess ais now-a-days.
That an LLM does not need to think to produce the output we want seems fairly uncontroversial. However, a statement like “LLMs may think, just not in the same way humans think, to produce the output we want” is problematic.
“The same way humans think” is the only kind of “think” that matters, for all intents and purposes. If we cannot define what it specifically is—because it loops us immediately back to the definition of consciousness et al.—the most precise definition of it will have to be along the lines of “the sort of thing that goes on in human minds”.
There's a third way; coming for some peoples' jobs due to false advertising and almost unthinkable levels of hype.
Like, there are actual companies who have stopped hiring junior developers or even laid off junior developers in favour of our robot overlords. This is, obviously, a terrible idea, and will hurt those companies, because the output of these things is pretty much crap, but meanwhile some people _have_ lost their jobs.
I'm growing weary of all the AI hype being shoved down my throat. Every time I dig into examples of what it can do, the result seems shittier than what a talented human would achieve instead. I'm forming an impression it's a kludge for the mediocre.
I don't want to be a crotchety old grump about this - I'd love to hear thoughts on truly notable examples where quality far exceeds our existing best in class work.
I think your only problem is too-high expectations.
>examples where quality far exceeds our existing best in class work
I'm no expert, but what percentage of the economy needs/is best in class work?
I agree running up against the limits of these models after reading the marketing material can give the impression they're "a kludge for the mediocre" and they certainly are in part that, but they're also so much more.[0]
[0] https://crawshaw.io/blog/programming-with-llms
> I think your only problem is too-high expectations.
Well, I mean, if it can only do crap work, then what use is it, really?
After the recent spate of announcements for coding LLMs, I dug up a problem from my early days in programming contests.
A simple, naive solution from these LLMs runs 1000-2000x slower than the winning solution depending on the algorithm the LLM uses.
Asking the LLM to "make it faster" can knock the runtime by 10x. Close but no cigar.
Further prodding for even faster solutions sometimes shows glimmers of bridging the mental gap - but inevitably, the LLM doesn't know what to do with the newfound knowledge and wallows in its local minimum. Typically ending with "this is the fastest code to solve the problem".
Oh, grasshopper. You fail.
It wouldn't be so bad if this problem solving session was consistently reproducible. But every time I create a new session and give the problem, often times the naive solution appears. Sometimes the LLM can't even produce a working naive solution - even though the comments in the proposed source code shows the LLM knows what the correct answer is - the actual source code can't produce that answer.
Once, I decided to give some hints to the LLM to see if I could get it to jump the chasm. The LLM decided to limit its search space to a certain range - which I knew was wrong, and resulted in incorrect results.
I prompted the LLM to try and expand the search range, which it cheerfully tried and deduced an expanded search range - which was still too limited. Further prodding resulted in the LLM reverting back to the original search range. No amount of prodding would get the LLM to expand its search range. In a fit of rage, I told the LLM to use an arbitrary large search range which would cover the necessary search space. The LLM refused!
Dang - I was chatting with HAL from 2001 - A Space Odyssey.
I have been unable to repro that particular chat situation.
I guess we should be happy that the LLM can even produce a working solution - probably 5-10 years ago, we'd be gobsmacked. Even call it magic... But it's not remotely consistent.
> the result seems shittier than what a talented human would achieve instead.
but what about compared to an average, or below average human would achieve?
chess or go.
but i agree with the others - AI currently shines not at being better than best-in-class humans, it's - sometimes - useful at doing intellectual chores.
> examples where quality far exceeds our existing best in class work.
You would be seeing far more than hype if we were at this point. This level of AI would be world changing.
I feel your expectations are too high. Realise what LLM’s are, what they can do and to what standard, and leverage them accordingly. It takes time to determine the scopes in which they increase your productivity. Trying it a few times and then disregarding it seems extremely foolish when so many developers are telling you it has enhanced their productivity to some degree.
It's easier to just tell yourself that everyone else must be mediocre than it is to learn the strengths and weaknesses of a new class of tools.
An AI doesn't need to think in the same way humans think. It just needs to achieve results (that are better, or at least equal to humans).
The same question has been asked of chess "ai" in the past - that chess ai isn't thinking, it's "just" searching through all possibilities etc. And yet, the result is that no humans can beat chess ais now-a-days.
"The question of whether computers can think is about as interesting as the question whether submarines can swim" - Dijkstra.
That an LLM does not need to think to produce the output we want seems fairly uncontroversial. However, a statement like “LLMs may think, just not in the same way humans think, to produce the output we want” is problematic.
“The same way humans think” is the only kind of “think” that matters, for all intents and purposes. If we cannot define what it specifically is—because it loops us immediately back to the definition of consciousness et al.—the most precise definition of it will have to be along the lines of “the sort of thing that goes on in human minds”.
The question is if it’s worth all the money and resources if it’s not thinking at all so it’s not a way to AGI.
Would you invest so much in a bunch of if-then-else lists?
In the news we seem to have reached Schrödinger's AI: Too dumb to do anything properly, but coming for everyone's jobs due to being too powerful.
There's a third way; coming for some peoples' jobs due to false advertising and almost unthinkable levels of hype.
Like, there are actual companies who have stopped hiring junior developers or even laid off junior developers in favour of our robot overlords. This is, obviously, a terrible idea, and will hurt those companies, because the output of these things is pretty much crap, but meanwhile some people _have_ lost their jobs.
You just need to convince the management
https://archive.is/whcKL