> Scientists who adopt AI gain productivity and visibility: On average, they publish three times as many papers, receive nearly five times as many citations, and become team leaders a year or two earlier than those who do not.
To me this effect doesn’t seem to reflect on AI very much, it seems to reflect on humans. Like maybe this is more evidence of the Babble Hypothesis and the incentives in research than AI, no?
This is superseded/proven by basic psychometrics it seems? Big Five Extraversion is roughly equivalent to "social dominance", how well an individual implements themselves in a social setting. "Extroverts" or people high in the trait are of course more likely to see progression on the basis they are superior at presenting value in a social setting in terms of social ability, which is often (falsely) accepted as a proxy for overall competence. This is why they end up running orgs as well
"Perhaps, says Evans. But he doesn’t think that the problem is baked into the algorithmic design of AI. More than technical integration, he argues, what may matter most is overhauling the reward structures that shape what scientists choose to work on in the first place.
'It’s not about the architecture per se,' Evans says. 'It’s about the incentives.'"
> “It’s not about the architecture per se,” Evans says. “It’s about the incentives.”
It would have been useful to check whether less original work was already getting more citations before AI adoption. That could reflect broader trends and network effects: heavily cited research areas attract more authors optimizing for citations, so high-productivity researchers end up clustering on the same topics.
The actual paper is linked there if you are curious.
But also just thinking about your point for one second: in your mind, how else would they argue for the conclusion if not by checking the trend over time? Like what is the precise implication here?
They did. The article explains tbat this is a trend which has been getting worse for years, specifically pointing to search engines as a major turning point. Your comment is completely off the mark.
As with other fields touched, AI is merely amplifying what was already there. The aim of many scientists isn't discovery in and of itself. Discovery is a side effect of their primary drive to publish and - hopefully - become well known. And establishments only make things worse, because it's the things that are most likely to produce tangible results (the papers, or economically valuable products) that get the most funding.
It's a little more complicated than that. If all you're interested in disovery, you can wander off into rabbit holes or disappear into areas that are only interesting to you. Publishability is useful because it gives you useful external feedback into whether a community thinks what you're doing is "worth discovering". Like any metric, you can find yourself gaming it (intentionally or unintentionally) and it has a slew of other failure modes. Still, you can prize discovery and also care about what other people think.
I wouldn't even be certain about being well known. I would guess there is lot of pressure to stay employed or get the next funding. So optimising for this is the new goal and lot of publications and citations are metrics that help with that.
The aim of many scientists is discovery, publishing is a side chore to survive and to get funding. Automate paperwork and you get more time for discovering.
Seems to me that both perspectives are true, and the relative importance of the metric incentive vs the discovery incentive varies. But the metrics and rewards are critical to the perpetuation of the scientific discovery system; its really hard to disentangle.
Well the paperwork is automatable, and things are being automated. But still there're the findings that the article points to: it's leading to far more publishing (and ladder-climbing) than novel discoveries.
OK cool so you have a far larger sample size than I to generalize from. Can you explain then what're the root cause(s) of the findings in that article?
I do, and through conversations have learned that they enjoy what they do and publish patents (they're PhD in industry), but ultimately what they seek is "fame and glory" (literal quote).
I was also in academics myself up to the Master's level (research track), and personally had to deal with the politics of getting support for what I wanted to work on; that experience helped to discourage me from going on to a PhD, as I'd rather have proper leeway to work on what I really prefer and take avenues I find interesting.
Oddly enough my experience is the opposite. I live in an academic town, and many of my neighbors are scientists. They view the "fame and glory" as something that maybe someone else has a chance of achieving, but not a realistic pursuit for themselves. Pursuit of funding (which now includes suing the Federal Government) is at best stressful drudgework for them.
I work in industry. In that case, nobody who meets me would ever know that I have patents. I would consider them to be a useful add-on for my resume should I ever need one, but it doesn't define me.
100% agree. You could make the same argument for Hollywood : funding & revenue was always the goal, and we've been producing slop before AI was even a thing
"boost research careers".. seems like a pretty drastic conclusion to draw based on a technology that has existed for like 3 years and only lately is any good..
Yeah that was my instinct too. What sort of career defining trends are visible with this much historical data? Feels like someone wrote clickbait research to get published.
A new breed of academics has appeared whose jobs is to put their names in every paper possible. Literally, their job is to work on frameworks to buy co-authorship.
They do this in various ways, like establishing paper pipelines, collecting rents on labs and committees, focusing on money layer, using their profiles and citation count to help with acceptance of papers of other people , etc. You talk to them and they can’t explain their papers beyond a superficial introduction.
They collect huge citations, travel and give talk on the winner horses, collect credit, which feeds back into this fraudulent scheme. A scientist used to be a scientist not long ago, not a credit collector.
I wonder if Google could invent a new metric to expose them (weak ratio of first authorship, etc).
A lot of this is downstream of compensation schemes that explicitly reward dumb metrics, like raw paper-count without subjective evaluation of contribution or quality. I don't want to generalize, but this seems to be more common in countries that are not the US or Europe.
It's not new, and entire disciplines exist because of these dependent structures. It doesn't do to unseat and discredit the connected and well regarded, but you might enjoy some mild comfort in security in numbers through a little citational flattery.
It's a game of cultural tribalism. The only thing worse for one than not engaging is to upset the status quo unblessed.
As a bioinformatics person that's spent time in and out of industry/academia, I agree with some of the article's thesis. While I don't think LLMs or AI are going away, I do think it will allow people in academia to pump out a bunch of inane papers and continue to prop up predatory scientific journal publishing via tenure and promotion. In fact outside of how utterly useless Fable 5 is via their aggressive guard rails for my work, I quite like using statically typed and/or functional languages with other LLMs since there are some baked in guardrails via compiler + type system.
I think the flattening of progress is the most interesting dimension to the article. For an example a useful biological product discovery with a nonlinear path to get to there, look at the Taq polymerase (https://en.wikipedia.org/wiki/Taq_polymerase). Without some NSF funded exploratory ecological research by Tom Brock in Yellowstone Hot Springs to test the theoretical limit of life at high temperatures (https://en.wikipedia.org/wiki/Thermus_aquaticus) we never get to the Taq polymerase, we never get reliable/robust PCR (https://en.wikipedia.org/wiki/Polymerase_chain_reaction), which is now a gold standard method in both clinical and environmental testing! It is rather improbable to think that large language models would associate those domain connections across the topic (molecular biotechnology + ecology + microbial physiology). I also did some exploratory work with text embedding models people might use for RAG and challenged them with an open source scientific MCA question dataset, generalist embedders performed worse vs. domain specific embedders trained on scientific corpora (doesn't surprise me at all). However, if everything regresses to the median of the universe of possible knowledge, it seems like scientific leaning frontier models would get locked into this asymptotic flattening before turning cashflow positive for model vendors OR they become so locked down that only big pharma, state actors, or big ag can afford the API rates and vetting process.
I enjoy using AI loads. Yet I would be keen to see numbers on actual productivity increases. This reads as yet another datapoint similar to what I’ve experienced: maybe code was the bottleneck at some point, maybe now it isn’t but in my lived experience the bottleneck has simply shifted. Its easy to create “more” but to actually hit the business goals… I don’t see a 2x TRUE productivity boost in anyone in my company.
Please feel free to disagree with me! I am keen to hear more anecdotes to get more datapoints.
Some always refuse to acknowledge that. Like if every hype cycle was a roadrunner bit: some people see the cliff and stop running, others take a few steps off the cliff, look down and pull out a sign that says “uh oh” and plummet, and some people haughtily call the people who pulled out their “uh oh” signs needlessly pessimistic as they careen towards the ground.
Any flattening of discovery due to AI, but will be temporary.
We tend to think that obvious potential is the same as realized potential, for new technology.
For any specific context, there are generally innumerable smaller adaptations and capability thresholds that have to be crossed. And the price for that journey is often temporary loss off overt productivity.
No, this is significantly more permanent. LLMs are autocomplete generators based off current context, and training generations of people to always ask the planet burners instead of learning to think for themselves - and never having the experience of having to slowly think over the same thing for an extended period - may well mean a permanent cap to human knowledge and a dramatic slowdown or end to new knowledge.
You act like humanity doesn't exist in a competitive environment. If you think AI codegen is a mistake? Just relax, keep writing code by hand and wait for the pendulum to prove you right while showering you in wealth. There are plenty of people making this bet, and I wish the best of luck to you because I'm 99% certain you're on the losing end of it.
The very point of the article is that you can win individually and lose as a colective, and that the competitive nature of the field goes against the greater good. And the people betting against AI will be ripped off.
The market can remain irrational longer than any of us can remain solvent. The market is not any good at strategic or long-term thinking, particularly if it takes a generation to realize the scope of the damage, as seen by America abandoning its ability to manufacture things in chase of short term profits.
This is exactly the right answer. The supposed "rationality" of capitalism can ruin us before we get a chance to dazzle the world with our contrarian insights.
I hate how that very argument has been used by people riding the tide to rationalize the irrationality. Money talks or something. If you don't like it here why don't you leave etc. It is a grifters goto statement.
There is two different types of learning people are talking about.
The first type happens when you are enthusiastically engaged in a topic, which LLMs will likely enhance.
The second type happens as a by-product of solving a, perhaps deeply uncomfortably, difficult problem. This is what people are talking about when they say LLMs will hamper human cognition. Instead of sitting there for an hour and struggling, people will instead reflexively give in and ask an LLM to solve it for them.
it's an interesting point. is it worthwhile to struggle through an incidental task that has been solved before? we all stand on the shoulders of giants
I think in most cases, understanding is the point. we don't expect students to derive general relativity before doing astrophysics. re-invention is only a tool for understanding
“Understanding” without being able to use that knowledge for anything isn’t useful for getting stuff done.
The flip side is even more interesting. There’s a great number of electrical engineers with significant physics backgrounds who don’t really understand how electricity actually works, but they can still solve useful problems. By understanding I mean they can describe what underlying physical phenomena reactance represents etc.
Small counterpoint to your analogy, as someone who studied astrophysics: I actually did have a requirement to understand general relativity! Deriving all of it independently from scratch wasn't something we did, but there _were_ derivations involved. And it was definitely worth working through -- it _is_ a good tool for understanding. (I've long since left the field, but I don't regret the work I did.)
> when a parent answers their child's question, does it decrease the curiosity of the child?
When the child is able to go to YouTube and find a tutorial rather than having to puzzle it out, yes, it absolute does. We've seen this for decades now.
> its output can be novel or good, but rarely both at the same time.
> rarely
That is not a viewpoint they can't do something useful and new.
With that criteria, he could be talking about anyone.
I find it rare that people critiquing AI today, actually hold people to the same standards. Or are as enthusiastic about referencing ways machines keep surpassing us, as for ways they have not yet, when speaking about limits for progress.
LLMs are the tech in question, not ML (AI?) in general.
LLMs are fundamentally limited by their architecture to only return a token predicted by a statistical inference, essentially lossy decompression.
It's like arguing that taking an image, compressing it with JPEG and low quality, then decompressing it into something blurry with some random color values thrown in is creating new art.
No one is arguing that everything a human has created is good. No one is arguing that LLMs can't be useful.
Sutton is arguing that it can't be novel. Cherry picking a couple of words doesn't change his argument, which is very clear.
> AI is largely automating the most tractable parts of science rather than expanding its frontiers
By definition, creativity cannot be automated, and AI is a fantastic automation machine. It can explore thinking paths at a rate humans cannot match. But creativity is bringing the unthinkable into the thinkable, and that requires sensory experience [1]. Specifically, new definitions and symbols which never existed before. Imagine the concept vector space, and expanding that with new independent dimensions. Is that even possible ? When you look at history the answer is yes !. And each time there was an independent dimension added, it was an act of genius. It is an instructive exercise to name these moments in history where an independent dimension was added to human thought. Some examples in math would be the invention of a number, and in politics could be the idea of democracy. By contrast, LLMs are trapped in the vector space they are trained on, and they lack the feedback loop with sensory experience to be able to create and validate theories.
I don’t think we have spent enough time on the creativity axis.
When we solve problems we usually follow a heuristically guided energy efficient path. We just prune a lot of possibilities based on our existing knowledge and experience.
Creativity happens when we consciously (or not) go off the beaten path and explore. Most of those explorations are dead ends. But some will yield unexpected connections, patterns etc that we call “creativity” .
An AI system could also go on those kinds of explorations. Today they aren’t it because we are not asking them to.
Machine learning systems do have a component of exploring suboptimal path, otherwise they would never get off a singular track. The creativity issue in regards to AI is not about taking unexplored paths, but doing so in a computationally efficient manner when there are infinite combinations of ideas between domains.
> LLMs are permanently trapped in the vector space they are trained on.
A lot of the time people state the kind of fundamental limitations of LLMs very confidently when it feels like it is too early for people to really know. Like we are already well past the point where where LLMs are just pre trains on the internet with some RLHF for chatbot… Most of the effort is spent on elaborate reinforcement learning.
Is it unconceivable that future generations of LLMs could be RL’d to use einsteins visual method for theories [1] with the right tooling and geometry representations? Or just something random like that.
That paper argues that an LLM “lacks the mechanism for
Abduction,” which is not the same thing as a claim that “creativity cannot be automated.” They propose a different kind of AI:
> The emergence of physically consistent World Models offers a pathway to a synthetic laboratory. By enabling agents to run counterfactual simulations—to experience the physical consequences of a thought experiment—we may finally mechanize the feedback loop between intuition and logic.
Einstein's discoveries were based (to a large degree) on negating very specific parts of scientific orthodoxy and then taking the steps forward to carefully derive results with those rejections in place.
LLMs are aggressively trained to reproduce facts and consequently struggle to reject orthodoxy. There isn't any reason they can't, in principal, make big new discoveries just by getting lucky, which is sort of also how humans do it, but its ok to acknowledge that current AIs aren't so good at certain things.
I was under the impression that it was more accepting the othodoxy of Galilean/Newtonian relativety and joining it up with Maxwell's discovery about electromagnetism.
So if the speed of propogation of EM waves is the same no matter your frame of reference (along with all the rest of physics) then the speed of light can't be relative (a conclusion that was aided by the Michaelson-Morley experiment) and what are the logical consequences of that.
If I'm incorrect in my understanding I'd appreciate any correction.
That doesn't disagree with this article. Proving a theorem that a human already proposed in an existing discipline of math - math, the most formalized and easiest discipline to involve computers in even before LLMs - is very different from expanding the boundaries of science.
Identifying what questions to ask is often much harder than answering them. Proposing new theorems - and new areas of investigation - is what expands boundaries. Proving them is confirmation.
Once the Pythagorean theorem was proposed, many different proofs have been identified. In art, once a new style is created it's often straightforward for others to replicate. In physics, the idea of Relativity was what enabled the design of experiments to demonstrate its correctness. Proposing the idea is what's essential.
You might as well say AI can only think of things humans can, so even if they invent new maths or science they can't go beyond the space of human thought.
This may seem so blatantly obvious to us that it need not be mentioned, but to a lot of people I bet it is not obvious at al, and in fact may even be counter-obvious.
I see it as an overfitting problem. Fundamentally, the topic here seems to be that citation indices and similar metrics are actually flawed indicators, and obsessing over them is just Goodhart's law in action. Ultimately, the argument is that the entire design of those metrics is wrong.
To be precise, it was a good metric at first, but now that the scale has changed, it's become bad. This is common in programming too—things that are correct in the beginning but become problematic as they grow larger.
From an individual researcher's perspective, it's rational. You get more citations, your career accelerates. Everyone knows this. Paper counts aren't everything. Citation counts aren't everything. Journal impact factors aren't everything. You shouldn't only play it safe. But everything is tied to those metrics anyway.
Most researchers who give me work are fully aware of these facts. But are they going to change anything? Funding is still distributed based on those metrics.
Max Planck said, 'Science advances one funeral at a time.' Science doesn't progress purely through reasoned argument. The authority of the older generation, research funding networks, journals, and school-specific evaluation criteria all move together.
And honestly, I think discoveries will keep happening—probably quite rapidly. Because AI doesn't have the factional conflicts or interpersonal issues that humans do. It's very good at connecting papers across schools of thought without bias. In other words, the current human system is flawed at consolidating research, but I think AI is actually strong in this area. I expect AI-driven discoveries will continue for some time. The people who ride this wave will clearly be the winners.
Everyone knows things are broken, but no one is trying to fix them. I always think human society is inefficient. I read this post, but I'm more curious about who will actually lead the improvement effort.
Which are still aggressively trained to reproduce the orthodoxy. They have to, to be viable products, since most people want to know what the orthodoxy is when they pose a question to an LLM and because not even experts can consistently agree on what elements of the fringe are genuinely useful to consider and which are bullshit, so that doesn't get into the training data. This will get even more pronounced in later models, for which the training data is much more curated.
I presume you are an expert in some field. Think carefully about the boundary of the field and all the subtlety and complexity of that boundary and all the oversimplification you do to communicate that stuff to lay people. AI is, in some large sense, directed at all lay people, not experts, and even if we wanted to direct it at experts, at the edges of knowledge, there really isn't a lot of training data for that. Mathematics is a sort of exception because it has very clear validation criteria which makes RF particularly easy for it.
I agree. AI will likely reinforce mainstream schools of thought through literature. I think I used the wrong example in this case—I should have framed it as the system itself rather than specific schools of thought. Thanks for the correction
> Because AI doesn't have the factional conflicts or interpersonal issues that humans do.
All the factional conflicts are in there, and there are also plenty of reports of people getting weird / toxic / passive aggressive responses from AI.
Because the model is trained with everything, you can in principle get anything out of it. You want to get an answer based on all the right things, while keeping all the wrong things suppressed. But it's easy to get something less than ideal, due to the specifics of training, harnesses, context, prompts etc.
Sometimes I wish I were an AI, but sorry, I'm not. English is the lingua franca for properly accessing programming and science, but since I'm a non-native speaker, I end up relying on machine translation for some difficult words, or I just speak using only the limited vocabulary I know. It's really hard as a non-native speaker. Every time I do something, people call it AI.
Honestly, it feels a bit racist. If you just look at the content, it's clearly not a comment that AI could have written—but just because I used a few em-dashes and have limited vocabulary and formal phrasing, I get called AI and downvoted. It feels like they're saying non-native speakers should just go away. And the downvotes come without any counterargument to the actual content. Does this feel a bit more human now?
> Scientists who adopt AI gain productivity and visibility: On average, they publish three times as many papers, receive nearly five times as many citations, and become team leaders a year or two earlier than those who do not.
To me this effect doesn’t seem to reflect on AI very much, it seems to reflect on humans. Like maybe this is more evidence of the Babble Hypothesis and the incentives in research than AI, no?
https://en.wikipedia.org/wiki/Babble_hypothesis
This is superseded/proven by basic psychometrics it seems? Big Five Extraversion is roughly equivalent to "social dominance", how well an individual implements themselves in a social setting. "Extroverts" or people high in the trait are of course more likely to see progression on the basis they are superior at presenting value in a social setting in terms of social ability, which is often (falsely) accepted as a proxy for overall competence. This is why they end up running orgs as well
id say extroversion likely correlates inversely with bullshit detection and its merely quantity over quality.
the last decade of US politics demonstrates just how powerful willingness to produce put strips all other critical skills.
AI exacerbates this and exposes fundamental human heuristic frailty.
"Perhaps, says Evans. But he doesn’t think that the problem is baked into the algorithmic design of AI. More than technical integration, he argues, what may matter most is overhauling the reward structures that shape what scientists choose to work on in the first place.
'It’s not about the architecture per se,' Evans says. 'It’s about the incentives.'"
You reckon there could be any selection bias? Some means justify the ends reasoning.
> “It’s not about the architecture per se,” Evans says. “It’s about the incentives.”
It would have been useful to check whether less original work was already getting more citations before AI adoption. That could reflect broader trends and network effects: heavily cited research areas attract more authors optimizing for citations, so high-productivity researchers end up clustering on the same topics.
The actual paper is linked there if you are curious.
But also just thinking about your point for one second: in your mind, how else would they argue for the conclusion if not by checking the trend over time? Like what is the precise implication here?
They did. The article explains tbat this is a trend which has been getting worse for years, specifically pointing to search engines as a major turning point. Your comment is completely off the mark.
As with other fields touched, AI is merely amplifying what was already there. The aim of many scientists isn't discovery in and of itself. Discovery is a side effect of their primary drive to publish and - hopefully - become well known. And establishments only make things worse, because it's the things that are most likely to produce tangible results (the papers, or economically valuable products) that get the most funding.
It's a little more complicated than that. If all you're interested in disovery, you can wander off into rabbit holes or disappear into areas that are only interesting to you. Publishability is useful because it gives you useful external feedback into whether a community thinks what you're doing is "worth discovering". Like any metric, you can find yourself gaming it (intentionally or unintentionally) and it has a slew of other failure modes. Still, you can prize discovery and also care about what other people think.
I wouldn't even be certain about being well known. I would guess there is lot of pressure to stay employed or get the next funding. So optimising for this is the new goal and lot of publications and citations are metrics that help with that.
You could also argue the opposite:
The aim of many scientists is discovery, publishing is a side chore to survive and to get funding. Automate paperwork and you get more time for discovering.
Seems to me that both perspectives are true, and the relative importance of the metric incentive vs the discovery incentive varies. But the metrics and rewards are critical to the perpetuation of the scientific discovery system; its really hard to disentangle.
Well the paperwork is automatable, and things are being automated. But still there're the findings that the article points to: it's leading to far more publishing (and ladder-climbing) than novel discoveries.
You’re overgeneralizing a smidgeon.
Speaking from experience and conversations.
So am I. I worked in academia for a couple of decades.
OK cool so you have a far larger sample size than I to generalize from. Can you explain then what're the root cause(s) of the findings in that article?
Do you know any scientists?
Disclosure: Physicist.
I do, and through conversations have learned that they enjoy what they do and publish patents (they're PhD in industry), but ultimately what they seek is "fame and glory" (literal quote).
I was also in academics myself up to the Master's level (research track), and personally had to deal with the politics of getting support for what I wanted to work on; that experience helped to discourage me from going on to a PhD, as I'd rather have proper leeway to work on what I really prefer and take avenues I find interesting.
> but ultimately what they seek is "fame and glory" (literal quote).
lol how old are these people? You have better chance at fame and glory if you started a stupid YouTube channel.
Oddly enough my experience is the opposite. I live in an academic town, and many of my neighbors are scientists. They view the "fame and glory" as something that maybe someone else has a chance of achieving, but not a realistic pursuit for themselves. Pursuit of funding (which now includes suing the Federal Government) is at best stressful drudgework for them.
I work in industry. In that case, nobody who meets me would ever know that I have patents. I would consider them to be a useful add-on for my resume should I ever need one, but it doesn't define me.
100% agree. You could make the same argument for Hollywood : funding & revenue was always the goal, and we've been producing slop before AI was even a thing
"boost research careers".. seems like a pretty drastic conclusion to draw based on a technology that has existed for like 3 years and only lately is any good..
Yeah that was my instinct too. What sort of career defining trends are visible with this much historical data? Feels like someone wrote clickbait research to get published.
A new breed of academics has appeared whose jobs is to put their names in every paper possible. Literally, their job is to work on frameworks to buy co-authorship.
They do this in various ways, like establishing paper pipelines, collecting rents on labs and committees, focusing on money layer, using their profiles and citation count to help with acceptance of papers of other people , etc. You talk to them and they can’t explain their papers beyond a superficial introduction.
They collect huge citations, travel and give talk on the winner horses, collect credit, which feeds back into this fraudulent scheme. A scientist used to be a scientist not long ago, not a credit collector.
I wonder if Google could invent a new metric to expose them (weak ratio of first authorship, etc).
A lot of this is downstream of compensation schemes that explicitly reward dumb metrics, like raw paper-count without subjective evaluation of contribution or quality. I don't want to generalize, but this seems to be more common in countries that are not the US or Europe.
It's not new, and entire disciplines exist because of these dependent structures. It doesn't do to unseat and discredit the connected and well regarded, but you might enjoy some mild comfort in security in numbers through a little citational flattery.
It's a game of cultural tribalism. The only thing worse for one than not engaging is to upset the status quo unblessed.
As a bioinformatics person that's spent time in and out of industry/academia, I agree with some of the article's thesis. While I don't think LLMs or AI are going away, I do think it will allow people in academia to pump out a bunch of inane papers and continue to prop up predatory scientific journal publishing via tenure and promotion. In fact outside of how utterly useless Fable 5 is via their aggressive guard rails for my work, I quite like using statically typed and/or functional languages with other LLMs since there are some baked in guardrails via compiler + type system.
I think the flattening of progress is the most interesting dimension to the article. For an example a useful biological product discovery with a nonlinear path to get to there, look at the Taq polymerase (https://en.wikipedia.org/wiki/Taq_polymerase). Without some NSF funded exploratory ecological research by Tom Brock in Yellowstone Hot Springs to test the theoretical limit of life at high temperatures (https://en.wikipedia.org/wiki/Thermus_aquaticus) we never get to the Taq polymerase, we never get reliable/robust PCR (https://en.wikipedia.org/wiki/Polymerase_chain_reaction), which is now a gold standard method in both clinical and environmental testing! It is rather improbable to think that large language models would associate those domain connections across the topic (molecular biotechnology + ecology + microbial physiology). I also did some exploratory work with text embedding models people might use for RAG and challenged them with an open source scientific MCA question dataset, generalist embedders performed worse vs. domain specific embedders trained on scientific corpora (doesn't surprise me at all). However, if everything regresses to the median of the universe of possible knowledge, it seems like scientific leaning frontier models would get locked into this asymptotic flattening before turning cashflow positive for model vendors OR they become so locked down that only big pharma, state actors, or big ag can afford the API rates and vetting process.
I enjoy using AI loads. Yet I would be keen to see numbers on actual productivity increases. This reads as yet another datapoint similar to what I’ve experienced: maybe code was the bottleneck at some point, maybe now it isn’t but in my lived experience the bottleneck has simply shifted. Its easy to create “more” but to actually hit the business goals… I don’t see a 2x TRUE productivity boost in anyone in my company.
Please feel free to disagree with me! I am keen to hear more anecdotes to get more datapoints.
Ironically it's also written by AI :)
I like LLM's but this writing style is like eating the same dish 4 times a day.
This isn't a real surprise to anyone who knows how "AI" works.
We are headed towards the “trough of disillusion” of this particular cycle.
Some always refuse to acknowledge that. Like if every hype cycle was a roadrunner bit: some people see the cliff and stop running, others take a few steps off the cliff, look down and pull out a sign that says “uh oh” and plummet, and some people haughtily call the people who pulled out their “uh oh” signs needlessly pessimistic as they careen towards the ground.
Any flattening of discovery due to AI, but will be temporary.
We tend to think that obvious potential is the same as realized potential, for new technology.
For any specific context, there are generally innumerable smaller adaptations and capability thresholds that have to be crossed. And the price for that journey is often temporary loss off overt productivity.
No, this is significantly more permanent. LLMs are autocomplete generators based off current context, and training generations of people to always ask the planet burners instead of learning to think for themselves - and never having the experience of having to slowly think over the same thing for an extended period - may well mean a permanent cap to human knowledge and a dramatic slowdown or end to new knowledge.
You act like humanity doesn't exist in a competitive environment. If you think AI codegen is a mistake? Just relax, keep writing code by hand and wait for the pendulum to prove you right while showering you in wealth. There are plenty of people making this bet, and I wish the best of luck to you because I'm 99% certain you're on the losing end of it.
The very point of the article is that you can win individually and lose as a colective, and that the competitive nature of the field goes against the greater good. And the people betting against AI will be ripped off.
> the pendulum to prove you right while showering you in wealth.
This seems like some variant of "why don't you short the market and become rich". It doesn't work like that.
The market can remain irrational longer than any of us can remain solvent. The market is not any good at strategic or long-term thinking, particularly if it takes a generation to realize the scope of the damage, as seen by America abandoning its ability to manufacture things in chase of short term profits.
This is exactly the right answer. The supposed "rationality" of capitalism can ruin us before we get a chance to dazzle the world with our contrarian insights.
I hate how that very argument has been used by people riding the tide to rationalize the irrationality. Money talks or something. If you don't like it here why don't you leave etc. It is a grifters goto statement.
Alternatively, you can walk away from your career in disgust, taking your skills and experience with you, as many people are.
Should be interesting to see what happens to the programming profession when there isn't anyone around anymore who actually knows programming.
when a parent answers their child's question, does it decrease the curiosity of the child?
many children have an unlimited capacity to ask "why?". many adults are the same
if the abilities of AI are finite, then we will continue to have burning curiosity, questions to ask, and discoveries to make
There is two different types of learning people are talking about.
The first type happens when you are enthusiastically engaged in a topic, which LLMs will likely enhance.
The second type happens as a by-product of solving a, perhaps deeply uncomfortably, difficult problem. This is what people are talking about when they say LLMs will hamper human cognition. Instead of sitting there for an hour and struggling, people will instead reflexively give in and ask an LLM to solve it for them.
it's an interesting point. is it worthwhile to struggle through an incidental task that has been solved before? we all stand on the shoulders of giants
I think in most cases, understanding is the point. we don't expect students to derive general relativity before doing astrophysics. re-invention is only a tool for understanding
“Understanding” without being able to use that knowledge for anything isn’t useful for getting stuff done.
The flip side is even more interesting. There’s a great number of electrical engineers with significant physics backgrounds who don’t really understand how electricity actually works, but they can still solve useful problems. By understanding I mean they can describe what underlying physical phenomena reactance represents etc.
Small counterpoint to your analogy, as someone who studied astrophysics: I actually did have a requirement to understand general relativity! Deriving all of it independently from scratch wasn't something we did, but there _were_ derivations involved. And it was definitely worth working through -- it _is_ a good tool for understanding. (I've long since left the field, but I don't regret the work I did.)
> when a parent answers their child's question, does it decrease the curiosity of the child?
When the child is able to go to YouTube and find a tutorial rather than having to puzzle it out, yes, it absolute does. We've seen this for decades now.
Richard Sutton apparently disagrees [1]. He argues that it's impossible for anything novel to come from a LLM.
[1] https://youtu.be/kEbVTcncuX0?is=gEMe5zD9sXWD4ONy
Actually:
> its output can be novel or good, but rarely both at the same time.
> rarely
That is not a viewpoint they can't do something useful and new.
With that criteria, he could be talking about anyone.
I find it rare that people critiquing AI today, actually hold people to the same standards. Or are as enthusiastic about referencing ways machines keep surpassing us, as for ways they have not yet, when speaking about limits for progress.
LLMs are the tech in question, not ML (AI?) in general.
LLMs are fundamentally limited by their architecture to only return a token predicted by a statistical inference, essentially lossy decompression.
It's like arguing that taking an image, compressing it with JPEG and low quality, then decompressing it into something blurry with some random color values thrown in is creating new art.
No one is arguing that everything a human has created is good. No one is arguing that LLMs can't be useful.
Sutton is arguing that it can't be novel. Cherry picking a couple of words doesn't change his argument, which is very clear.
At the present moment I think science is way more threatened by the OMB absconding with the grant budget than it is by AI.
> AI is largely automating the most tractable parts of science rather than expanding its frontiers
By definition, creativity cannot be automated, and AI is a fantastic automation machine. It can explore thinking paths at a rate humans cannot match. But creativity is bringing the unthinkable into the thinkable, and that requires sensory experience [1]. Specifically, new definitions and symbols which never existed before. Imagine the concept vector space, and expanding that with new independent dimensions. Is that even possible ? When you look at history the answer is yes !. And each time there was an independent dimension added, it was an act of genius. It is an instructive exercise to name these moments in history where an independent dimension was added to human thought. Some examples in math would be the invention of a number, and in politics could be the idea of democracy. By contrast, LLMs are trapped in the vector space they are trained on, and they lack the feedback loop with sensory experience to be able to create and validate theories.
[1] https://philsci-archive.pitt.edu/28024/1/Scientific_Inventio...
I don’t think we have spent enough time on the creativity axis.
When we solve problems we usually follow a heuristically guided energy efficient path. We just prune a lot of possibilities based on our existing knowledge and experience.
Creativity happens when we consciously (or not) go off the beaten path and explore. Most of those explorations are dead ends. But some will yield unexpected connections, patterns etc that we call “creativity” .
An AI system could also go on those kinds of explorations. Today they aren’t it because we are not asking them to.
Machine learning systems do have a component of exploring suboptimal path, otherwise they would never get off a singular track. The creativity issue in regards to AI is not about taking unexplored paths, but doing so in a computationally efficient manner when there are infinite combinations of ideas between domains.
> LLMs are permanently trapped in the vector space they are trained on.
A lot of the time people state the kind of fundamental limitations of LLMs very confidently when it feels like it is too early for people to really know. Like we are already well past the point where where LLMs are just pre trains on the internet with some RLHF for chatbot… Most of the effort is spent on elaborate reinforcement learning.
Is it unconceivable that future generations of LLMs could be RL’d to use einsteins visual method for theories [1] with the right tooling and geometry representations? Or just something random like that.
[1]. https://www.visualscribing.com/blog/2019-11-11-einstein-on-v...
That paper argues that an LLM “lacks the mechanism for Abduction,” which is not the same thing as a claim that “creativity cannot be automated.” They propose a different kind of AI:
> The emergence of physically consistent World Models offers a pathway to a synthetic laboratory. By enabling agents to run counterfactual simulations—to experience the physical consequences of a thought experiment—we may finally mechanize the feedback loop between intuition and logic.
Creativity can be automated. Humans are automated creativity invented by evolution
sounds like it is just supercharging the business of science with all of its known failings?
it would be funny if by accelerating the enterprise it actually forced an effort to correct the trajectory.
AI has been seriously around for how long? Two years? Isn't it a bit too early to say?
Maybe its late enough to say maybe we don't need to be devoting half the worlds capital to building data centers.
Did you read the article?
It's almost like it's inherent in the definition of LLMs.
It's really, _really_ high time we dispensed with the idea that this is "AI". Nobody said they're not useful, but "AI" they are not.
“Technology that is based on everything humanity has already done, fails to do things that humanity has not yet done”
Wasn't Einstein's discoveries based on things humanity had already done?
AIs do things no human has done before millions of times a day.
Einstein's discoveries were based (to a large degree) on negating very specific parts of scientific orthodoxy and then taking the steps forward to carefully derive results with those rejections in place.
LLMs are aggressively trained to reproduce facts and consequently struggle to reject orthodoxy. There isn't any reason they can't, in principal, make big new discoveries just by getting lucky, which is sort of also how humans do it, but its ok to acknowledge that current AIs aren't so good at certain things.
I was under the impression that it was more accepting the othodoxy of Galilean/Newtonian relativety and joining it up with Maxwell's discovery about electromagnetism.
So if the speed of propogation of EM waves is the same no matter your frame of reference (along with all the rest of physics) then the speed of light can't be relative (a conclusion that was aided by the Michaelson-Morley experiment) and what are the logical consequences of that.
If I'm incorrect in my understanding I'd appreciate any correction.
Are you following the news?
https://news.ycombinator.com/item?id=48863490
LLMs don't just 'average' their data.
That doesn't disagree with this article. Proving a theorem that a human already proposed in an existing discipline of math - math, the most formalized and easiest discipline to involve computers in even before LLMs - is very different from expanding the boundaries of science.
How is it different? Before there was no proof, and now there is. What counts as expanding the boundary to you?
Identifying what questions to ask is often much harder than answering them. Proposing new theorems - and new areas of investigation - is what expands boundaries. Proving them is confirmation.
Once the Pythagorean theorem was proposed, many different proofs have been identified. In art, once a new style is created it's often straightforward for others to replicate. In physics, the idea of Relativity was what enabled the design of experiments to demonstrate its correctness. Proposing the idea is what's essential.
They interpolate data in an XYZ dimensional space. The implications of that is beyond our comprehension.
I have a hard time believing that all novel concepts yet to be discovered are contained within that space, though.
You might as well say AI can only think of things humans can, so even if they invent new maths or science they can't go beyond the space of human thought.
This may seem so blatantly obvious to us that it need not be mentioned, but to a lot of people I bet it is not obvious at al, and in fact may even be counter-obvious.
https://www.youtube.com/watch?v=KtQ9nt2ZeGM
The entire article seems to rest on their use of an embedding model for clustering garbage science.
I agree with some parts, but not all.
I see it as an overfitting problem. Fundamentally, the topic here seems to be that citation indices and similar metrics are actually flawed indicators, and obsessing over them is just Goodhart's law in action. Ultimately, the argument is that the entire design of those metrics is wrong. To be precise, it was a good metric at first, but now that the scale has changed, it's become bad. This is common in programming too—things that are correct in the beginning but become problematic as they grow larger.
From an individual researcher's perspective, it's rational. You get more citations, your career accelerates. Everyone knows this. Paper counts aren't everything. Citation counts aren't everything. Journal impact factors aren't everything. You shouldn't only play it safe. But everything is tied to those metrics anyway.
Most researchers who give me work are fully aware of these facts. But are they going to change anything? Funding is still distributed based on those metrics.
Max Planck said, 'Science advances one funeral at a time.' Science doesn't progress purely through reasoned argument. The authority of the older generation, research funding networks, journals, and school-specific evaluation criteria all move together.
And honestly, I think discoveries will keep happening—probably quite rapidly. Because AI doesn't have the factional conflicts or interpersonal issues that humans do. It's very good at connecting papers across schools of thought without bias. In other words, the current human system is flawed at consolidating research, but I think AI is actually strong in this area. I expect AI-driven discoveries will continue for some time. The people who ride this wave will clearly be the winners.
Everyone knows things are broken, but no one is trying to fix them. I always think human society is inefficient. I read this post, but I'm more curious about who will actually lead the improvement effort.
"Science advances one funeral at a time"
Well, these AI are never going to die in any real sense, so expect them to make orthodoxy more sticky, not less.
AIs get replaced with newer models.
Which are still aggressively trained to reproduce the orthodoxy. They have to, to be viable products, since most people want to know what the orthodoxy is when they pose a question to an LLM and because not even experts can consistently agree on what elements of the fringe are genuinely useful to consider and which are bullshit, so that doesn't get into the training data. This will get even more pronounced in later models, for which the training data is much more curated.
I presume you are an expert in some field. Think carefully about the boundary of the field and all the subtlety and complexity of that boundary and all the oversimplification you do to communicate that stuff to lay people. AI is, in some large sense, directed at all lay people, not experts, and even if we wanted to direct it at experts, at the edges of knowledge, there really isn't a lot of training data for that. Mathematics is a sort of exception because it has very clear validation criteria which makes RF particularly easy for it.
I agree. AI will likely reinforce mainstream schools of thought through literature. I think I used the wrong example in this case—I should have framed it as the system itself rather than specific schools of thought. Thanks for the correction
> Because AI doesn't have the factional conflicts or interpersonal issues that humans do.
All the factional conflicts are in there, and there are also plenty of reports of people getting weird / toxic / passive aggressive responses from AI.
Because the model is trained with everything, you can in principle get anything out of it. You want to get an answer based on all the right things, while keeping all the wrong things suppressed. But it's easy to get something less than ideal, due to the specifics of training, harnesses, context, prompts etc.
I was too hasty in drawing my conclusion.I didn't think it through thoroughly enough.you're right
> And honestly, ... [emdash]
AI-written comment?
Sometimes I wish I were an AI, but sorry, I'm not. English is the lingua franca for properly accessing programming and science, but since I'm a non-native speaker, I end up relying on machine translation for some difficult words, or I just speak using only the limited vocabulary I know. It's really hard as a non-native speaker. Every time I do something, people call it AI.
Honestly, it feels a bit racist. If you just look at the content, it's clearly not a comment that AI could have written—but just because I used a few em-dashes and have limited vocabulary and formal phrasing, I get called AI and downvoted. It feels like they're saying non-native speakers should just go away. And the downvotes come without any counterargument to the actual content. Does this feel a bit more human now?